1

Re: Extract to mapping software

Hello,

I have got a report that extracts data from surveys. Information included is common name, record type.x,y details. I use this information to be able to plot the positions of the findings from the survey in GIS software. Currently, I run the extract and save it in an excel spreadsheet. I then manipulate the data in excel to break it down into record types or species etc.

What I was wondering is if I could do this breakdown in Recorder 6. i.e. create one report that could create several extracts broken down into record types or species or whatever choice. The report that I have written uses the date and sample reference as variable inputs. I guess that I would have to have multiple sql selects to create the extracts. Would that mean extracting to temp tables then outputting the data from each of them? Would that be the best way?

Any suggestions? Or perhaps the extract/excel data manipulation is the simplest?

Cheers for now

Gary Pocklington

2

Re: Extract to mapping software

You can almost definitely do the manipulation in Recorder, but Recorder can only output one file at a time so you woudl need to run the extract with different parameters one at a time and save the output after each extract.  An xml with the parameters you need would do the job. Alternatively, you could do all the work in Recorder so the output file contains all the data you need in the format you require, sorted into order based on your selection criteria. You could then just copy and paste each Group into a new spreadsheet, or design an Excel macro which did the split automatically for you. If you can provide a more detailed information I can advise further.

Mike Weideli

3

Re: Extract to mapping software

Hello Mike,

I was having another re-think on this and looking at what the driver for the mapping is. Currently I have an XML report that I can input date and sample reference. I create my own unique sample references so I can either get single records or the lot. In that extract I get details such as common/actual name, grid ref, x/y locations, record type(short_name), sample reference, sample comment field.

So, as far as the mapping goes there are lots of variations. There does not seem any point in just producing a single map which shows all the places where species were found. I think that it needs to be broken down into species and then that is broken down into the record types. For instance when I have recorded rabbits I have used record types of field record('live' sighting), droppings and tracks/trail. If I just put rabbits on the map without breaking them down into these categories then it would be meaningless.

So, thats were I am trying to get to. I guess that I could add further selection(s) to my XML report which would allow species and record type to be input. That would give me the file I am looking for. Then I can either show these on a map as individual files or have them as one map with different symbols/colour. It is probably easier to do it in Recorder as I am more familiar with SQL than excel macros.

Anyway, I would be interested to hear any further thoughts/ideas. Or indeed if the method I am using is not sound then any comment would be interesting.

Cheers for now

Gary Pocklington

4

Re: Extract to mapping software

Gary

It depnds a bit on how many files yiu need to get the results you want,  If there are not too many  then generating and exporting them individually would be fine.  If there are a a large number then generating one export with columns identify the splits might be easier. Which GIS s system are you using ? 

Mike

Mike Weideli

5

Re: Extract to mapping software

Hello Mike,

I have only recently applied updates to Recorder 6. I now notice that there are options when you have extracted details for reporting that you can then filter them and create excel or report files. This might also work as an option.

I use Mapmaker GIS which is the only affordable GIS solution I found that is feature rich. It does have an ability to be able to create Databases but I do not want to start replicating data. However it does have options to be able to load data in with enough details to be able to plot positions and break down into categories. In my opinion it is a pretty solid piece of software and I keep on finding options that make it work and look better.

Cheers for now

Gary Pocklington

6

Re: Extract to mapping software

Gary

I am not familer with Mapmaker, perhaps someone else can advise. In most GIS system  you can easily import an Excel file into the system database and used the attributes within the sofware  to control the maps.

Mike

Mike Weideli

7

Re: Extract to mapping software

Hello Mike,

I am fairly new to Mapmaker myself so it is a case of learning as I go along. I am sure that there is a similar facility in Mapmaker.

I think as the data is in a relatively low volume then at the moment extracting an using excel files seems to be the best option. Recorder has the facility to filter the data so I can create individual files. I will try that and see how it goes. Thanks for your replies.

Cheers for now

Gary Pocklington

8

Re: Extract to mapping software

Gary

I downloaded MpaMaket trial and had a look. You can import attributes into the database, but so far I can't see how you can use these to control what is displayed. Possibly the ID can be used, but I couldn't immediately see how this would work. Perhaps when you get more familer with MapMaker you may find a better way.  Let us know how you get on.

Mike Weideli

9

Re: Extract to mapping software

Hello Mike,

As I so far understand it, in MapMaker the output format is controlled by style files. i.e. the polyogons, lines, symbols etc. You can customise these to your requirements. Each of these styles has a number allocated to it. Starts at zero which is default then 1 upwards is whatever you want. There are then various options on how you can use this to display objects. They do say in the manual that understanding the style concept is one of the keys to understanding how MapMaker works.

There is also a facility in it which does allow you to create a vector file from an excel spreadsheet.

I had a thought yesterday that what I could do was to add the last two characters of the record_type_key to my extract. If I then created styles in MapMaker to match this it might work.

Anyway, that is something I am going to experiment with and I will update when I have tried out some options. Definitely looks as though its possible to get a result from it.

Cheers for now

Gary Pocklington

10

Re: Extract to mapping software

Hello,

I have come up with a solution that seems to work ok so far. I have changed my extract to include some more fields to tally in with MapMaker. Although, I am sure with further investigation it could be further refined. As well as x,y co-ordinates, species name (common and actual), record_type.short_name and grid ref I have included a couple more. I added taxon_occurrence _key which gives a unique identifier. I have also extracted the last two chars. from the record_type_key. These can be used in MapMaker to control the style.

Once the extract is run I output the result to an Excel file. This is an area that is still not clear to me with its relation to MapMaker. I am presuming that every time I extract from Recorder 6 I have to do a similar extract to MapMaker. i.e. if the spreadsheet changes MapMaker does not get automatically updated.

Anyway, once the Excel file is there I can either manipulate the data there or import into Mapmaker from the Excel file and manipulate the data in there. MapMaker allows me to create a vector file from Excel. The output in MapMaker can be controlled by the use of styles and to a degree is flexible. What I am really looking for is flexibility. i.e. can I easily switch between showing the data by species or by record type or date or all of it?
At the moment from what I can see that is controlled by the extracts that you feed into it.

Perhaps there is a more flexible way of linking the data to MapMaker but I haven't progressed that far.

Anyway I am getting some results that are getting to where I want it.

Maybe, if others have some thoughts on how other GIS software works it  might give me a clue for investigation into finding similar functions in MapMaker.

Cheers for now

Gary Pocklington

11

Re: Extract to mapping software

Hi Gary
Not sure if you are aware of these, but there are other open source GIS products you could look at including uDig, Gaia and Quantum GIS. I'm not suggesting they are better or worse than MapMaker but there are other free options.
Using style files might let you filter out different species/record types etc, but doesn't sound the most efficient way when you have lots of data. Also it means making style files for each possible report combination which could mean quite a lot.
Best Wishes

John van Breda
Biodiverse IT

12

Re: Extract to mapping software

Hello John,

I think in your last point it highlights what the main problem is when I try to put the data into GIS. For instance on the small amount of data I have got so far an example can be where a species can have four record types field record, casualty, droppings or tracks. The question then becomes how can I present the information in GIS?

My current understanding of GIS is that you build layers to make up a map. I have got files from the OS Open Data series. In the vector files they supply they are broken down into for example roads, railways etc. within each of these they are categorised as motorway, a-road, b-road or main line, single track etc. So that is similar to how I built my extract from Recorder 6. In essence I am extracting all the species but my categories are the record types i.e. droppings, field record etc.

At the moment I can only see that in GIS I would have to build layers for the variations that I would be looking for. If the data is not shown in GIS as broken down by the record types then it becomes meaningless. In Mapmaker, at the moment the simplest way I can see of doing it is, to extract all the records from Recorder 6 into an excel file. This then becomes the 'database' input to MapMaker. I would then need to create layers for species/record type. Again, at the moment the only way I can see how to do this is to create 'snapshots' from the excel file of the individual species. These then form the base for the layers in GIS.

I have had a quick look at one of those 'open source' GIS programs and as far as I can see it operates on a similar basis?

So, it might be a combination of me not fully understanding how GIS works and the limitations within that software. I think my initial thoughts were that I would only need to do one extract of all the data and then within GIS I would be able to manipulate the display without having to do further extracts. Anyway, perhaps it is in there somewhere but I have not yet come across it?

Cheers for now

Gary Pocklington

13

Re: Extract to mapping software

Hi Gary
To some extent, GIS applications are designed to build maps up from layers as you suggest - this works well for geographical applications but less well for a dynamic analysis tool. I'm afraid my hands on experience with GIS is limited as I've tended to work more with bespoke solutions for a particular problem, so I am not sure how good the support for filters on the layers is in each of these applications. But I do know, for example, that the WFS standard (a web standard for loading spatial data e.g. onto a GIS map) supports a filter which would let you create a single datasource (=feature type) and add multiple layers for that data source, each with a different filter.

Can I ask what your reasons for not doing this in Recorder are? I suspect the answers are obvious, but I thought it would be worth checking.

Regards

John van Breda
Biodiverse IT

14

Re: Extract to mapping software

Hello John,

I must admit that I didn't find the mapping particuarly friendly in Recorder. So, for more flexibility I looked for a solution in GIS. I sort of feel that it gives me more options.

However if you are saying that what I am trying to do could be acheived in Recorder then I would certainly give it a try.

Cheers for now

Gary Pocklington

15

Re: Extract to mapping software

Well, I'm not saying that the mapping in Recorder will give you the flexibility of a GIS, but if you have a suitable background layer then it becomes quite easy to do what you need. For example you could use the Report Wizard to filter out each different "layer" of data and send them to the map, or you could write an XML report with a simple set of parameters (species, record type etc) which outputs a list of occurrences with their spatial references, then send the output from this report to the map.

John van Breda
Biodiverse IT