1

Re: How can we make importing Mapmate data easier?

It has been suggested that making the importing of mapmate data easier in the import wizard would be good.

So far the suggestions have been around abundance data and names - the proposals:

1.    Provide two new columns: ‘Measurement Data’ and ‘Measurement Qualifier (sex/stage)’. These columns will require each other when one is selected (but cannot be used with the exisitng column Abundance Data)  The Measurement Data column should be free text of maximum 20 characters. The entries in the Measurement Qualifier (sex/stage) field would be matched against the Measurement Type term list (Abundance/Measurement Qualifier).

2.    Provide a facility for multiple recorder name parsing to be turned off for a given import. This would allow multiple names as stored in Mapmate to be imported as groups of names. This could be either as an option in the import wizard itself or in the Tools menu whichever is cheapest.

Do these sound reasonable ways of dealing with the data - do you think this would make importing mapmate data easier? Are there any other ways to make importing mapmate data (or R2002/R3/Adit etc) easier?

Many thanks in advance for your input.
Best Wishes,
Lynn

2

Re: How can we make importing Mapmate data easier?

Lynn, I'm not very familiar with the Recorder data model, so may be misunderstanding what you're trying to do here, but in MapMate there are three separate fields for abundance, sex and stage, so if the sex and stage information is to go into a single 'Measurement Qualifier (sex/stage)' field then it will need to be concatenated in any query that is used to run the data out of MapMate, which means there's be an extra bit of tweaking that will have to be done at the MapMate end.

And I'm not sure what you're proposing should go into the 'Measurement Data' field?

Martin Harvey
Biological Records Centre
CEH Wallingford

3

Re: How can we make importing Mapmate data easier?

Lynn

Re Map Mate data - The Abundance/sex stage  part seems ok, but I am not too happy about the ideas on the name.  This seems a big departure from the principles of  Recorder. I think we should be encouraging the proper use of names rather than encouraging an inferior approach. Once you start going down this route then you are effectively abandoning the Recorder model.   Also from a practical point of view, you would have to increase the length  of one of the fields (Surname ? ) considerably to take the data or create a new field in the individual table to hold the unparsed names.  Would it not be beter to look at improved ways of parsing the map map names ? 

With regard to Recorder 3. The procedures provided are a bit complicated, but no more than neccesary to do a reasonably accurate transfer.  There are a few bugs in the conversion software which don't stop it working, but which produce  inaccurate results. There are also  number of things it doesn't do to Recorder 6 standards.  However,  generally the process is  fairly robust except where  the original data has been corrupted in some way, or is not in a standard format, which is most likely to have occured with larger systems.  There is little consiststency  in the problems encountered so it may be difficult to deal with these in the automated process,


With Recorder 2002, most of the problems seem to be with corruptions caused by early version of the import wizard, which didn't work properly. It would be possible to define a number of pre-import checks which would highlight these problems so that that can be corrected either automatically, if possible.or manually if not.
 

Mike.     


.

Mike Weideli

4

Re: How can we make importing Mapmate data easier?

Another thought: is there a field in Recorder that could store the unique ID (GUK) of each incoming MapMate record, so that this could be used to flag up subsequent imports of the same records (or edited versions of them)? I imagine that this would also be useful as a way of tracking data coming in from many other programmes or customised Access databases, not just specific to MapMate.

Martin Harvey
Biological Records Centre
CEH Wallingford

5

Re: How can we make importing Mapmate data easier?

Hi Martin,
That's a good idea, and one which I have implemented in Indicia. It can be quite useful to allow an "external id" to be associated with each record so that if the record comes from an external system you have a unique reference to the source. It makes future synchronisation of data much easier.

Best Wishes

John van Breda
Biodiverse IT

6

Re: How can we make importing Mapmate data easier?

I would have thought given the limited budget, improving the core of Recorder would be a better idea.

Charlie Barnes
Information Officer
Greater Lincolnshire Nature Partnership

7

Re: How can we make importing Mapmate data easier?

The Recorder import of MapMate, data does in effect retain the MapsMate key so new imports will update existing records rather than creating new ones.  This should work fine with the main Record (Taxon_Occurrence), but can run into difficulty with fields which need to be parsed (eg names). In these cases if the MapMate data is changed  (ie adding or deleting  or changing what is there) the keys generated may not be the same for the new import as the original. This can give rise to anomalies. However, data is not that often changed so this isn't  perhaps  that big a problem in reality.

With large MapMate datasets it can take a long time to get them into a format which Recorder can import so there is scope for improving the import process, by being able to accept MapMate fields without the need to manipulate them. This applies  particulary to abundance, sex and stage where Recorder should be able to process these fields without any prior manipulation.   The free format nature of the MapMate recorder name, where all sort of combinations and orders are possible within the same file,  makes this particularlly difficult to deal with. Nevertheless, I don't think it would be right to just accept the MapMate name unparsed into an existing Recorder field.  Possibly an additional field could be added to the Sample to hold the names of the Recorders in the MapMate format  (I think there may already be an unused field in there called Recorders ). The Sample and Survey Events Recorder could then use a generic name  (Sundry ?). It would then be posible  add the unparsed field into the Recorders field on report output by making a minor adjustment to the UDF which does the concatenation.         

With data from other sources, retaining some sort of external id would  be a good idea.

Mike

Mike Weideli

8

Re: How can we make importing Mapmate data easier?

Hi Charlie
Just to be clear, Indicia is a separate product to Recorder and is Open Source, so it wasn't impacting on Recorder's funds. I often do a few fixes and enhancements to Indicia when on the train!

Cheers

John van Breda
Biodiverse IT

9

Re: How can we make importing Mapmate data easier?

johnvanbreda wrote:

Just to be clear, Indicia is a separate product to Recorder and is Open Source, so it wasn't impacting on Recorder's funds.

My reply was in reference to the original post by Lynn which was suggesting improvements to Recorder.

johnvanbreda wrote:

I often do a few fixes and enhancements to Indicia when on the train!

If only the same could be said of Recorder.

Charlie Barnes
Information Officer
Greater Lincolnshire Nature Partnership

10

Re: How can we make importing Mapmate data easier?

Thank you all for your posts. I agree that it needs to be easier to import Mapmate data (which is why am asking for your opinions!) and it looks like abundance data needs three fields which I've noted and that the names need a bit more thought - I do like Mike's ideas on how to fdeal with these.

Are there any other areas that cause a lot of processing time from Mapmate into Recorder or are these the key ones?

Despite having little funds left this year it is always a good idea to forward plan for areas that could be improved and part of this is gathering information and ideas about these for when the time comes that the money becomes available (funds can occasionally crop up with specific criteria late in the financial year). It would be a shame to not have thought ahead and therefore have nothing to spend it on!

Thanks,
Lynn