1

Re: Importing MapMate records

Does anyone know if the problem where importing MapMate records produces duplicates has been cured?

I happens when an import of MapMate records assigns custodianship to MapMate - if the import is repeated (or another import including the same records) then SOME (but not all) records can be duplicated - I thought this was impossible, but there was obviously a routine in Recorder to process these records and make 'arbitory' decisions

The first import generates the GUI MMEEExxxEEEyyyyy
The second import creates GUI MMEEExxx1EEyyyyy
where xxx id the MapMate CUK and yyyyy is the other 5 characters of the MapMate GUI - I have noticed sometimes the '1' can be 2 or even 3 - I don't know if this means I have 3 duplicates of some records :(

The reason I ask is I'm just about to reimport a load of records and don't want a load more duplicates

Craig Slawson
Staffordshire Ecological Record

2

Re: Importing MapMate records

Craig

For Taxon_Occurrence the key should be directly releted to the MapMate key and there should  never be any duplicates.  Is it definitely taxon occurrence keys you are seeing as duplicated ?


Mike

Mike Weideli

3

Re: Importing MapMate records

YES !

I'm currently going through deleting them from a previous import which managed to create well over 1,000 "new" duplicate records - I know they are duplicates, because my import routine puts the MapMate CUK in the "Surveyor's Ref" field, so I can easily run a "Find Duplicates" query in Access to pick up the duplicates - most are in the same sample as the original, but not always, sometimes the sample has been duplicated as well!

Craig Slawson
Staffordshire Ecological Record

4

Re: Importing MapMate records

I think I've found a pattern with the duplicates - if the second import has a different Taxon Version Key allocated it seems to duplicate - i.e. if the taxon was chosen off a different checklist

Craig Slawson
Staffordshire Ecological Record

5

Re: Importing MapMate records

I can confirm the bug is still present, an import I did yesterday has created 1 duplicate, the original entry was Acleris ferrugata whilst the new entry was Acleris ferrugata/notata - this created a new record, so I finished up with both in the system

People need to be aware of this bug if they are importing lots of MapMate data

Craig Slawson
Staffordshire Ecological Record

6

Re: Importing MapMate records

Craig

This is serious problem which undermines the whole idea behind the MapMate import, so I have spent some time looking at it over the weekend and consulted  John Van Breda.  John  says the keys are generated by referring to the temporary import  tables not the existing data so whatever it is doing it is something in the actual import file and not because you importing the data for a second time. Both John  and I have tried to duplicate the problem but with no success. I have imported 83.000 records  with as many permutations I can think of, then changed this in various ways and reimported it, but without managing to generate any duplicates. I have also run a number of specific tests  containing different fields. The only way which I can find of getting near  to creating this problem is to have duplicate  MapMate keys in the import file.  We do need to get to the bottom of this and are hoping that you can supply a  file which is actually generating duplicates so we can investigate. It probably makes no difference, but do you import from Excel or though a link with an Access table ?  If you use a link to Access  is it possible that you are using a query which is duplicating the MapMate keys in some circumstances.   


Mike

.

Mike Weideli

7

Re: Importing MapMate records

Mike,

I have been importing from MapMate via TAB delimited text files, I'm wary of using Excel, had too many occasions when Excel thought it knew better and corrupted dates.

I'm not convinced supplying an export file will help, since it needs the original data to be present to cause the problem - I tried re-entering a record and redetermining it - it did not trigger the bug!

Craig Slawson
Staffordshire Ecological Record

8

Re: Importing MapMate records

Craig

Thanks. I will give  a TAB delimited file a try. The problem we have is that as far as  John can see the code is not looking at the original data when it generates the keys, so we are looking for something in the actual file to generate the duplicated record. We are a bit stuck without being able to duplicate the problem.

Has anyone else experience of this  please.  It can be recognised by the 9th character in  Taxon_Occurrence_Keys being numeric  (1 normally) instead of an 'E'.  For example a MapMate key of kw2o43jd should give a Recorder Taxon_Occurrence_Key  of MMEEE3JDEEEKW2O4, but  a duplicated one would be MMEEE3JD1EEKW2O4. 


Mike

Mike Weideli