1 Topic by ser

  • ser
  • ser
  • Administrator
  • Offline
  • From: Staffordshire Ecological Recor
  • Registered: 24-09-2008
  • Posts: 603

Topic: error in import: automatic creation of individuals

I have just run an import where I let the import automatically create the individuals. It didn't do exactly what I expected!

The import has created c1000 duplicated individuals! On checking, I have found that the import created two people for each record, one for the recorder/observer and one for the determiner, even when the two were identical

A secondary problem, is how can I combine all these duplicates, it is not feasible to do it individually as the merge routine takes over a minute per merge, so over 16hrs of continuous editing, with the inherent chances of making a mistake!

Craig

Craig Slawson
Staffordshire Ecological Record

ser's Website

2 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: error in import: automatic creation of individuals

Never heard of this happening before, and I can't duplicate it. The only thing I can think of is that it hasn't coped with the very large number of new records . Do you know if it showed duplicates at the point where you selected automatic create new entries ?

However, it has happened for whatever reason so it needs to be put right.  First, question is did you save the filter file at the end of the export ?  If you did this will make the correction a lot easier.

Mike Weideli

MikeWeideli's Website

3 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: error in import: automatic creation of individuals

I attach an XML Report which will identify duplicates and provide the information necessary to remove the unwanted ones.  The output from this report should be saved as an Excel file. Any lines which are not considered duplicates should be removed. This will then provide the input of a Batch Update which will remove the unwanted entries.

Post's attachments

JNCC_Pr10_Duplicated_Individuals.xml 3.04 kb, 3 downloads since 2015-09-22 

You don't have the permssions to download the attachments of this post.
Mike Weideli

MikeWeideli's Website

4 Reply by ser

  • ser
  • ser
  • Administrator
  • Offline
  • From: Staffordshire Ecological Recor
  • Registered: 24-09-2008
  • Posts: 603

Re: error in import: automatic creation of individuals

Mike,

I have already extracted the offending GUIs, what I don't know is which of the pair is the observer and which is the determiner

Secondly, no I didn't save the filter, there were 35,000 records (not elements) in the import, hence the large number of new people - in fact the creation of all the people was almost a month ago, on the first import attempt which failed (some edits to the sites was necessary) - I only successfully ran the import last week, and it was after this that I discovered the problem - the duplicates were easy to find because they were all created in a 6 minute window on one day, hence my belief that they were created by the automatic routine.

I will try the batch update tomorrow after backing up the database

Craig

Craig Slawson
Staffordshire Ecological Record

ser's Website

5 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: error in import: automatic creation of individuals

I havnt done the batch update yet. What I have provided is a report which will identify the duplicated individuals. It should be easy enough with this information to do a batch update to update both the recorder and deterrminer and remove the unwanted indiduals.  The report will identify all duplicates which could include ones which have occurred at other times. The batch update would fix these as well, but if you dont want this to happen then the list of the taxon occurrence keys involved needs to be taken into consideration.

Mike Weideli

MikeWeideli's Website

6 Reply by charliebarnes

  • charliebarnes
  • Veteran
  • Offline
  • From: Greater Lincolnshire
  • Registered: 07-09-2008
  • Posts: 1,396

Re: error in import: automatic creation of individuals

I think this may have just happened to me as well.

In a batch of 12000 records from Excel, Smith, A. exists only with one record - as Observer and Determiner, however I now have two A. Smith's in Recorder, one created a minute after the other. I suspect it has something to do with having both Observer and Determiner fields being set? I haven't (knowingly) encountered it before, but we usually don't import Determiners, using "use first Observer as determiner".

I.e. Recorder appears to be create new individuals for both the Observer and Determiner.

Craig - it would be interesting to know if you also imported the Determiner names?

Charlie Barnes
Information Officer
Greater Lincolnshire Nature Partnership

charliebarnes's Website

7 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: error in import: automatic creation of individuals

I can duplicate this now, but it only where the name in the 'Smith, A', format (may be some others as well). Certainly, using A. Smith is OK. On the matching sceen where Smith, A is immported as both the recorder and the determiner the Recorder will be shown as A. Smith and the Determiner as Smith, A. Recorder creates two different name here, but both for A. Smith and uses these for the Recorder and Determiner respectively. I thought this might be a new problem introduced in 6.23, but it isn't. I will try to get this fixed asap. 

Graig - does this fit with your problem

Mike Weideli

MikeWeideli's Website

8 Reply by ser

  • ser
  • ser
  • Administrator
  • Offline
  • From: Staffordshire Ecological Recor
  • Registered: 24-09-2008
  • Posts: 603

Re: error in import: automatic creation of individuals

Yes to both questions, The bulk of the records were "surname, forename/inits" and I was importing a determiner field as well as the observer field

Craig

Craig Slawson
Staffordshire Ecological Record

ser's Website

9 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: error in import: automatic creation of individuals

Hi

I have investigated this and perhaps surprisingly the parsing of Determiner Name  has never worked in the same way as the Observers. It was never designed  to do so. At the stage where the names are displayed the Determiner name and also when new names are created  it is just text in the format it was in the import file. 

I am not sure how difficult this is going to be to fix, but it becomes development rather than a bug fix.

Mike Weideli

MikeWeideli's Website

10 Reply by charliebarnes

  • charliebarnes
  • Veteran
  • Offline
  • From: Greater Lincolnshire
  • Registered: 07-09-2008
  • Posts: 1,396

Re: error in import: automatic creation of individuals

I would say that's a bug.....

So the 'workaround' is to be very careful when importing determiner names which might be new to the system?

Charlie Barnes
Information Officer
Greater Lincolnshire Nature Partnership

charliebarnes's Website

11 Reply by stevemcbill

  • stevemcbill
  • Veteran
  • Offline
  • From: Runcorn, Cheshire
  • Registered: 21-11-2006
  • Posts: 803

Re: error in import: automatic creation of individuals

I must admit Mike that I agree with Charlie - to me that IS a bug and one that should be addressed asap.

Steve

Steve J. McWilliam
www.rECOrd-LRC.co.uk
www.stevemcwilliam.co.uk/guitar/

stevemcbill's Website

12 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: error in import: automatic creation of individuals

I take your point, but it doesn't look like something which can be fixed quickly or cheaply.

Mike Weideli

MikeWeideli's Website

13 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: error in import: automatic creation of individuals

This issue will be fixed in version 6.25 which will be available shortly as a Release Candidate.

Mike Weideli

MikeWeideli's Website

14 Reply by stevemcbill

  • stevemcbill
  • Veteran
  • Offline
  • From: Runcorn, Cheshire
  • Registered: 21-11-2006
  • Posts: 803

Re: error in import: automatic creation of individuals

Excellent news Mike - thank you.

Steve

Steve J. McWilliam
www.rECOrd-LRC.co.uk
www.stevemcwilliam.co.uk/guitar/

stevemcbill's Website