Hi Ian

Duplicate records, as far as the import process is concerned, are explained as follows:
1) Each record, on entry, is allocated a unique NBN key. This key includes the site identifier so never clashes with another record in the same table.
2) If you export this record to other datasets via zipped Access or NBN data formats, the other dataset will also hold a copy of this same record.
3) If the record is imported into a dataset which already holds a copy of a record with the same key then it is identified as a duplicate. This could occur if you export the record to someone and they export it back to you, for example.

Recorder does not identify duplicates on import for 2 reasons, firstly that its not clear exactly what constitutes a duplicate record in terms of the data (John Smith could record a blackbird in Poole on a given day, if there are 2 records with this information how do you determine if its genuine or a duplicate?). The other reason is that scanning an entire import dataset for this sort of duplication would be quite intensive and would slow down imports significantly.

An option would be for an XML report to be written that lists, in order, potential duplicate records based on a simple rule (any occurrences of the same taxon at the same location on the same day, for example). It wouldn't be perfect but it might help.

Kind Regards

Dave, maybe I have it wrong but I think that the question would refer to the import of a record that is already in the database rather than 2 identical records in the same import.

In the second case you could do your own checking within the file to be imported, although sequential ordering could not be assumed it should be possible to rearrange the data to do this, but potentially a large job. In the first case it would be an immense job to check the records in the importing file against the Recorder database.

A tool like John mentions would presumably find duplicates that had arrived by either of these routes, or any other possible route, and would be extremely useful as I am sure our dataset contains duplicate records but do not know or where or how many there are. I can see a situation of regularly running it to ensure no duplication has crept in. Should this be put in as a feature request?

Gordon

Hi Gordon,

In the case of duplicate data in the input, I agree that in most cases it can be seen and removed beforehand (using sotware tools) and this is what I check for with new data. Some formats make that simpler, e.g.flatform text file, spreadsheet etc. If it is in NBN zipped, then the work is a little harder as one has to open that up and join the tables to examine for duplicates.

Even if duplicates are removed from the input, then we could still have records in that data which are already in Recorder.
During the import process the import data is put into a temp DB. Unless I've got this all wrong, then at this point - given the underlying Recorder data model - the next step is to copy that data into the model. It is a at this stage that I would argue that the data could be checked for duplication.

Yes, a very usefull tool indeed. But rather than a report, it would be of more use if it actually took action to rectify the situation. Also, if it can find duplicate data that exists, then it could also find it before new data is appended. In a post import process what should be done with duplicates?

For two identical records that exist in seperate surveys the question is, which is the correct survey for that record? [1] Once a choce is made to keep the data in a particular survey, should the duplciate data to be removed and dumped out to a seperate DB.(i.e. NBN zipped)?

Cheers.

[1] I've assumed that the model, for the example here, starts at the survey event level.

Hi

I'm working through my duplicate data and deleting those that are lower down the list on the screen (so presumably entered later) and when I press the delete key to delete samples, I frequently get errors, as below:

Exception occurred in application Recorder 6 at 29/06/2007 08:49:36.
Version : 6.10.4.120

Exception path:
EAccessViolation : Access violation at address 00565FC6 in module 'RecorderApp.exe'. Read of address 00000061

Last event\actions:
  TfrmSampleDetails created
  TfrmSampleDetails destroyed
  TfrmEventDetails created
  TfrmEventDetails destroyed
  TfrmTaxonOccurrences created
  TfrmTaxonOccurrences destroyed
  TfrmEventDetails created
  TfrmEventDetails destroyed
  TfrmTaxonOccurrences created

Operating System : Windows XP  5.01.2600  Service Pack 2
Physical Memory available : 2,095,196 KB

DLLs loaded:
  advapi32.dll (5.1.2600.2180)
  comctl32.dll (5.82.2900.2982)
  comdlg32.dll (6.0.2900.2180)
  gdi32.dll (5.1.2600.3099)
  HHCtrl.ocx (5.2.3790.2847)
  kernel32.dll (5.1.2600.3119)
  mpr.dll (5.1.2600.2180)
  MS5.Dll (5.0.0.12)
  MS5User.Dll (5.0.0.4)
  odbc32.dll (3.525.1117.0)
  ole32.dll (5.1.2600.2726)
  oleaut32.dll (5.1.2600.2180)
  shell32.dll (6.0.2900.3051)
  user32.dll (5.1.2600.3099)
  version.dll (5.1.2600.2180)
  winmm.dll (5.1.2600.2180)
  winspool.drv (5.1.2600.2180)

Information has been saved to the file C:\Program Files\Recorder 6\LastError.txt

I don't understand all that is said here, but wonder why R6 produces errors when a delete is required. Is this okay? Or is there a glitch that needs to be sorted?

Cheers, Ian