1 Topic by CharlieDarling

  • CharlieDarling
  • Member
  • Offline
  • Registered: 13-01-2012
  • Posts: 43

Topic: Record Cleaner Update

Graham,

First the good news. The Hoverfly and Spider rules are now working properly. Also unassessed records are not being exported. However, after running a set of invertebrate records which included hoverfly and spiders, I found that in the passed records exported there were duplicates. Out of 1663 records there were 1049 duplicates leaving only 614 unique passed records. In the case of the duplicated records instead of 1 entry per Record Key there were 3. On inspecting some of these in Recorder, I found that those that I looked at had more than one entry for Determination Type having already been passed through the Record Cleaner. Some have 2 entries, some have 3. I don't know if this has any significance.

Charlie

Biodiversity Data Assistant
Scottish Wildlife Trust

2 Reply by Graham French

  • Graham French
  • Administrator
  • Offline
  • Registered: 23-05-2006
  • Posts: 430

Re: Record Cleaner Update

Hello Charlie

The export functionality was rewritten to get round a limitation in the tool that was causing the issue with exporting unassessed records. The export is now done in a series of batches which may be causing the duplications. Is it possible to send me the records that are causing these duplications and I will have a look into it further

Best wishes

Graham
NBN Technical Liaison Officer

Graham French's Website

3 Reply by CharlieDarling

  • CharlieDarling
  • Member
  • Offline
  • Registered: 13-01-2012
  • Posts: 43

Re: Record Cleaner Update

Graham,

I just updated the Record Cleaner and tried rerunning the set of invertebrate records which I sent to you on 28/02/13.
Again, there were duplicates in the passed verification output. I then ran each set of rules one at a time.
The Odonata rules and Hoverfly rules produced no duplicates. The Spider rules produced 3 entries in the
output for each record which passed. The Lepidoptera rules produced 3 entries for some passed records and
only 2 for others. Examining the latter, I found that Vanessa atalanta, Vanessa cardui and Neozephyrus quercushad entries in the flight period and period rules but not in the tenkm rules. It would seem that for the Spider and
Lepidoptera rules the Record Cleaner is producing an output in the passed records for each of the rules which
is passed.

Charlie

Biodiversity Data Assistant
Scottish Wildlife Trust

4 Reply by Graham French

  • Graham French
  • Administrator
  • Offline
  • Registered: 23-05-2006
  • Posts: 430

Re: Record Cleaner Update

Hello Charlie

The latest update brings the species list up to date and does not update other areas of the NBN Record Cleaner.

It is difficult getting to the bottom of what is going on when exporting records that have passed verification, it seems to work when a few rules are run for example Odonata and Hoverfly) but as you say breaks down for taxa when larger rule sets are run (eg Lepidoptera and Spiders).

The NBN Record Cleaner is primarily intended to highlight records that "fail" the rules so that further investigation can be targeted towards these records as to whether they are correct or not. The number of these are likely to be much less than the number of records that pass the rules and I suspect this may turn out to be a limitation of the application when exporting passed records generated from running a large number of rules. I need to investigate this a bit further and apologize for the time this is taking

Best wishes

Graham
NBN Technical Liaison Officer

Graham French's Website

5 Reply by Graham French

  • Graham French
  • Administrator
  • Offline
  • Registered: 23-05-2006
  • Posts: 430

Re: Record Cleaner Update

Hello Charlie

A new version of the NBN Record Cleaner (V1.0.8.5) has been released which should fix the above issue of duplicated passed records being exported when running multiple rules

Best wishes

Graham
NBN Technical Liaison Officer

Graham French's Website