1

Re: Error contributing to recording scheme (2)

Don't know if this is a problem or impatience! :)

I'm trying to send all our Lepidoptera records to the local recorder and twice now I've run the export, only to come in the following morning to find Recorder apparently frozen with no apparent CPU usage according to the Windows Task Manager

My question is should this export take in excess of 16 hours?
Database 1.3 million records
Lepidoptera records c200,000 records
processors        : 4x Intel(R) Core(TM)2 Quad CPU    Q8300  @ 2.50GHz
physical memory   : 2048 MB

Couldn't tell how far through it was second time as the Recorder window had gone blank (but certainly past Performing external validation checks), but first time it was at 'Collecting data: Name_Relation'

Craig Slawson
Staffordshire Ecological Record

2

Re: Error contributing to recording scheme (2)

Have just tried the contribute to recording scheme again on a different dataset - c12,000 records - this ran through in c 10mins, so the Lepidoptera looks even more like a problem!

I did try the Lepidoptera export again this morning, but had to stop it after 5.5 hours

Craig Slawson
Staffordshire Ecological Record

3

Re: Error contributing to recording scheme (2)

I set off an export of 300,000 records at 10am  this morning. Looks like after 8 hours it is still running, but nowhere near complete. Will let you know what happens.

Mike

Mike Weideli

4

Re: Error contributing to recording scheme (2)

Craig

It was still running after 16 hours, but  collapsed overnight. Could have been automatic virus checks or windows updates, or just that the machine ran out of resources. I will try as few more experiments.

Mike

Mike Weideli

5

Re: Error contributing to recording scheme (2)

I ran mine over the weekend and (woohoo!) came in this morning and it was stuck in the trying to e-mail (the machine I'm using doesn't have MAPI installed), so it was just impatience.....

BUT .....

The resultant NBN Zip files was only 3k in size, and when I unzipped the database it was totally empty! So.. complete waste of time and I've got to start again!

I do wonder if this problem was because I had already run an export earlier in the day which had created an NBN zip files of EXACTLY the same name - could the fact this already existed have cause the problem - if not this is a serious problem, because setting up a Recording Scheme is quite likely to have in excess of 180,000 records

SECOND PROBLEM

Now Recorder "thinks" it has successfully sent ALL the Lepidoptera records to the Recording Scheme, a repeat of the export results in 'no new records' so again an empty export!

Craig Slawson
Staffordshire Ecological Record

6

Re: Error contributing to recording scheme (2)

Craig

I have manged 200,000 records, but not 300,000,  I assume there is some resources issues here, because 200,000 records is taking several times as long as 100,000.

This definitely needs looking at more closely. For the moment can you break down the first transfer to the scheme by Survey using a filter on Surveys or Survey Tags ?  Thereafter you can use the scheme to export changed records.  I am unhappy about the fact that you can't  alter the date in a Recording scheme export. within Recorder as there can be all sorts of reasons why a Recording Scheme export needs to be re-run from a specific date.  It can be changed in the  table. There should also be an option to turn off the email facility.

Mike

Mike Weideli

7

Re: Error contributing to recording scheme (2)

It was impatience ;)

The export has finally worked (I had to use the ordinary export using the same filter as the Recording Scheme)

It took 33 hours and the resultant database contained 152,000 taxon occurrences.

The time taken for exports is not linear - I also exported ONE survey which contained 180,000 records and this completed in less than 8 hours, whereas the above included records from any survey.

One worry, is that this still appears to be a lower number of records than I expected:

1) Exported 152,000 taxon occurrences
2) A basic report on ALL Species on the LEPIDOPTERA preferred checklist reports 273,000 taxon occurrences
3) A basic report on the Recorder 3 checklist where taxon="Lepidoptera" reports 276,116 taxon occurrences
4) Surely (1) and (4) should produce the same result since the export filter, I used filter on Taxonomic Group="Lepidoptera" - I can't understand the discrepancy

Considering the time taken to create the taxon dictionary this discrepancy is unbelievable - how can I promote using Recorder when it cannot make such a simple request reliably -   really ALL three requests above should produce the same result, since all I want is a list of butterflies and moths

Craig Slawson
Staffordshire Ecological Record

8

Re: Error contributing to recording scheme (2)

Craig

Generally speaking the allocation of taxa the R6 taxon group is reliable, so this would suggest that something else is going wrong.  It will only get fixed if the reason can be identified. I will try to duplicate the problem on a reasonably large dataset to see if I can identify why it has happened. If it only occurs with  very large numbers of records, then I may not find it, so any help you can give in identifying why it has happened would be appreciated.

Mike

Mike Weideli

9

Re: Error contributing to recording scheme (2)

Craig

A quick attempt at this shows that I was wrong to think that a filter on 'Taxonomic Group'  meant on  the Recorder Taxonomic Group when what is actually means is the taxonomic hierarchy.  I will do some more work to discover eaxctly what is happening.

Mike

Mike Weideli

10

Re: Error contributing to recording scheme (2)

Craig

I think I know what is happening here. The query generated by the filter, does not expand the hierarchy across the Dictionaries.  You need  to have one entry in the filter for each dictionary.  You can do this, but it is far from intuitive and not very user friendly.  To do this click the + sign for the Taxon Group Filter the search for the  taxon (eg. Lepidoptera) with the 'unrestricted ' dictionary option ticked. This will list all the available dictionaries with Lepidoptera in the hierarchy. Select one of the dictionaries and click OK. Repeat this process for every dictionary you are likely to have used. When the query is generated for the filter it will expand out the hiercarchy for all the selected dictionaries. While this seems a bit complicated and far from perfect, it does allow the flexibility to use several  taxa for the top level of the query and mutiple ones if necessary, so it is probably a good thing considering the inconsistency of the top levels of the dictionary lists.



Mike

Mike Weideli

11

Re: Error contributing to recording scheme (2)

Hi Craig and Mike
You are correct, the code for export filters by taxonomic group does not use the Full Name Translation option properly, so although it does expand across lists it uses the old technique of doing this and therefore does not work in many cases.
I have fixed this in the code so it will work correctly in the next release of Recorder. I also changed the "Taxon Groups" box on the Export Filters dialog to read "Higher Taxa" which more accurately reflects what it does.
Best Wishes

John van Breda
Biodiverse IT

12

Re: Error contributing to recording scheme (2)

Mike,

will your 'fix' get all the Lepidoptera listed on the "Additional Names" list which doesn't have any hierarchy?

Craig Slawson
Staffordshire Ecological Record

13

Re: Error contributing to recording scheme (2)

Does this 'fix' mean I will have problems exporting all the birds entered using the Recorder 3 list - because "Aves" does not occur on the Recorder 3 list, will I need to enter ALL the bird orders?

I'm am beginning to despair about the taxonomy side of Recorder, there seem to be SOOOOOO many inconsistencies

Craig Slawson
Staffordshire Ecological Record

14

Re: Error contributing to recording scheme (2)

Craig

This fix doesn't solve the Dictionary problems, because different lists have different hierarchies. On  postive note there is a lot of work going on behind the scenes  to try and fix the issues.

Mike

Mike Weideli