1

Re: Serious NBN Exporter add-in bug?

We have just noticed what appears to be a very serious bug in the NBN exporter add in.

Having sent in our latest submission recently, Lynne Healy returned a short list of records from one of the datasets for which the grid reference precision value did not match the actual precision of the grid reference. Close  inspection revealed that the grid reference as exported included spaces so instead of SU2456 it had SU 24 56. This is not in itself a problem with the addin, rather some of our data has invalid dates which Recorder has somehow allowed and revalidation has never identified. As it happens the records were all historic data imported in bulk from our old Recorder 3.3. It would appear that Rec6 validation was not working very well when this data went in. That again is not the bug though.

I have not checked all the faulty records yet, but one in particular drew my attention when I inspected it in recorder because the sample spatial reference was in fact correct, but the survey event spatial reference had spaces in. There is the bug. It would appear that the NBN exporter has taken the grid reference for the record from survey event not sample. Now the sample spatial reference must always be the occurrence spatial reference (by definition) but there is no reason why survey event and sample should have the same grid reference of necessity.

It seems likely then (unless I am missing something) that the exporter always takes the survey event grid reference to use as occurrence grid reference. In most cases this will make no difference, but in many (potentially thousands) of instances the data being submitted to the gateway may have incorrect spatial references. Anyone who puts data on the gateway (via the exporter addin)  where there are multiple samples within an event should probably check whether this affects them.

During the same process Lynne also identified a small number of records (again imported from Rec3.3) where the value in vague date start was in fact larger than that in vague date end. Recorder 6 reports these dates (possibly correctly, but equally possibly incorrectly) without noticing the error. I guess this means that there is an assumption in the coding that this will never occur. In addition revalidating the data has never identified that these records were problematic. So there is also a bug in the validation routines because they clearly do not check that vague date start, vague date end and vague date type are consistent.

An example of this is a record which R6 reports as having the date "Winter 1996", where VDS is 01/12/1996, VDE is 28/02/1996 and VDT is "P" (=season). Apart from highlighting the error and the bug this also begs the question what exactly is meant by Winter 1996? Is is winter 1995-6, or winter 1996-7. My inclination is to assume the former, since most of that winter is in 1996, but what does Recorder assume?

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

2

Re: Serious NBN Exporter add-in bug?

A number of the issues you raise are known problems in Recorder 6 databases. Mike Weideli and I have written various xml reports and batch updates to enable users to identify and correct many of them. To check whether your system contains any, run System Reports – Information – Sy06 - Problems(Summary) via Reports – Run. This is available in later versions of Recorder 6. It often reports thousands of problems many of which can be corrected by batch updates. This report and batch updates to help correct some of the problems are in Problems V2.zip which can be downloaded from http://forums.nbn.org.uk/uploads.php . To help users find it, I uploaded it on 6/05/11. The documentation in Problems V2.zip is designed to help users with correcting the problems found. It also specifies how the problems arose where this is known. Running the batch update DATA_Correction 1 will correct several types of errors very quickly, including removing the spaces in grid references. Often the spaces can be seen in Management Studio but not in the Recorder front end.

Sy06 etc. is designed to pick up problems not found by revalidating data so users are advised to revalidate data and run Sy06 etc. It would be a good idea to do this before doing exports. System Managers can revalidate a whole database using Tools – Database Tools – Revalidate Database. When I did this on a database containing 517,000 species observations it took 25 minutes.

The Winter dates problem comes from Recorder 3 to Recorder 2002/Recorder 6 data transfers. It will not be a problem in transfers that Mike and I did as we picked it up fairly early on and corrected the data in Recorder 2002/Recorder 6. We had to go back to Recorder 3 to find out what the dates should have been then correct them in the transferred data.

The problems not corrected by the above will need further investigation.

Sally Rankin, JNCC Recorder Approved Expert
E-mail: s.rankin@btinternet.com
Telephone: 01491 578633
Mobile: 07941 207687

3

Re: Serious NBN Exporter add-in bug?

Thanks Sally, I will pass that on & get back to you.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre