1 Topic by AndyFoy

  • AndyFoy
  • Consultant
  • Offline
  • Registered: 09-08-2006
  • Posts: 219

Topic: NBN Exchange v6.23

Hi

I've just run a report using the report wizard to query all records for a list of species in a rucksack.  Then I exported these records (30,875) using the NBN Exchange Format v6.23.  I got the warning "These are invalid records in the data you are about to export. These records will not be exported."  I then selected the option in the NBN Exchange Format interface to 'Exclude' records failing validation.  The output .txt file contained 30,851 records (so I assumed 24 records were invalid in some way).

Just to check what the invalid records were I re-exported the same 30,875 records but this time selected the option in the NBN Exchange Format addin to 'Include' records failing validation, and instead of getting 30,875 records I got 71,055 records.  And when I compare the two outputs (30,851 vs 71,055) I only get 663 records that are in common.  So there seems to be something wrong with the addin ... when trying to 'include' invalid records it is actually exporting almost a completely different set of records.

What is also odd is that, if I re-run the original report wizard but with the <invalid state="1"/> option (to include all invalid records) I get 30,877 records (instead of 30,875), suggesting there are 2 invalid records using the same rucksack.  So why, when I export the records using the NBN Exchange Format do I only have 30,851 records in the output.  What happened to the other 24 records?  How can I found out what they are?

I'm a bit confused and concerned.  Can anyone help?

Thanks
Andy

Andy Foy
Systems Manager
Greenspace Information for Greater London (GiGL) CIC
www.gigl.org.uk

2 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: NBN Exchange v6.23

Will take a look. Not sure what is causing this or how big a problem it will be to fix.

Mike Weideli

MikeWeideli's Website

3 Reply by AndyFoy

  • AndyFoy
  • Consultant
  • Offline
  • Registered: 09-08-2006
  • Posts: 219

Re: NBN Exchange v6.23

Hi

On closer inspection it appears that, when including invalid records in the export, the addin appears to be doing something like including all invalid records for all surveys in the report, not just those records in the report that are invalid.  In other words it seems to be using the SurveyKey from the report to find all invalid records, not the RecordKey.  Is this what it is supposed to be doing?

Andy

Andy Foy
Systems Manager
Greenspace Information for Greater London (GiGL) CIC
www.gigl.org.uk

4 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: NBN Exchange v6.23

It should just pick up the invalid records included in the report.   It could be that R6 itself is at fault here and not the addin.  Are you using v6.23 of R6. There were some major changes in here around exporting and may have introduced a problem  Invalid records in this context means records which fail database validation. (ie the ones you see if you right click on a survey, events etc.  and choose Revalidate Record). Things like the date of the  Event being after the date of the Sample. If you revalidate the whole database how many records is it reporting as invalid ? (Tools/Database Tools/Revalidate Database).

Mike Weideli

MikeWeideli's Website

5 Reply by AndyFoy

  • AndyFoy
  • Consultant
  • Offline
  • Registered: 09-08-2006
  • Posts: 219

Re: NBN Exchange v6.23

Hi Mike

I'm using v6.23 of the addin with v6.22 of R6 because I wanted the new features in the latest version of the addin.  Is this okay or is the addin relying on v6.23 of R6 to work?

I haven't revalidate the whole database yet but, because the export is based on a rucksack of specific species, the total number of invalid records may be irrelevant.

I don't think there is a user function to validate records is there?  Are you able to supply a copy of the code (even if it is in Delphi) or even just a list of the steps involved so I can write a SQL user function?

Thanks
Andy

Andy Foy
Systems Manager
Greenspace Information for Greater London (GiGL) CIC
www.gigl.org.uk

6 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: NBN Exchange v6.23

I think the 6.23 addin will work with the 6.22 database, but may not be fully functional as one of the stored procedures it uses has changed.  I can't think of a reason why this could cause the problem you are having, but it might be.

When you have run a report you can go to Database Tools and there is an option to revalidate records. I think this is validating the records in the report, but it gives a message which I don't fully understand.  From within R6 you can validate at any level Survey. Event, Sample, Occurrence simply by right clicking on revalidate.  A database should have very few invalid records as under normal circumstances R6 will make sure they don't go in. Changing the Records Allowed from and  Records Allowed to on a Survey is the only way you can get  large numbers of errors.

The code for doing the validation is in R6 and it is complicated -  picking it out from delphi isn't really practical.

Validating the full database shouldn't take that long. The method I use after doing R3 transfers etc, is to validate each Survey in turn.

Mike Weideli

MikeWeideli's Website

7 Reply by MikeWeideli

  • MikeWeideli
  • Administrator
  • Offline
  • From: Littlefield Consultancy
  • Registered: 05-07-2006
  • Posts: 2,645

Re: NBN Exchange v6.23

I have been running further tests on this and cant duplicate the problem in a 6.23 database, bur I am only using limited test data.  The very large number of  invalid records you are reporting looks spurious and as a first step in further investigation it would be helpful to confirm this by running the validation check on the whole database.

Mike Weideli

MikeWeideli's Website