1

Topic: Crashes Exporting All records

I am trying to export all records using a filter for All Records in NBN data format, and destination the Recorder 6 folder. After waiting something like 5 hours or so, Recorder 6 reported that it was "Out of Memory while expanding memory stream"
with the options of restart program, exit, etc.
An output file was created of 138,245 kBytes in size

Recorder 6 version 6.24.0.280, with dictionary version 3B, running under Windows 8.1 64 bit, and using SQL Express 2008. All latest Microsoft patches applied.

Database contains:
685 Locations
6,675 Names & addresses
134 References
144 Surveys
708.400 Species observations
7,744 Habitat observation

Error report attached

Post's attachments

Error-73601174737264.txt 28.51 kb, 1 downloads since 2016-02-18 

You don't have the permssions to download the attachments of this post.
Harry Clarke
Surrey County Butterfly Recorder

2

Re: Crashes Exporting All records

This is going to be tricky to find. Nothing in the  report is specific enough to identify where the problem is. I would need to data in order to investigate this. Could be something in the data or some change in the tables which has caused a bug in the NBN data format export. Is there is a specific reason why you have used the NBN data format and not the NBN Access Data based zipped format. I can't guarantee that the zipped Access method will work, but it is totally different process  and has been updated and tested more recently, so it may get around the problem. If it works the data should  be just as easy to import into another copy of R6 unless it is an earlier version than 6.23.

Mike Weideli

3

Re: Crashes Exporting All records

There is nothing in the data that I can think of that would cause a problem.
I will try the NBN Access Data based zipped format and see how that behaves.

Harry Clarke
Surrey County Butterfly Recorder

4

Re: Crashes Exporting All records

Nearly 24 hours later the query is still running.
Recorder is stating that "gathering information for export - Reference Author", with no progress bar shown.
Task manger is showing that SQL Server Windows NT (32 Bit) is using about 31% CPU, and with memory slowly growing, currently sitting at 1001.5 MB (last night is was 980 MB).
There doesn't seem to be any disc activity from SQL Server.
Does this provide any clues about what is going on? Or what could be investigated further to get to the root of the problem?

Harry Clarke
Surrey County Butterfly Recorder

5

Re: Crashes Exporting All records

The zipped Access export was tested for 6.23  with some very large live datasets (certainly much  larger than  your database)  and nothing took as long as 24 hours, so this suggest that the process is stuck somewhere. I am going to run a further test with a 1million or so record database to see if I can duplicate the problem.

Mike Weideli

6

Re: Crashes Exporting All records

The Zipped Access export finally completed, creating the Zipped data file.
Given the time it has taken, could there be something in the data that is causing the problem?

Harry Clarke
Surrey County Butterfly Recorder

7

Re: Crashes Exporting All records

Could be something which is not working efficiently.  How much memory does your machine have and is it a desktop or a laptop ?

Mike Weideli

8

Re: Crashes Exporting All records

My laptop has 8GB of memory, and has solid state storage, with dual processors.
I couldn't see anything obvious that would cause the PC to run slowly, there were plenty of free resources, when it was running the export.

I have now imported the records into an empty copy of Recorder 6, which it has eventually completed.
The import reported 6 duplicate records, one Individual and five Names, which I decided to Import all records.

Comparing the Summary Information between the two copies I have noticed the following differences
1) 134 References in original copy, 119 in destination copy
2) 144 Surveys in original copy, 142 Surveys in destination copy.
3) Tags for all the Surveys  has been lost. I had forgotten to enable this option before the import. Are these the missing references?

Other than Tags, what else has not been copied?

Harry Clarke
Surrey County Butterfly Recorder

9

Re: Crashes Exporting All records

Hi

Certainly your machine seem to be up to the task, but an export of this size is going to involve many very complex queries and will in effect create a temporary table of equal size  to the original database, which is going to use a lot of resource.  Just because you are exporting all records it doesn't stop the process going through the very complex routines to work out what needs to be included.

As I don't know you data or your starting point I can't tell if there is a problem or not.  I can see why perhaps some 'references' may not have been exported. however, I am concerned that not all Surveys have gone over. If you start from a list of Surveys  defined say in the Export filter then they should all be included.  If you start by specify a list of species or sample then the Surveys will only get included if they actually have Taxon or  Biotope Occurrences. Is there anything about the missing Surveys which might account for them being excluded ?   


The aim of the export is share Observations between R6 systems. It is designed only  to include from the other tables the minimum required to give a complete structure.  The process is only exporting things like references, measurements etc. if they are actual used by the other data being exported. The process works through the database structure in the appropriate order working out from what has been extracted already what is needed in the next level. So for example if a Reference has not been used then it will not get extracted. This could be the reason why you are mssing some.  Similarly if you have entries in other tables, which are not actually used then they will not be exported.   

I don't think Survey tags are exported. The tag structure is something controlled by the receiving system and exporting these would mess up the structure in the receiving system. 

A number of things affect what is exported.

1. Confidential records are only exported if the Option is set to export Confidential.
2. Unchecked records are not exported
3. Unless Tools/Export Management is used  Failed/or Pending
Verification records (based on the Determination Type) will be excluded.
4. Records in Surveys with a Media type of Temporary will never be exported.
5. Records which fail the External validation checks will not export through XML Export, and will only export
through the Access zipped method if  'Do not validate' is chosen during 'Data Export' '     
6. Data in the two 'Private' fields on samples is not exported. 

   
If we can't explain the missing data then I will need a copy of your database to investigate.

Mike Weideli

10

Re: Crashes Exporting All records

Hi
On checking the differences I have found the following so far.

1) One empty Survey was not copied.
2) Some Documents not referenced where not copied.
3) I had created a new Measurement Unit which was copied, except for its Allowed Values.
4) A new Measurement Type had been created which was not copied. I suspect it was not used anywhere.
5) The login user has been changed. This explains the duplicated Individual during the import. How do I correct this?
Presumably the ID of the login user needs to be changed.

Harry Clarke
Surrey County Butterfly Recorder

11

Re: Crashes Exporting All records

Hi

Login as the default user (password will be 'password'). Go into Tools/User configuration  and remove the User set up with the new copy of R6 (ie not the imported one). You should now be able to delete this user from data entry Name/Addresses. Set up the imported person as  a user through Tools/User configuration. You should now be able to login as this user.

Mike Weideli

12

Re: Crashes Exporting All records

Many thanks Mike

Harry Clarke
Surrey County Butterfly Recorder