1

Topic: Exporting millions of records

Hi,

I suspect no-one else has this problem but I may be wrong? I need to export 16+ million moth records, the entire National Moth Recording Scheme database, into a single file for a data requester. The Report Wizard won't handle this much data in one go, having tried on numerous occasions. I can create numerous WZD files and run these but this is unsustainable in the long run.

What I do have is an excellent SQL Server Integration Services (SSIS) batch utility written by Dr Stuart Ball which exports the whole database in the NBN Gateway format but this is of no use to a data requester as all it exports is a table of keys. Is there such a batch utility in existence that will do a similar job but with meaningful, user-friendly data?

In the meantime, I will look at redeveloping Dr Ball's utility to see if I can get it to work as needed.

Best wishes,
Les

Les Evans-Hill
Senior Data Officer
Butterfly Conservation, Butterflies for the New Millennium and National Moth Recording Scheme

2

Re: Exporting millions of records

Hi Les,

I don't know how to resolve your problem directly. But when I took over as County Recorder for butterflies, I initially put all my records from one recording scheme into one survey. This became unmanageable, so the solution I now use is to create one Survey per year for each recording scheme. I then apply Tags to the survey so they are logically grouped together. This makes it a lot easier to find records, and see what records I have for each site, recorder etc. For example for 2013 I have five surveys covering the different recording schemes.
Using Additional Filters field enables a number of conditions to be applied, so you could generate a few reports covering all records in a particular date range, or survey name, with the aim that report one extracts say 2m records, report two for the next 2m or so records. Thus for 16m records this would require 8 reports to be run, assuming 2m was the maximum you could export at any one time.

Kind regards
Harry

Harry Clarke
Surrey County Butterfly Recorder

3

Re: Exporting millions of records

Hi Harry,

Many thanks for responding. It takes 36 hours straight (minimum) to export the database into the NBN Gateway format using SSIS. Splitting the export into multiple reports with additional filters isn't an option as this would vastly increase the amount of time to export all of the data. Potentially this export could take up to a whole working week (with idle time overnight) and the database would be unavailable for any other activity during that period which is unacceptable. For it to work it must be a single report I can just leave to run to its conclusion.

Otherwise the NMRS is absolutely fine, all surveys for info are defined by VC as it receives data from VC recorders which makes replacing VC datasets (or surveys) a breeze. This has worked well now for 7 years.

Best wishes,
Les

Les Evans-Hill
Senior Data Officer
Butterfly Conservation, Butterflies for the New Millennium and National Moth Recording Scheme

4

Re: Exporting millions of records

If the issue is not to tie up your database while exporting, why not install a second standalone copy on another machine, backup the original database & restore to the new copy? Then export from the copy, leaving your main  db free for other uses.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

5

Re: Exporting millions of records

Hi Rob,

A good idea; however, we've decided in this instance to give the data requester access to the dataset on the NBN Gateway to download what she wants from there.

Best wishes,
Les

Les Evans-Hill
Senior Data Officer
Butterfly Conservation, Butterflies for the New Millennium and National Moth Recording Scheme