1

Re: Import thousands of records via import wizard

The import wizard at version 692 (I would have tried it on a 693 but my fresh install and upgrade over night on a laptop crashed with an error so I gave up on that), seems to only be able to import from a tab/csv etc about 4456 records.

In our data exchange with BMERC I am looking to import over 600,000 records in this format from an export from MapMate. Now, this equates to over 150 imports. Matching/adding: people, locations, species etc on my test import of the first batch took a while.

So, has anyone ever tried to import such a large amount of data from another system, like MapMate, using the import wizard, and been successful? Or have you done it another way, or not?

Would love to hear from you :)

Thanks

Brian

Brian Miller
(Conservation Officer (Buckinghamshire), BBOWT)

2

Re: Import thousands of records via import wizard

Yes and no. I've imported 700,000 records from Excel spreadsheets, but I found that a sensible maximum number of rows to import at one time was around the 10,000 mark, so I broke down that import into 10K blocks. Also, don't forget that Excel can only handle 65K rows.

When importing from an Access database (same spreadsheet tables, only imported into Access) using the ADO connection option, then I've found I can import in excess of 25,000 at a time. I'm sure it could handle more, but the problem lies in the matching stages. If you have too many species to match, too many locations etc. then the stability of the import wizard becomes a factor. You could spend all day matching species and locations only to have the import wizard crash, losing you the matching work you've done.

So the bottom line is, do large imports using Access if you can. Mike Weideli has also mentioned to me in the past that building a separate utility for importing large quantities of data would be relatively simple. Such a thing wouldn't be as flexible as the import wizard though.

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

3

Re: Import thousands of records via import wizard

Thanks for the reply Charles, will look into some of these ways today. Tried the other day though to import from Access but must have missed an option on how to do this. Will look again.

Cheers

Brian

Brian Miller
(Conservation Officer (Buckinghamshire), BBOWT)