1 Topic by iain

  • iain
  • Veteran
  • Offline
  • Registered: 19-01-2009
  • Posts: 91

Topic: Bulk Import

Hi

I've just spend around 1 day waiting for an import of around 20000 occurrences using "index.php/services/data/save" to copy records individually over a LAN. Is there a better (faster) way to bulk update the database (perhaps I can update numerous records at once using a JSON submission). Currently it takes around 4 seconds per record for saving, although this does include checking for the existence, or otherwise, of a user ID against recorder name before writing. Any thoughts or suggestions on how to improve performance?

Many thanks

Iain

2 Reply by Jim Bacon

  • Jim Bacon
  • Veteran
  • Offline
  • From: BRC
  • Registered: 03-11-2008
  • Posts: 980

Re: Bulk Import

I use the CSV upload at the bottom of the Entered Data > Occurrences page.
I have uploaded 80000 records over probably half a day by this means.
I had to split up the file in to smaller ones to conform with PHP file upload limitations (although I might equally have changed the php.ini settings)

Jim Bacon.

3 Reply by iain

  • iain
  • Veteran
  • Offline
  • Registered: 19-01-2009
  • Posts: 91

Re: Bulk Import

Hi Jim

Thanks for the response - I assume the csv import still uses web services as it's communication mechanism - do you know if it's still using the save service to do the actually delivery of data to the database or is there some magical import service that I don't know about?

Ta

Iain