1 Topic by DaveSlade

  • DaveSlade
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 332

Topic: Record Limit

Is there a known record limit before RecordCleaner throws the 'out of memory' exception, or is this a system limit rather than the software?

I've been trying to validate and verify a large dataset before sending it up to Gateway but have had to split it up into chunks to get it through. It would save me some time in future if I knew I had to limit chunks to 250,000 records.

Dave

Dave Slade
Senior IT & Records Officer,
15 Talbot Road, Talbot Green, Pontyclun, CF72 8AD
www.sewbrec.org.uk, www.sewbrecord.org.uk

2 Reply by Graham French

  • Graham French
  • Administrator
  • Offline
  • Registered: 23-05-2006
  • Posts: 430

Re: Record Limit

Hello Dave

When the NBN Record Cleaner was developed it was designed to handle up to 500,000 records, though this is not a fixed limit. With larger datasets the mapping part in the verification stage is likely to struggle.

Best wishes

Graham

Graham French's Website

3 Reply by DaveSlade

  • DaveSlade
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 332

Re: Record Limit

Hmm, I'm getting memory errors with c250,000 records (NBN Exchange format) when it gets to the mapping stage. It is going to make my 750,000 tricky to validate!

Dave Slade
Senior IT & Records Officer,
15 Talbot Road, Talbot Green, Pontyclun, CF72 8AD
www.sewbrec.org.uk, www.sewbrecord.org.uk