1 Reply by Ian M Spence

  • Ian M Spence
  • Veteran
  • Offline
  • From: North east Wales
  • Registered: 30-06-2006
  • Posts: 193

Re: Error with locations and grid references on import

Hi

I have imported about 100,000 records for use in our local bird atlas project. When I was preparing the maps in DMAP I found that dots for four locations were appearing in the wrong places - obviously so - in the sea and S of our southern boundary.

When I checked to locate the source of the problem I was surprised by what I found. In the Data Entry, Locations, the grid references were correct for each tetrad.  The errors were found in Observations, choosing the relevant survey and checking the grid reference against each location. In three cases SJ was stored as SH and in one a grid ref of 9933 had become 9923. I couldn't understand how this had happened.

When importing, the data file has to match all the data held in R6, eg the observer names, the location names and their grid references, etc. So, I can't see how the incorrect grid refs could have been imported. If the errors were not imported how and why should R6 make some unwanted changes to only a few grid refs (but there could be many more, but they don't show up as they are located within the area of our map).

When data were imported, I had assumed that when there was a match with the eg location data held in R6 that the data from the Location files would be used to store against each observation.  Is this assumption correct?

If the assumption is  not correct, does this mean that errors could creep through on import. When the Import Wizard shows matches for all locations I do not visually check that they are correct. This should not be necessary, but if errors happen on import it will have to become another part of the process.

Can anyone explain:

- how the errors occurred?
- how to prevent them occurring again?

I hope I have explained the problem well enough (and there were no error messages that appeared - from an R6 point of view, all had worked well) and look forward to hearing from you.

Thanks, Ian

2 Reply by sallyrankin

  • sallyrankin
  • Administrator
  • Offline
  • Registered: 07-04-2006
  • Posts: 371

Re: Error with locations and grid references on import

Validating the grid references on records for locations in the location hierarchy is done by checking that they are within the Grid Squares on the locations – see the Geo Info tab in the location hierarchy where F1 will bring up the Help. If there are no grid squares this check can’t be done. A word of warning: if data is being imported by the import wizard and a grid reference falls outside the grid squares for its location, the record will be flagged as an invalid item so it won’t be imported, but the error message doesn’t tell you what the offending grid reference is. Hence it is very difficult to track down the problem in your import file unless you are able to use the temporary database – see Contents tab in the Help – Tasks – Exchanging data – Import data – Invalid Items.

The inclusion of the offending data in the error message for invalid items is a long standing enhancement request. An extension of this is listed as ‘Facility to export invalid rows to Excel with an information column to indicate the issue’ at the end of the document referred to in Lynn’s post on the Forum 24/11/09 http://forums.nbn.org.uk/viewtopic.php?pid=5401#p5401 detailing proposed enhancements that JNCC requested feedback on.

Sally Rankin, JNCC Recorder Approved Expert
E-mail: s.rankin@btinternet.com
Telephone: 01491 578633
Mobile: 07941 207687

3 Reply by Ian M Spence

  • Ian M Spence
  • Veteran
  • Offline
  • From: North east Wales
  • Registered: 30-06-2006
  • Posts: 193

Re: Error with locations and grid references on import

Hi Sally

Thanks for your reply. I'm not sure that it helps me as during the imports I always check any records flagged as 'invalid' and there have been none. I would certainly welcome some more helpful messages in relatively plain English with the invalid records when they do occur.

I'm not sure that your reply answers either of my questions... Any clues?

All the best, Ian

4 Reply by johnvanbreda

  • johnvanbreda
  • Administrator
  • Offline
  • From: UK
  • Registered: 27-04-2006
  • Posts: 1,237

Re: Error with locations and grid references on import

Hi Ian
From what I understand, you have a location with a grid reference attached, and you have imported data from a spreadsheet which has been linked to that location. What you have found is that the samples now how incorrect grid references, but you had expected the Import Wizard would have told you they were invalid.
What Sally said in her first paragraph could well be the problem - there is no validation against the centroid grid reference for a location, because with only this information the validation library does not know the shape of the site. For example a river may have a centroid many miles from the occurrences in the river. So, if you are importing data against locations where the imported data has not got the correct grid reference, Recorder cannot warn you that the data is wrong. The only time it will do this is if you have populated the location's grid squares tab with the list of grid squares that cover the site - in which case you get invalid records if the record spatial references do not fall in one of the location's grid squares.

Does that make sense?

John van Breda
Biodiverse IT

johnvanbreda's Website