1 (edited by RobLarge 29-03-2010 15:51:21)

Re: Another import problem

Here's a puzzling one...

We have added to the Measurement Type term list, a number of categories of observed evidence of the presence of various species and groups. This seemed to be the most appropriate way to record such data consistently. However we are now having a rather odd problem which is preventing import of some data.

The data in question is records of birds, with which we are using the BTO's codes for breeding status evidence (listed below). To contain the data I have added a new category to the term list which I have called Observation (it has been through a variety of incarnations but that wording seems to make sense when reporting on the data), which contains a single Measurement Qualifier called Evidence and a number of measurement units, each with lists of allowed values. One of these measurement units is Bird breeding which has allowed values as follows (in bold text) Of course only the letters in bold appear in the list of allowed values, the key below is copied from the text description in the term list.

X = Breeding behaviour not indicated or recorded by observer

Non-breeding = Nature of evidence not specified
F = Flying over
M = Migrating
U = SUmmering non-breeder

Possible breeding = Recorder did not supply evidence.
H = Observed in suitable Habitat
S = Singing male

Probable breeding = Recorder did not supply evidence
P = Pair in suitable nesting habitat
T = Permanent Territory (defended over at least 1 week)
D = Courtship and Display
N = Visiting probable Nest site
A = Agitated behaviour
I = Brood patch of Incubating bird (from bird in hand)
B = Nest Building or excavating nest-hole

Confirmed breeding = Recorder did not supply evidence
DD = Distraction Display or injury feigning
UN = Used Nest or eggshells found from this season
FL = Recently FLedged young or downy young
ON = Adults entering or leaving nest-site in circumstances which indicate an Occupied Nest.
FF = Adult carrying Faecal sac or Food for young
NE = Nest containing Eggs
NY = Nest with Young seen or heard

The data we are trying to import has just one of these codes for every record in a field called Breeding evidence. At the column selection stage we set this column as Taxon occurrence data/Observation of Evidence (Bird breeding)

After column selection, when the Validate records button is pressed a number of records are highlighted red in the breeding evidence column. Oddly however (and this is a behaviour I have witnessed before at this stage of imports, but cannot explain) as the cursor passes over each of the highlighted records, it disappears, until all are gone and there are no errors left. Even after the errors have gone though, the Next button is still unavailable and the text on the left of the window still indicates that there are errors to be corrected, but does not say what they are, so we are unable to procede with the import.

There is an interesting clue as to what the problem might be, because the items highlighted as errors all contain either D or F and you will notice that the list of allowed values also includes DD and FF.  Could it be that the importer is getting confused somehow?

I could try modifying the list of allowed values, but since these are the ones BTO recorders use I would rather not.

Any suggestions?

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

2

Re: Another import problem

Hi Rob,

Do you have data in your system that is already related to these terms that you added? If so is it possible for you to export it so that I can import the terms into my system. I could then try and import your data to try and get to the bottom of this (or pass it to Dorset to investigate).

Best Wishes,
Lynn

3

Re: Another import problem

I'll see what I can do, but it will be after easter now I would imagine

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

4

Re: Another import problem

Hi Rob,

Would you be able to send me some data from your system as identified by Lynn above, then can test and/or send on to Dorset.

Many thanks, Mary

5

Re: Another import problem

Hello,

Whe have exactly the same problem as Rob when trying to import presence/absence data with allowed values p/a (odd cursor behaviour, no possibilties to continue the import procedure after errors have been 'solved' by passing with the cursor). Has there been any progress on this?

Many thanks,

Frederic

6

Re: Another import problem

Sorry I haven't got round to exporting the data yet, things have been rather busy. I can confirm however that when we changed the allowed values (which appeared to be causing the problem) to something else (D to CD and F to FO), the import worked normally, so the importer is having trouble telling the difference between D and DD somehow.

I could still do an export of the appropriate data, but it will no longer exhibit the problem as it has been imported with the modified values.

Mary, you should be able to reproduce the problem with a much reduced set of items in the term list, following my description above (I think you would only need the items D and DD in the list) and this would probably be quicker than me sending you an export file, but let me know how you want to progress this.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

7

Re: Another import problem

Hi Rob,

I will follow this up at our end. No need for an export.

Many thanks,
Mary

8

Re: Another import problem

Just wanted to bump this one up again by saying that we are now getting the same problem with our extended DAFOR data.

Again we have DAFORs set up as measurements and have entered a huge amount of data using this, but on import we get the same situation as above, rows marked as errors then disappearing as the cursor passes over them and no next button available.

The extended DAFOR set includes terms like LA, VLA, AD etc. and it seems that the import wizard doesn't like these much

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

9

Re: Another import problem

Thanks Rob and sorry for the delay - the bug is still in my list to test. I will prioritise it and get back to you if I have any queries.

10

Re: Another import problem

Hi Rob,

Quick update:

On my system, all DAFOR data can be successfully imported in 6.14.5. However values 'd','f' and 'a' cannot be matched against a TOcc_BreedingBirds list. This is odd behaviour as involves matching the same values, just in different lists. I can confirm the equally odd behaviour of the flagged invalid cells changing from red to white when the cursor is passed over them, although this still does not allow you to proceed through the wizard.

This needs further investigation to work out where the error is occurring and whether it can be fixed as an incident or is a CCN. Any more information or related behaviour would be appreciated and we will look into this further.

Best wishes,
Mary

11

Re: Another import problem

Thanks Mary,

That is an interesting idea, that there is some kind of cross contamination between lists. Looking at the table Measurement_unit_value, which contains all the allowed values for lists there are only two relevant fields, Measurement_Unit_Key which identifies the list and Data which contains all the allowed values.

My initial thought was that the importer is checking for matching Data values, but not paying attention to the Key values, so is recieving multiple matches. The fact that you are able to match DAFORS but not the bird data suggests that where the first (or maybe the last) matching item in the list has the correct unit key then the match is allowed, but if not it fails. If that is the case it should be an easy fix (he says optimistically)

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

12

Re: Another import problem

OK, bumping this one up the list.

We are still having problems with this. One of our staff is now unable to import data with the DAFOR measurement A. All other data in the set was importable and she whittled the import down until only the A's remains, which she had to input manually.

This is tedious, any chance of a fix?

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

13

Re: Another import problem

Rob

The error we know about is where the EXCEL column letter is the same as the entry in the column.  Was this the data 'A' in column A of the spreadsheet - this seems unlikely so perhaps the problem is slightly different. If it was then moving the column to a column where none of the entries match the column heading will solve the problem or you can  save the file as a csv file and import that.   

Mike

Mike Weideli

14

Re: Another import problem

Bumping this one back up again with the addition of some useful observations here

http://forums.nbn.org.uk/viewtopic.php?pid=8025#p8025

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

15

Re: Another import problem

Bumping again, can anyone tell me if this bug ever got fixed?

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre