1

Re: Important - Import Wizard Bug

The following bug has been observed in Recorder 6 - when using the import wizard, if you have two or more rows that include different location names but the same spatial reference, the import wizard assigns all taxon occurrences to the same event/sample with location/location name recorded as the first location specified. This occurs both when you match the entries to existing locations and if you specify for these entries to be created as new locations. If you do select to create new locations, new locations appear in the location hierarchy, but only the first location specified is recorded against.

For example, if you import several records that have the following attributes:

Location              Grid
Aaa                  TQ1234
Bbb                  TQ1234

All of the records will be linked to the Aaa location only, even though you have matched the Bbb records correctly during the location matching step or you have selected to create two new locations.

This will be fixed in the next version of Recorder 6, however for the time being users need to be wary of this bug when importing data.

Best wishes,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

2

Re: Important - Import Wizard Bug

Sarah

I get very confused with the Location and Location Name fields.  I'm hoping that the bug you mentioned only applies to the Location field (we have many examples where different Location Names have been used within the same kilometre squares). Is that the case?

Cheers

Bob

Bob Saville
LWIC coordinator

3

Re: Important - Import Wizard Bug

Bob,

It's probably worth testing to see if this bug is evident when using only location names. The severity of this bug shouldn't be underestimated. I'm having to remove and re-import (when the fix is released) over 700,000 records because we can't be sure which have and haven't been affected by the bug. Just to re-iterate the problem, if you're importing records with one grid reference (say, a 10KM reference) but multiple locations, then only the first location matched with be imported, even though the import wizard does actually ask you for the matches at the location matching stage.

So, check you data!

Charles

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

4

Re: Important - Import Wizard Bug

Thanks for that Charles

I'm still a bit confused though! You mention that only the first location matched will be imported. Surely it is only possible to match Locations. Location Names are simple text fields which can't go through any location matching stage so does that mean there isn't going to be a problem with them?

Cheers

Bob

Bob Saville
LWIC coordinator

5

Re: Important - Import Wizard Bug

Hi Bob and Charles

I've just quickly checked this with V6.9.3 and this only appears to be a problem when you try to match entries to actual locations within R6.

If I import data with different location names but with the same grid reference, and at the 'Location Specification' stage of the import wizard select 'Place any remaining unmatched locations in the location name', then the resultant records are not attached to an actual location and have the correct grid reference and 'location name' at the survey event and sample level.

Kind regards,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

6 (edited by brianmiller 10-01-2007 15:28:13)

Re: Important - Import Wizard Bug

Hi,

Seeing as I was the one who highlighted this originally I thought I would check it out now I have the latest version installed as of yesterday.

Sarah, I have just imported four records with three different location names (not in our location table) but all with the same grid ref. They import under the survey all under the first location name, in the event and sample level.

Brian

Brian Miller
(Conservation Officer (Buckinghamshire), BBOWT)

7

Re: Important - Import Wizard Bug

Ah, so it was you who submitted that report. By pure coincidence, we found it here independently, although Sarah mentioned someone else had reported it when I submitted my bug report. Thing is, in your incident report, it mentions that the problem only occurs with newly created locations; I have found that it doesn't matter whether the locations are new or not - the problem occurs either way.

Charles

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

8

Re: Important - Import Wizard Bug

Agreed.

Brian

Brian Miller
(Conservation Officer (Buckinghamshire), BBOWT)

9

Re: Important - Import Wizard Bug

Hi Brian

This problem was raised after V6.9.3 was developed therefore it is still a problem at the moment, but we hope to have it fixed as soon as we can.

In the post above, I just wanted to add that (from my investigations anyway) there doesn't appear to be a problem with the import wizard if you select to store location information under location name (instead of matching entries to actual locations or choosing to create new locations).

If possible, can you confirm that this is the case with your system?

Many thanks,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

10

Re: Important - Import Wizard Bug

Just an aside!

Is it just me or does it seem a bit crazy to talk about location names (i.e names of Locations) and location names (i.e. Location Names). Is there no way to go back to using the word Site instead of Location and end the confusion?

Bob

Bob Saville
LWIC coordinator

11

Re: Important - Import Wizard Bug

I agree, it's one of the more confusing aspects of Recorder. I think we're pretty much stuck with that terminology though I would imagine.

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

12

Re: Important - Import Wizard Bug

Sarah Shaw wrote:

This problem was raised after V6.9.3 was developed therefore it is still a problem at the moment, but we hope to have it fixed as soon as we can.

In the post above, I just wanted to add that (from my investigations anyway) there doesn't appear to be a problem with the import wizard if you select to store location information under location name (instead of matching entries to actual locations or choosing to create new locations).
JNCC

Hi Sarah,

I thought it was a bit strange that you found it would work now in 6.9.3. when you said it would be fixed in the next version...

Having tested both ways, firstly without the locations in the location table and then adding them to the location table the problem is the same on our system adding them to the first location in the imported list, be it a location or merely location_name.

I have tested adding the other two locations only to the location table then importing and get the same result with the records associated to the location that is not in the location table and not the others which are.

Brian

Brian Miller
(Conservation Officer (Buckinghamshire), BBOWT)

13

Re: Important - Import Wizard Bug

Hi Brian and Charles

I'll post the incident form on the uploads page. If you have a spare minute I'd much appreciate it if you could have a look at it just to ensure I haven't missed anything off.

The 'location' and 'location name' terminology does make these things difficult to word! However, I think a change in the terminology would be fairly low on the priority list at this stage.

Please let me know if I need to add anything more.

Thanks again for your help,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

14

Re: Important - Import Wizard Bug

Hi Sarah,

Looks very comprehensive and covers most things except that with my testing the problem arises also when choosing to just assign the records to the location name field, whether or not any of the other locations are matched to locations in the Location Table.

Brian

Brian Miller
(Conservation Officer (Buckinghamshire), BBOWT)

15

Re: Important - Import Wizard Bug

Hi Brian

My apologies - I can now reproduce this too. The key thing appears to be dates. If the entries in the import file have the same date, then the occurrences all go in with the first entry's location details (saved in the 'location name' field), even if all entries have been left unmatched. If the entries have different dates then Recorder seems to behave correctly.

Let me know if you spot anything else.

Many thanks,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

16

Re: Important - Import Wizard Bug

Hi

With a little bit of help from my LRC I have managed to import some data to R6 with abundance and qualifier information. The vital bit of information that was new to me was concatenating in Excel. This will allow me to gather data with abundance and qualifier in separate fields, that make editting and correction a lot easier, then they can be combined (=A1&" "&B1) if the data are in cells A1 and B1. It is a bit messy as the result needs to be copied and Paste Special with 'values and number format' to retain the information when the original abundance and qualifier columns are deleted. So far so good!

However, the Import Data process has an unwanted habit that I cannot see how to alter. Any number without a qualifier, gets called 'adult'. With birds that is not correct and I wish to alter that setting so that when there are no qualifiers, there are no qualifiers added on import. Does anyone know how to alter that default in the import process? If so, please tell!

I have found that deleting unwanted records produces a lot of errors. I ought to copy one here, but there are far more errors than there ought to be.

Cheers, Ian

17

Re: Important - Import Wizard Bug

Hi

I have been experimenting so that I can import data until the bug of having every abundance without a qualifier called 'adult' is fixed. I have managed to find a way of entering 'fullgrown' - by editting the term list - and I see there are many other qualifiers that will need similar editting (why was that not done as part of the upgrade to R6 if the import wizard was altered at that time?). I have noticed inconsistencies that I do not understand:

The BOU checklist for 2006 is there with most of the new names and new scientific names, but, on import when the Species in the to be imported list and the Recorder list are compared, we get:

Blue Tit / Blue Tit (Parus caeruleus (Linnaeus, 1758) / BIRDS British Ornithologist Union Bird List (Categories A, B, C)

so why is the old 'Parus' name shown and not the new 'Cyanistes' that is in Recorder - in the list I had chosen for the import process?

Also, in the list of Taxon occurrences Blue Tit appears as 'Blue Tit (Parus caeruleus)'.

If there is a bird list available, why does is it not used, consistently, across the different aspects of R6?

Cheers, Ian

18

Re: Important - Import Wizard Bug

Hi

I spent about 6 hours yesterday deleting records with no abundance and have been reimporting. Today I have found a problem with one of the imports. The Excel file does not look any different to others I have successfully imported from.

I wanted to inspect the imported data and went to: Data Entry, Observations, (Name of my survey) and the list of Events comes up. I choose the event I wish to examine and click on the + in a box and sample shows with a + in a box. I click on that + and it disappears and no species names appear as part of the sample. This has occurred to all the samples from this particular file.

I thought I would delete the samples, etc and reimport but I cannot as R6 says that the 'item is referenced by other records in the database'. If so, where are they and how can I see them? I tried to right click to see what options there were. Trying 'Quick Report, Species seen by an observer' an empty table appeared saying '9 occurrences' and trying 'Quick Report, Species reports, Detailed species list' shows nothing (ie no occurrences).

Does anyone have any idea why the import has not pulled through the data that are in the Excel sheet?
Does anyone know how I may delete the offending Events to try to import them and have data visible?

[In addition to my post immediately before this one, my example of Blue Tit was unfortunate in that the common name has not changed. The common name is not that from the BOU checklist 2006, as eg 'Great Cormorant' appears as 'Cormorant'.]

I look forward to hearing from someone!

Thanks in anticipation, Ian

19

Re: Important - Import Wizard Bug

If you can't delete a sample my guess is that it contains taxon occurrence(s) that can't be displayed because there is a problem with their taxon list item (species) keys. The show metadata icon will tell you the key of the sample and with that, if you have Access, or other tools that allow you to look at the database, you can look in the taxon occurrence table to see if there are any rows containing the key of the offending sample.  If you are going to try deleting these, back up your system first, and be very careful. Do the deletions starting at the bottom of the hierarchy and work upwards, i.e. taxon occurence data entries, taxon determinations, etc. have to be deleted before the corresponding taxon occurrences can be deleted. 

If this doesn't help, I hope it gives you clues to other things you can try.

Sally Rankin, JNCC Recorder Approved Expert
E-mail: s.rankin@btinternet.com
Telephone: 01491 578633
Mobile: 07941 207687

20

Re: Important - Import Wizard Bug

Hi Sally

Many thanks for your suggestion - the errors are now corrected.

I was not sure what you meant by 'the hierarchy' but looked in the tables you mentioned and could only find the sample key in the Taxon Occurrence table. I deleted those records and then went back to R6 where I was then able to delete the Sample and Events from Data Entry, Observations.

I checked the Excel file and could find nothing wrong with it, but did ensure that the 'Determiner' field had an entry (it was blank before). The second import worked as expected and I was able to see the records in Data Entry, Observations.

Again, many thanks! Ian

21

Re: Important - Import Wizard Bug

A word of warning: if you've deleted things manually from a table, make sure the entry for that table in the LAST_KEY table is up-to-date. So, in the case of the TAXON_OCCURRENCE table, look at the very last key and make sure it matches the TAXON_OCCURRENCE key in the LAST_KEY table. This is really important. If not done and the keys are out of sync, then you might start seeing odd behaviour or data corruption.

Charles

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

22

Re: Important - Import Wizard Bug

Ian,

Adult as default abundance
I agree, this is a frustrating and bad assumption to make. It's not technically a bug as the Recorder system design document specifies this behaviour. However, I do feel this is a design choice that needs to be changed. As a work-around, we add a generic qualifier of 'present' to all abundances that have no qualifier.

BOU list issues
You need to ask species dictionary questions in the Species Dictionary forum, which John Tweddle, data manager of the Species Dictionary Project, monitors. If you ask species dictionary questions in other places such as this thread, there's a strong chance he won't see it and thus your question is likely to go unanswered. This time, though, I'll need to ask a few questions to get the the bottom of your bird list problem:

The BOU checklist for 2006 is there with most of the new names and new scientific names, but, on import when the Species in the to be imported list and the Recorder list are compared, we get:

I'm not sure what you mean here; could you possibly reword this?

Blue Tit / Blue Tit (Parus caeruleus (Linnaeus, 1758) / BIRDS British Ornithologist Union Bird List (Categories A, B, C)

so why is the old 'Parus' name shown and not the new 'Cyanistes' that is in Recorder - in the list I had chosen for the import process?

Where do you see this exactly? If I import a record of a blue tit (i.e. 'blue tit' is what's in the species name column of the import spreadsheet), at the species matching stage I choose the BIRDS BOU list and click search. The species that gets matched is this:

Blue Tit (Cyanistes caeruleus (Linnaeus, 1758))

On final import, the record appears as Blue Tit (Cyanistes caeruleus) in the the Observations Hierarchy.

Finally, a word of warning about concatenating abundance qualifiers - make sure you remove spaces from the qualifiers first. So 'Adult female' would become 'Adultfemale'. The import wizard will allow you to add the space back in when you come to the qualifier matching stage. If you do not do this, an abundance of '1 Adult female' will be interpreted as two abundances: the first with a value of 1 with a qualifier of 'Adult' and the second with a value of 'female' and a qualifier of none, which as we know converts to 'Adult'; net result being two entries in your measurements tab looking something like this:

1 Adult
female Adult

Hope that's clear. It's an issue that's easily worked around - you just have to be aware of it.

Charles

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

23

Re: Important - Import Wizard Bug

Hi Charles

Thank you for your reply and the extra information regarding concatenating.

Ian's dictionary hasn't updated completely, which we were aware of but thought that it had gotten to a usable state. This is what is causing the blue tit problem.

We're working with Ian at the moment to resolve the problems.

Kind regards,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

24

Re: Important - Import Wizard Bug

Hi Charles

Many thanks for your warning about deleting fields manually. Since I posted my reply to Sally, I have imported more data, but I have checked  and found that the TAXON_CCURRENCE_KEY for the last item in TAXON_OCCURRENCE and appears in the LAST_KEY table! Relief!

As Sarah has said, she and Steve Wilkinson are helping me to find out why my dictionary upgrade did not complete, so I will not reply to the appearance of species names in the import wizard. Hopefully, all will be corrected soon.

As for concatenation - thanks for your warning, but I have found out a bit about how the new import works. I have to ensure that large numbers do not appear as, say '10,000' but as '10000'. I have modified my term lists so that the short version of the kind of qualifiers I am likely to need are now written as one word, eg 'secondsummer', so that they will not match on import if I have not ensured they are one word in the source file. Now that I have a reasonable understanding what importing does, I am delighted by concatenation as it allows me to edit qualifiers (only a small number of options) separately from the abundance entries. Putting them together just before import is very little extra work, but makes the preparation much easier (at least,  for me).

Thanks again for your advice and suggestions,

Cheers, Ian