1

Re: Recorder 6 import errors

Hi,

As briefly mentioned ina previous thread, we have had a strin gof problems getting R6 runing as a trial system, now on a separate machine to R2K2.

I have been trying to import NBN zipped surveys from R2K2, with no success.

When I go to the import, and select a NBN zipped file, I can proceed, and the analysis leads to for instance "1629 items, 2 duplicated, 9 invalid".  If I then click on accept import, then do import whenit is not greyed out, I get a new box with Unhandled error, repot to JNCC yes/no, and the import will not happen.  I have tried this with seveal different importa, and always get the same tyhing.

Am I naive in thinking that it should proceed to import the items which are valid?

Anyway, the invalid items come in three forms:

Survey_Event or Sample: not in bounding box of survey.    (most of our locations in R2K2 are listed as bounding box UK, not with specific inputted co-ordinated, and this is not in location field in R6 when I have created locations form scratch).

Survey_Event or sample; Spatial reference does not fall within the specified location grid squares

Taxon_Occurence : determination date cannot be before the sample date

As far as I know, as these records are in recorder2002, they should be correct wrt these things, so I really do not understand what the problems are, and why the rest of samples within an import do not go ahead.

Advice on how to proceed greatly appreciated.

Cambs & Peterborough Biological Records centre

2

Re: Recorder 6 import errors

Hi Louise

You are right in thinking that valid items should be imported at this stage, provided that these items do not rely on an invalid item. For example, if a survey event is regarded as invalid, then all samples and taxon occurrences underneath that survey event are not imported.

Is the import data coming from the most up to date version of Recorder 2002?

I can't be sure whether the problem you describe is a bug with Recorder 6 or is a problem with the import specifically.

I don't know if anyone else has any suggestions?

If no-one else on the forum has any further advice, I would recommend contacting your reseller in the first instance.

Kind regards,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

3

Re: Recorder 6 import errors

Hi all, have now discovered that if logged on to PC as administrator, it works, and I know there is a thread on here somewhere about that, so I will go away and study it before posting further requests....
Logging on to PC as me, cannot import from an excel file either.....


didn't have chance to try that one last week for various reasons.

Louise Bacon
CPBRC

4

Re: Recorder 6 import errors

Have investigated further, and it seems to be like this on our computer.

If I log onto the PC via any user with full administrator rights, I can import things into Recorder 6.  If I remove the administrator rights to the PC for that user, or log on via a user with no administrator rights, Recorder6 will not allow me to import anything, but you can open the programme, create surveys, users, locations, single records, etc, its just the import mechanism which doesn't seem to work.

We only have so far one user of R6 within the software - the default user, so there shouldn't be a conflict there.

Even when logged on as someone with administrator rights, sometimes the imports come through as empty surveys, everything being rejected during import, even when there is only a list of 10 invalid items.

Still, I feel I have made some progress - at least I have some data in R6 that I can play with.

Recorder2002 has never been on the machine on which we have R6, so I cannot blame anything left behind from it for this quirky behaviour.

So, has anyone got a solution to why we need to set administrator-level access rights to a user to allow them to import things into Recorder6?

Cambridgeshire & Peterborough Biological Records Centre

5

Re: Recorder 6 import errors

Good Morning there,

Quick question: are you using a standalone version or server?

The permission minefield regarding permissions is well documented through this site ( just do a search for 'permissions' and you'll see what I mean). We've noticed this problem & have temporarily resolved it by simply having the system manager (ie me) do all imports. As there have been a number of key concerns with common name/nameserver/dictionary by having one person do all import produces some consistency. I have a number of talented data-entry volunteers who are becoming increasingly proficient at creating/cleaning excel sheets in particular.

As to why this is the case, I can only surmise that admin rights contain some more elaborate read/write/create permissions with whichever version of server you are using.

Cheers now, Rob.

6

Re: Recorder 6 import errors

Hi

I tried to start a new post about this but it has disappeared into the ether so I have to type it all again. Not over happy about that.

I am using R6.9.3 and have found as reported elsewhere that R6 is not handling names, on importing, as R2002 used to do. For example, the entry 'Joe and Enid Bloggs' is not accepted during the import process and it has to be rewritten as 'Joe Bloggs, Enid Bloggs'. Not critical, but not helpful and why the change? What purpose does it serve?

A similar problem arises with information in the Sex/Stage column. This is critical and needs to be fixed. No matter what entry is in the Term list, R6 does not accept the following:

'1st winter', '1st-winter', 'First winter', '2nd winter', etc as only word 'winter' appears in the matching process. For some birds this is important information and the sex/stage column is the place where it should be stored.

I hope that this bug can be fixed asap!

Thanks in anticipation, Ian

7

Re: Recorder 6 import errors

Hi Ian

The new system in Recorder 6 allows you to import more than one observer, but this does has the disadvantage of
causing problems when you have the old-style "Joe and Enid Bloggs".In Recorder 6 the import wizard was designed to interpret "and" as a separator used for different observers, therefore this would be considered to be expected behaviour, i.e. "Joe and Enid Bloggs" would be interpreted as "Joe" and "Enid Bloggs". In order to import both Joe and Enid Bloggs are separate individuals with full names, you would have to list them as "Joe Bloggs and Enid Bloggs" in the import file.

Similarly, for sex/stage matching, there is a whole new parsing system where you can no longer have a separate Abundance and Qualifier column - the data and the qualifier must appear in the same column in the import file. The following are some explanations of how Recorder interprets the text you have tried to use.

"1st winter" - R6 will interpret this as a measurement with a value of "1st" and a qualifier of "winter".

"1st-winter" - R6 interprets this with a value of "1st-" and a qualifier of "winter"

"First Winter" - R6 will interpret this with a value of "First" and a qualifier of "Winter".

To correctly import "1st Winter", first type in "1st Winter" as a measurement qualifier using Tools - Terms List - Measurement Type - Abundance - Mesurement Qualifier. Then in your import spreadsheet in the abundance column express it like so:

1 FirstWinter

Charles Roper has kindly provided a spreadsheet with some examples which I will post on the uploads page (the data contained within the spreadsheet isn't actual data and has been made up just to illustrate the example). The last 2 rows demonstrate the methods you can use to import this data.

Hope this helps!

Thanks Charles!

Best wishes,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

8

Re: Recorder 6 import errors

Hi

I cannot add this excel file to the uploads page - I've just realised it doesn't accept this file format.

As it is my last day in the office before christmas I'm going to email this to Ian. If anyone else would like a copy of this please let me know.

Many thanks,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

9

Re: Recorder 6 import errors

Sarah, you can zip the files and add them like that.

Update: I've just enabled uploading of spreadsheets too.

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

10

Re: Recorder 6 import errors

Hi Charles

I have added the file in zip format - didn't see your update beforehand!

Thanks again,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

11

Re: Recorder 6 import errors

Hi Sarah and Charles

Thanks for sending the spreadsheet for me to see. Its contents trouble me, more than a little. I enter all my data via spreadsheets, importing the contents to R6. With the 'sex/stage' separate from the 'abundance' it is easy to edit any errors by altering the qualifier entry and copying down (having sorted on that column). Now, if abundance and qualifier are mixed, there are many more options and editting/checking prior to import will be a far harder job. Why is this change thought to be an improvement? I can see no advantages, but lots of disadvantages.

Also, what happens to the qualifiers that were in Recorder anyway, like 'Adult female' and 'Adult male'. What happens to the space with those (I have not been able to test it yet)?

I have just looked at the data I have imported to R6 and found that for every record I have imported there is no figure for abundance - the measurement tab is blank. This means that the change to the method of import has had a huge effect and there was no warning on any of the paperwork that came with R6. I am horrified. Hours of work has been wasted and I will have to delete many records and reimport. What on earth is going on? The import process shows the abundance and qualifier fields, as before, with no error message and all along it has not been importing either. I can't believe it.

How can I get the abundance and any qualifier to show?
Why does the import data process not say that the abundance and data have to be mixed for them to be recorded (even though this is a very bad idea)?
I have my job as a recorder to do, and Recorder only seems to make the job more difficult than it has to be. I feel in utter despair.


Help....! Ian

12

Re: Recorder 6 import errors

Hi Ian

Although I wasn't around when this functionality was developed, it is my understanding that this was implemented so that users could import multiple occurrences with abundance/qualifier information within the same row, rather than having to use multiple rows for this information.

For example, in the past, two rows would have to be used to enter information such as the following:

Date           Observer            Location           Grid Ref      Species          Abundance       Qualifier

01/01/07     Sarah Shaw        Birch Wood       TQ1234      Blackbird        1                     Male

01/01/07     Sarah Shaw        Birch Wood       TQ1234      Blackbird        2                     Female

However, in Recorder 6 this can be imported as:

Date           Observer            Location           Grid Ref      Species          Abundance/Qualifier

01/01/07     Sarah Shaw        Birch Wood       TQ1234      Blackbird        1 Male; 2 Female

Also as far as I’m aware, if you have entered an occurrence of an adult female or male (which no count specified) into Recorder 2002 and have transferred that information into Recorder 6, then an abundance of 1 adult female will be displayed in the measurement tab. If you have specified a count in Recorder 2002 such as 2 Adult males, this also transfers correctly.

I am very sorry for the inconvenience this has caused you. I agree that it is essential that this is clearly documented and will ensure it is in the future. One of our priorities is to revise and improve much of the documentation such as the Help, Getting Started Guide and installation guides that accompany Recorder 6 (though I know this doesn't help your immediate situation).

If I can be of any further assistance please do not hesitate to contact me.

Best wishes,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

13

Re: Recorder 6 import errors

Hi Ian,

There are several advantages to the new system. As Sarah has pointed out, it is more efficient to have a combined abundance column. Furthermore, it is more efficient once the data are imported because, instead of importing one occurrence for each abundance type, Recorder now imports just one occurrence and includes multiple abundance types within that. This is how the data should be stored within Recorder; the old import wizard failed to make use of this ability to store multiple abundances against an occurrence and instead rather clumsily created multiple occurrences where one would have been preferable.

Another advantage is flexibility and ease of typing. For example, instead of typing 1 male; 2 female, you could instead type 1m 2f, which is much quicker and less prone to error. At the abundance qualifier matching stage, Recorder will ask you what you meant by "m" and "f" and at this stage you can match it to the Male and Female qualifiers that exist in the database already. What's more, Recorder will remember your choices, so that you don't have to specify them every time. The same goes for things like AdultFemale - because Recorder can't find "AdultFemale" in its database, it will ask you what you meant by that. You can then match "AdultFemale" to the "Adult Female" (with space) that already exists in the database and Recorder will remember that choice for next time. It's a system that works really well once you've tried it a few times.

In addition, you can have one abundance type per column. Have a look at Sheet2 in the TestImport spreadsheet for an example of this. You can also mix and match these styles, so you could have separate columns for the more common abundance types, then have a combined column for more unusual ones.

I feel your pain with the changes, though. I have always wished that backward compatibility had been maintained as our simple data entry tool (Species Recorder) relied on the old style abundances, even though doing it the old way is far from ideal. As I understand it, though, the new import wizard was a complete redesign from the ground up done by Dorset Software, rather than an upgrade or progression of the old wizard, which was built by Stuart Ball (I think), thus although they look and act in similar ways, they are, for better or worse, completely different beasts. Having said all that, it turned out to be very easy to fix what was broken in the spreadsheets generated by Species Recorder and so all is now well here at least.

But you have highlighted a very important issue regarding backward compatibility: it's important to maintain it where possible and where is has to be broken, users should be given plenty of advanced warning and consulted on the changes, otherwise processes and tools that have been developed over time will just keep getting broken, which is not only annoying but also can be very expensive!

Charles

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

14

Re: Recorder 6 import errors

Hi Charles and Sarah

I must admit that I was nearly in tears, the evening before Christmas when I discovered that the 3000+ imports had not abundance data with them. It is not good enough to say that we ought to have been informed - the fact is that hours of my own time (I am a bird recorder as a volunteer) have been wasted, with the prospect of many further hours being wasted to try to amend the situation. Currently, I cannot think of any way of correcting the errors as a batch.

I totally disagree with your assertions about the combination of different types of data in the one field. As a bird recorder I am receiving records from a range of people who are more, or less, fastidious about how they enter their data. I have to do a lot of editting to tidy the data before importing. To combine abundance and qualifiers in the one field makes that job much worse as it vastly increases the number of possible combinations in a sort in Excel and therefore makes for more work (for me). I think this sounds like a change to suit the software developers and nothing to do with the users. Considering that for a recorder like me, the import wizard is the key interface that I have with R6, to have that changed without any notification feels very bad. I know that I am a recent user of Recorder (only for the last 18 months) but have not seen any reference, anywhere to this radical change in the importing of data. I am even more annoyed that the revised import wizard gives no warning message about the abundance and qualifier being in separate fields (though that is how they are shown in Data Entry, Observations!).

It does seem that R6 is not a program for single taxon recorders like me. The time that it takes, the huge dictionaries of taxa in which I have no interest, etc, make the program so unwieldy. For people who are paid to work with it and have to record data about lots of taxa, then I can see it must be useful. If the data entry system for the import wizard had not changed I would probably have continued with it, but I do not relish the hours of my time that it would take to correct what is now in R6.

I still feel considerable despair and disappointment. I think that I must find a working alternative, somehow. Having put so many hours into finding how to use R2002 and then all the difficulty migrating to R6 to stop using Recorder will be a big decision....

Ian

15

Re: Recorder 6 import errors

Hi Ian, the changes to the import wizard are in there at the request of many users. People have asked again and again (myself included) for the ability to import multiple abundances for each occurrence and also the ability to import multiple Recorders; the fact is that it is much more efficient in terms of data entry and data storage to be able to express multiple abundances in one row of a spreadsheet rather than repeating a whole record (row) multiple times. I was, however, disappointed that backward compatibility was broken when the new import wizard appeared as it can be very difficult to transition to a new way of doing things, even if the new way is an 'improvement'. But software changes and improves (and sometimes regresses) and so I cannot emphasise enough how important it is to test software before committing to it and using it in a 'live' situation. Always, always, do some small test imports when you get a new version of Recorder. I tested Recorder 6 for about 6 months before committing to it, so pretty much all of the changes, bumps and issues were fully ironed out before we made the switch to make it our live database.

Did you have a look at the alternative way of expressing abundances in sheet2 of the testimport spreadsheet? Would this be useful to you?

The question as to whether R6 is a good program for single taxon recorders is a tricky one. As you are aware, different recorders are fastidious in how they record, and different groups tend to have their own requirements too. The end results is that Recorder tries to be so many things to so many people and thus end up being rather complex and unwieldy (although it has improved immensely in the 3 years I've been using it). Creating software that is both robust and flexible enough for scientists while at the same time is also easy and loose enough to be used by non-scientists is incredibly difficult. I tend to find, working in a record centre as I do, that having multiple taxon groups to deal with makes life more complex, not less. So it's actually much easier if all you're dealing with is one group. Ultimately, thought, I do feel that Recorder isn't a 'light' piece of software - it takes a large investment in time and energy to get the best out of it. Given that investment, one can get a huge amount from it; much more so than any other similar system. As with all things it has its strengths and weaknesses, but unfortunately its strengths can often only be realised by the truly dedicated or those with plenty of help from their reseller. We here wouldn't have been able to use Recorder in the way we use it if it weren't for the phenomenal amount of help we've received from Mike Weideli and Sally Rankin.

Answers to the problems are difficult. Software alone is not the answer, although projects like Recorder Web may go a some way to alleviating some of the problems: being able to deliver tailored recording solutions to recorders, with the only requirement being a web browser and an internet connection is fairly compelling. No one piece of software is going to be ideal for everyone. The best thing to do is to get involved in the community and make your voice heard (as have have been doing). It is only with plenty of feedback that Recorder can improve.

Charles

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

16

Re: Recorder 6 import errors

If I can just reiterate what Charles says above, the changes to the Import Wizard were at the request of the users, not the software developers. In fact considerable effort was required to write parsers that could split the abundance data up in a single field into its appropriate units. Also, in Sarah's example of a Recorder 2002 import with 2 abundance records there is a problem with the interpretation of this. How does this system know whether this is to be recorded as 1 occurrence with both males and females present, or 2 separate occurrences? I know its possible to make an assumption that 2 records entered with identical data except for the abundance data are actually the same observation, but if Recorder makes assumptions like that then there is bound to be a case where the assumption is incorrect! So finding a method of importing abundance data in a single row was necessary.

John van Breda
Biodiverse IT