1

Re: R6 v6.18 candidates for inclusion

The consortium (Mike Weideli, John van Breda and Sally Rankin) that JNCC have outsourced the support and maintenance of Recorder 6 to have reconsidered the plans for future releases of the system and have decided to release another version – v6.18 – before we release the version that will use the new taxon dictionary, subject to the approval of the Recorder Steering Group. Delays in the delivery of the new dictionary mean that it will be some time before we can deliver the version of Recorder 6 that will use it and we want to deliver some improvements before then.

I have compiled a list of changes and fixes to be considered for inclusion in v6.18 and we would like users’ views on prioritising them – see R6v618 candidates for inclusion.xls. The list is rather long and the funding from JNCC will only pay for a limited number of changes so we need to ensure that it is used for the maximum benefit. We are, however, open to users paying for changes they require and you will see that two changes are being funded by Hampshire BIC. Could you please tell us, say, your top five priorities and whether there are any changes or fixes you would be willing to fund in part or in full. The priorities in the Excel file are, more or less, my personal view. They need revision in view of users’ requirements and affordability.

Many ideas have been put forward for improving the import wizard, particularly to improve the import of MapMate data. The Mantis documentation doesn’t fit easily in the Excel file so it has been stored in Selected Mantis Issues 120229.doc along with a few other proposals. We would welcome users’ views on these as well. Mantis is the system we use for logging bugs and suggestions for changes.

The files are in R6v618 candidates for inclusion.zip which is available for download from http://forums.nbn.org.uk/uploads.php .

Thank you to everyone who responded to our request for statistics from their Reorder 6 systems. The total number of records was 68,243,904, nearly as many as are on the NBN Gateway, with over 20% added in 2011. 21 systems have over a million records. We hope that this will emphasise the importance of Reorder 6 for collecting and collating biological records and that it will help secure adequate funding for it. If you didn’t send in the statistics we would still be please to have them – see http://forums.nbn.org.uk/viewtopic.php?pid=10970#p10970 Please e-mail them to me at s.rankin@btinternet.com .

Sally Rankin, JNCC Recorder Approved Expert
E-mail: s.rankin@btinternet.com
Telephone: 01491 578633
Mobile: 07941 207687

2

Re: R6 v6.18 candidates for inclusion

Hi Sally,

I've just been through the list and the majority of fixes are for problems that either I'm not aware of or I simply haven't experienced. Of the items there, the ones we'd be most interested in are 134, 275, 325 and 206.

Two other feature requests that we have need of frequently:

1. A species list report type. I think the most flexible way of handling this might be to add an item to the report grid menu that would 'roll-up' the current report and present it as a species list with counts.

2. A way of reporting and exporting all records added or updated within a specified date range. We constantly need this but it remains hard to do in an easy, flexible way. We like to send county recorders and groups regular data updates that contain only those records that have changed or been added in a certain time frame. This sort of task is fundamental to the workings of LRCs and groups and yet Recorder remains rather poor at handling it.

Hope that's of help.

Charles

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

3

Re: R6 v6.18 candidates for inclusion

Charles,

on point 2, have you tried Mike's report linked in this topic http://forums.nbn.org.uk/viewtopic.php?id=1874 ? Does part of what you want and shouldn't be too hard to make the changes you need.

Gordon Barker
Biological Survey Data Manager
National Trust

4

Re: R6 v6.18 candidates for inclusion

Thanks Gordon. The problem there is that modifying an XML Report to include the data one needs isn't very easy. It's a job that I have to do and limits what the rest of the team are able to do for themselves. That's fine in many cases where bespoke reports and analysis are needed, but this is a day-to-day task that requires modification of the query in many instances.

Having said that, Mike's query acts as a great stop-gap and I suppose I could create a suite of reports for each export we do on a regular basis.

Another feature request that I'm not sure has been mentioned before: allow import / use of online maps in the mapping UI. I'm thinking of Google Maps, Open Street Map, etc. Also a nice and easy way to bring in OS Open Data basemaps. Not sure how feasible any of this is, but I thought I'd mention it. Having better access to more basemap data within Recorder would make the mapping much, much more useful.

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

5

Re: R6 v6.18 candidates for inclusion

I'll second that Charles anything that improves the mapping tool would be good for me.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

6

Re: R6 v6.18 candidates for inclusion

Not much feedback here yet. I think what is offputting perhaps is the very dry technical nature of the proposals. I had to force myself to read through them, and even then I found little of relevance to us here and little to catch the imagination.

So perhaps an executive  summary of the updates here would be good along with perhaps more ad-lib ideas on what would make Recorder better? A good way to kick-off: list the top thing you like least about Recorder. If you're feeling daring, how about the top 3 or 5 or 10? What would you like to see more of? What would you like to see done differently?

Once we have a better idea of what people like and dislike (and need!), Mike, Sally, John et al can translate this into priorities and specifications for future work.

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

7

Re: R6 v6.18 candidates for inclusion

Hi

I agree with Charles last post, I don't have the time to go through the spreadsheet or enough technical knowledge to make much sense of it.

However, here are a few existing problems which we think should be fixed:-
a) I am told that when you do an export and change custody, it sometimes fails due to a bug(?).
b) When you produce a report, you cannot set the order of the columns as you want. I understand that you used to be able to do this but the functionality disappeared after an earlier upgrade.
c) Quick reports> places for occurrences and occurrences for places reports- Sally has advised us not to use them as they are not well configured. If this is the case they should be removed, or preferably replaced with some very simple to use reports that non-experienced users can get to grips with quickly. They should have an spreadsheet output not what is currently available.

Hope this helps.

Kathryn Hewitt
Terrestrial Environmental Evidence Manager
Countryside Council for Wales

8

Re: R6 v6.18 candidates for inclusion

Hi,

The only item we have come up with is the export of results from the Report Wizard. On the occasions we produce in excess of 65,500 records, it would be nice to go direct to Excel 2010, rather than via a text file.

Cheers,

Dave

9

Re: R6 v6.18 candidates for inclusion

In roughly this order:
* 216
* 319
* 97
* 325
* 134

In addition to these, we've also got some fixes/improvements to suggest (again in roughly this order):

* Ability to use the hierarchical tag structure in the observation window
* Allowing importing of “site” records and auto-assigning grid reference from location details
* Highlight failed/pending verification, confidential and zero count records in the observation hierarchy
* Sort Measurement Qualifier drop down alphabetically

We may be able to provide funding for some of our suggestions, and possibly 216 if needed. Obviously we would need to know how much these changes would cost - even a ball park figure would do. Is it possible to put some £££ next to some of these changes (ours or those on the spreadsheet - or even past changes) ?

Charlie Barnes
Information Officer
Greater Lincolnshire Nature Partnership

10

Re: R6 v6.18 candidates for inclusion

Hi,

Like others, the majority of fixes are for problems that either I'm not aware of or we haven't experienced at HBIC.  The two items that we have provided funding for are obviously important to us but I don't feel qualified to comment on any of the others.

One major item that we would like to see developed in the future is an improvement to how locations with multiple survey_events are handled.  Like some other users of Recorder (Mike Weideli knows of some I think) we create a new "dated" location for every new survey_event as a child to the relevant location.  In other words if a location has 3 survey_events we create 3 child locations named using the location name followed by the survey_event date (e.g. "Noar Hill - 15/08/2010").  This is so that we can add a site description to these "dated" locations specific to the survey of the site on that date.  We can also attach features to each "dated" location specific to the features idenfied during each survey.

This process works well but has several downsides, not least of which are:

1. We have to manually add these "dated" child locations manually.
2. Standard reports and addins do not expect this data structure and hence don't always work (e.g. the NBN Gateway addin uses the name of the "dated" location rather than the parent location name without the date).
3. The "Last Survey Date" field for the location shows as "Site not visited" because there are no survey_events attached to the parent location.

It would be great if Recorder could be enhanced to "properly" resolve this situation so that location details relating to mutiple survey_events can be held multiple times for a location (e.g. site description, features, designations, uses, tenure, sources).

I've uploaded a screenshot to the upload area called "Example of dated locations" to show what I mean.

Thanks
Andy

Andy Foy, Ecologist & IT consultant
Andy Foy Consulting

11 (edited by RobLarge 29-03-2012 16:55:20)

Re: R6 v6.18 candidates for inclusion

I'll second that Andy. Although I have found a different solution to the problem. We place all descriptive text for the survey event in the survey event comment itself (which after all is where it belongs), but reserve the Location Comment field for a standardised site description. By which I mean all this field holds is a short (one or two sentence) objective summary description of the site (e.g. "Ancient semi-natural broadleaved woodland on a steep north facing slope.") which is suitable for export to the GIS layers we submit to partners (and this is usually all they want). This description can be changed when a new survey is done (at the operator's discretion), but a copy of the description goes into the survey event as well to keep an audit trail. In most cases the short description is found to still hold true for the new survey and will not be changed unless there has been major change (e.g. total loss of a habitat) at the site.

I have often wondered why the location comment field is so large. A detailed description needing more than say 100 words will undoubtedly include some information which is subject to change and thus should be both dated and attributable to an individual. My feeling is that any description which needs to be dated does not belong on the location page. If it is the result of a survey visit then it goes with the survey.

Of course my system needs to be maintained properly, but at present we manage this through an external program (actually an Access database). It would be nice to do it directly in Recorder.

There are number of instances where recorder lacks comment fields, some of which I have mentioned before. A good example is on the Locations Tenure tab. We use this extensively to record ownership and other contacts relating to the site and the tenure type drop down is useful, but it would be better if we could record notes about which contact is the best point of contact as well as information qualifying their relationship to a site, for example where a site has multiple owners we might say "Owns land to the west of the dividing hedge".

I am sure I have mentioned this before as well, but I have never been happy with the way access levels are managed in Recorder, there is simply not enough discretion available in the existing system. Here my best example is to do with data relating to individuals (and thus relates to tenure as well). Names and addresses are effectively sensitive information and it should be possible to exclude access to them for particular users or classes of users. In particular I am thinking of situations where volunteer users have passed out contact information without regard to the data protection issues.

Another point is that the ability to make records confidential is useful, but would be moreso if it were possible to make all records of a species confidential (which we do here by the use of a local designation and careful application of stored procedures and additional report wizard attributes, but this system is prone to errors). Equally it would be nice to make all data relating to a site confidential (site owners sometimes require this) or all records from a given recorder. I appreciate however that this change would be fairly complex to implement.

Even with the normal occurrence confidentiality, for which it is possible to set a minimum access level, it would be useful if users who do not have sufficient access were to be informed when something has been hidden from them. I have found our volunteers are often perturbed when they find a sample which apprantly has no data in it (because the only occurrences within are confidential).

I am sure I could go on, but as I have no funds to offer I think I had better stop now.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

12

Re: R6 v6.18 candidates for inclusion

One immediate improvement that springs to mind would be the ability to import more fields than currently allowed. 

For us the fact that you cannot currently import any fields from the Administrative Areas category except Vice County is a big problem, particular the local authority area.  We need to assign all our records to a district.  When we are importing data that already has this information attached it is simply lost and we have to re-enter it, which is a complete waste of time.

Suzanne

13

Re: R6 v6.18 candidates for inclusion

Hi Sally

My preferred fixes from the spreadsheet would be:
126 – import wizard validation
128 – proposed change 4 - import multiple surveys
216 – grid change on location change
134/325 – allowing use of Excel 2007/2010
335 – sort by taxonomic order (and similarly an option of sorting by Taxon_Occurrence_Key)

Other suggestions:

1. Improvements to Report Wizard
a) First screen - option to ‘Restrict report to one or more Custodians’
b) Constraints - there are times when it is useful to report on ONLY the Confidential/Zero Abundance, etc; can the screen be reconfigured to provide this option as well as the current ‘Excludes’.  Also an option to exclude ‘Not Verified’ as well as ‘Failed/Pending Verification’.
c) Select Attributes - allow output of Entry_Date, Changed_Date and ‘table’_Key for each of the 5 main tables, (currently only ‘Obs Key’ is possible).  Also allow sorting and condition selection on these fields.
d) Output of and sorting by additional Grid Square resolutions of 100m and 10m, now that more records are received with GPS readings.
e) A quicker way of altering the priority of the Sort Attribute fields.

2. Could the Report Wizard use an external filter of Taxon_Occurrence_Keys, e.g. of those saved following an import, so that they can be viewed in the grid format rather than the Observation hierarchy.

3. The number of records received from houses and gardens is increasing. In the interests of recorder confidentiality, an additional field to hold property numbers/names which would not be exportable would be useful. Road and street names would stay in the existing Location_Name field.  I have considered separating this sensitive information into a different field such as Sample_Comment but this is not a workable solution.  For properties where we receive regular records such as moth traps or bat roosts we do treat these as Locations and so they end up with a coded Preferred Name from which it is not possible to work out the actual address; but we don’t want to do this for all the one-offs.

4. Go to Key addin.  Can improvements be made so that it retains the previous request within a session; reselecting the table and typing in 16 characters each time when you probably only need to change a couple of digits gets rather tedious.

I also agree with Rob's points on setting confidentiality.

I think providing some idea of costings would be useful, I currently have no idea whether we would be in a position to fund anything we really wanted.

Thanks

Alison

Alison Stewart
Dorset Environmental Records Centre

14

Re: R6 v6.18 candidates for inclusion

I think that the way abundance data is handled needs to be rectified.  While importing it is fine, exporting it is an absolute nightmare!  You have to export several fields just to make sure you cover every possibility that has been entered and then you end up with massive file (whether GIS, Excel etc.) that has to be manually edited to get it to make any sort of sense.  Is there any way that abundance could be a single field?  Ideally still with the drop down options that would be entered into this field as well (male, female, individuals etc.), though if I had to enter this manually then it would still be a lot easier and less time-consuming than the current system.

-----------------------------------------------------------------------
Steve Hannah - GIS and Data Infrastructure Officer
The Wildlife Information Centre

15

Re: R6 v6.18 candidates for inclusion

Have you tried using ‘Obs Abundances (LC)’ in the Observation group of attributes in the report wizard? This outputs all abundance data for an observation as a comma separated string in a single column. LC is Littlefield Consultancy, Mike Weideli’s company – Mike wrote the function that provides this option. Variations on it have been produced for a few users and are relatively easy to do.

Sally Rankin, JNCC Recorder Approved Expert
E-mail: s.rankin@btinternet.com
Telephone: 01491 578633
Mobile: 07941 207687

16

Re: R6 v6.18 candidates for inclusion

I actually hadn't known about that!  I withdraw my request as this handles it just fine.

-----------------------------------------------------------------------
Steve Hannah - GIS and Data Infrastructure Officer
The Wildlife Information Centre

17

Re: R6 v6.18 candidates for inclusion

I will look through and try and make comments - is there a deadline?
There is one minor thing I find irritating which I realise won't be a priority - when exporting to Access via the export via other formats from the Report Wizard, the default name of the table is "Select". I quite often forget to change this, then of course when I get to Access and try and write a Select SQL query it throws a wobbly. Could the default be easily changed to something like "rec" or "obs"?

-----------------
Teresa Frost | Wetland Bird Survey National Organiser | BTO
Other hat  | National Forum for Biological Recording Council
(Old hats  | NBN Board, ALERC Board, CBDC, KMBRC)

18

Re: R6 v6.18 candidates for inclusion

We are hoping John will be able to start on the changes for v6.18 in the next week or so but if you can let us have your feedback soon we will do what we can to take it into consideration.

Sally Rankin, JNCC Recorder Approved Expert
E-mail: s.rankin@btinternet.com
Telephone: 01491 578633
Mobile: 07941 207687

19

Re: R6 v6.18 candidates for inclusion

Another something that we/I would like to get fixed:

See this thread from last year: http://forums.nbn.org.uk/viewtopic.php?id=2165

It would be good if these attributes could _at least_ be shown in the observation hierachy. And going even further, be pulled out using the report wizard, although for our purposes this isn't mandatory (we tend not use to the report wizard for anything mission critical).

Charlie Barnes
Information Officer
Greater Lincolnshire Nature Partnership

20

Re: R6 v6.18 candidates for inclusion

OK I have just posted in toubleshooting an item which I would consider to be my number one priority for fixing. See here http://forums.nbn.org.uk/viewtopic.php?pid=12635#p12635

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

21

Re: R6 v6.18 candidates for inclusion

When importing data directly from Recorder v3.4, free text comments in the Site_Description field end up, as I would wish, in the Description box of the General tab of the relevant Location in the Location Hierarchy.  However, it is not possible to effect this result when using the Import Wizard on Site_Description data contained in an Excel spreadsheet.  Please would you add Site/Location Description to the list of supported fields?

22

Re: R6 v6.18 candidates for inclusion

Bob, I foresee an issue with what you propose. What happens when importing data for a loaction which already exists in the db and has a description already? I can see how it would be useful to be able to import a description when Recorder is generating a new location, but would not like to see existing data overwritten or appended to automatically.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

23 (edited by Bob Merritt 24-04-2012 14:25:56)

Re: R6 v6.18 candidates for inclusion

Thanks, Rob, for your comments.  The problem you foresee would not occur if the data were imported to a new survey.  Or am I wrong in thinking this?

24

Re: R6 v6.18 candidates for inclusion

Location description was requested by Bob Marsh – see http://forums.nbn.org.uk/viewtopic.php?pid=11077#p11077 . I have added it to R6v618 candidates for inclusion.zip which is available for download – Mantis ID 344 – but it needs to be remembered that the import wizard is a tool for importing observations not locations. Is what is required a location importing tool? To import it as part of the existing wizard consideration would need to be given to handling locations with multiple descriptions in the import file, as well as the point Rob raised. How would the wizard decide which to use? Perhaps in this situation the descriptions really belong in Survey Event or Sample Comment, import column types that the wizard already handles.

Location description is stored in the location hierarchy and Recorder 6 only holds one per location so importing data into a new survey would make no difference. Using Survey Event or Sample Comment wouldn’t require a new survey either although you may choose to use one for other reasons.

Sally Rankin, JNCC Recorder Approved Expert
E-mail: s.rankin@btinternet.com
Telephone: 01491 578633
Mobile: 07941 207687

25

Re: R6 v6.18 candidates for inclusion

344 - Import wizard improvements please, particularly the ability to import using keys - taxon (TVKs?), location, individual - and an external reference.

This would be hugely beneficial for importing data from online recording facilities (particularly Rodis, now being used by multiple LRCs) but also in sharing a copy of data with another LRC etc - when you exchange data between recorder instances using spreadsheets (which is the commonest way of doing this in practice, even though disapproved of I know) it is ridiculous how much time it takes when importing to match up species names that have *just* been exported from Recorder.

-----------------
Teresa Frost | Wetland Bird Survey National Organiser | BTO
Other hat  | National Forum for Biological Recording Council
(Old hats  | NBN Board, ALERC Board, CBDC, KMBRC)