26

Re: Possible enhancements to Recorder 6 - please contribute your ideas

I’ve looked through the list of enhancements proposed so far and if we had migrated from R2002 to R6 many are probably the sort of things we’d be asking for too. However we have a more basic requirement first:

1)    To be able to reliably migrate our records from R2002 to R6

(See http://forums.nbn.org.uk/viewtopic.php?id=422, http://forums.nbn.org.uk/viewtopic.php?id=410 and http://forums.nbn.org.uk/viewtopic.php?id=475 for details)

We’d then like to have back good features of R2002 that were removed, such as

2)    The Location “Report” Button,
3)    The single column for “observation measurement data” and the single column for “observation measurement qualifier” in the Report Wizard. (We don’t want 33 columns!)

4) We would also like to be able to run the network workstation on our x64 XP machine.

The next two wishes may already be in R6(?) – we haven’t got far enough with R6 yet to check…

5) The ability to simply report on lists of different status, e.g. UKBAP priority species, CROW Section 74, UK protected species, European protected species, RDB, Na, Nb, Red, Amber, etc. without LRCs having to independently maintain rucksacks of standard lists/status.

6) As we do much of our work in MapInfo, direct connection into GIS would be very helpful. (I’ve dabbled with DBMS queries). Records would either have to plot at the centre of their resolution, or be a square polygon of the resolution size. Plotting at the SW corner is of little value. (We don’t need Recorder’s mapping enhanced – we rarely use it).

7a) Hierarchies of Locations and Name/addresses get my vote too. Imports from other organizations need to be isolatable from the main database so they don’t fill it with sites outside the county, duplicates of sites within the county, or duplicate people.

7b) Related to this is the means to tell the import wizard to not match against these. We only want to match against our own site list and people list.

(When I imported some British Dragonfly Society records into R2002 I first hacked the data to prefix all the sites and surnames with “BDS-“ so that the import wizard would never match to them, and I could “isolate/distinguish” their sites and people from ours).

8) Bulk manipulation is highly desirable. e.g. I want to move a set of species from one survey to another because different access rights are needed on the NBN gateway and I want to keep Surveys and NBN datasets equivalent. Even though I can find all the records with a rucksack I can’t then bulk move them to another survey. It’s probably going to be quicker to dump them all out and re-import them.

9) Others have commented on MapMate->R6 imports. We get updates from Moth recorders in MapMate but don’t bother trying to import them into Recorder (2002). (I’ve never really felt any need to do so). They are exported from MapMate to MapInfo, which is where we do most of the work.

No matter which features are decided upon next, R6 first and foremost should be solid and reliable. Our experiences with item 1)  and problems I reported earlier with the import wizard discarding data in every 64th record, suggest it is otherwise. I feel it is better to consolidate what is there already before adding too many new features.

Keith Balmer
Beds and Luton LRC

27

Re: Possible enhancements to Recorder 6 - please contribute your ideas

Improvements I would like to see; where others have also mentioned these points I have not expanded on the topic.

1) More flexibility in the fields available for use in the Import Wizard.

2) More fields to be available within the Report Wizard.  In particular the following:
   a) Reference documents.  In the case of our transfer of data from Recorder 3 the reference for each record appears in the Observations hierarchy view against Taxon Occurrence-Sources, so we would like the appropriate fields for author, year and title, etc.  at least included, but there is probably a case for including the fields for Sample-Sources, Event-Sources and Survey-Sources.
   b) Keys.  Being able to output Taxon_Occurrence_Key (likewise Sample_Key, Survey_Event_Key, Survey_Key) would be of great help, so that when a possibly erroneous record is spotted in a report, the key could be used in conjunction with the GoToKey addin, to check the original record.  There will be times when I wouldn’t want to do this direct from the report wizard grid (Charles’ point 2.2.2) which would also be a very useful feature.

3) Direct connection to MapInfo, with option of plotting record at centre of resolution, (preferably resolution size related).

4) Batch updating and manipulation.

5) I don’t know if it is the Recorder team or the NHM Species Dictionary team who set the Recommended Taxon Sort Order within Recorder 6, but where a published taxonomic order is available, could it be followed.  The checklist available on the BSBI website is taxonomically ordered yet the Recorder 6 preferred Vascular Plants and Stoneworts list is ordered alphabetically for any taxon rank below phylum.  Likewise the Bryophytes list at all levels.  The lists for Birds, Lepidoptera, Odonata and Reptilia do seem to be ordered taxonomically at all taxon ranks.  The inconsistency is frustrating when reporting across all taxonomic groups; I regularly resort to using a lookup table of the old Recorder 3 SP_CODE.

6) Concatenating into a single column for ‘observation measurement data’ and ‘observation measurement qualifier’.

7) Continuing on from Charles’s 2.3.5, for the Taxon Occurences I find the easiest way of checking another persons data entry is the order in which it was entered, so could the Taxon_Occurrence_Key be a sorting option besides Scientific Name and Common Name.

8) A higher level grouping of surveys, using the ‘tagging’ option.  Currently, our surveys are nearer to a dataset level.


I also have a couple of comments which I have not seen raised before, where a minor change to what is shown on some tabs in the Data Entry windows would make a big difference.

9) Sources tabs. Internal Documents shows Document, Original and Type.  We have a large number of references which are effectively the same author and with almost identical document titles but it is the year which distinguishes them.  The Author, Year and Document Title would be of more use as the visible fields.

10) Locations. Geo Info tab-Administrative Areas and Grid Squares tabs.  Would it be possible to make these sortable so that e.g. the parishes are grouped together alphabetically, or all 1km squares listed together in order.  We have many large sites that cross several parish boundaries or fall in more 1km squares than are shown in the available space and I have recently wasted quite a lot of time scrolling up and down checking through out-of-order lists.

Alison Stewart
Environmental Database Manager
Dorset Environmental Records Centre

Alison Stewart
Dorset Environmental Records Centre

28

Re: Possible enhancements to Recorder 6 - please contribute your ideas

Alison and Keith, you both mention a direct connection into MapInfo but could you elaborate on what you mean by this exactly? I know in ArcGIS we can connect directly to the Recorder database (Recorder can't connect to or control ArcGIS though) but this is only a tiny bit of the story. Using live data is impractical as the number of joins and data involved causes things to grind to a halt, so we have to create flattened tables of data for our specific purposes. This isn't too far removed from using the report wizard and manually dumping out data into a GIS although I have more flexibility and can do overnight runs.

But anyway, could you possibly elaborate on what goals you're trying to achieve or simply provide further info when you mention direct connection to MapInfo.

Ta,
Charles

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

29

Re: Possible enhancements to Recorder 6 - please contribute your ideas

Charles,

We do practically all of our data searches in MapInfo and have exported flattened records from Recorder 2002 to produce various GIS layers, the two most often used being:

“All Records” – Everything held in R2002 – Currently 532,000 records.

“Notable Records” – Rucksack-maintained list of species that we think consultants should know about in data searches – About 38,000 records.

In the course of creating these layers the grid ref is converted to three additional fields: Resolution (m), X and Y, with a “5” inserted at the relevant place in X and Y to map to the centre of the record’s resolution. Thematic maps can colour the resulting dots according to resolution. (Square polygons of the record’s resolution size might be a desirable alternative, but I don’t yet have the skills to know how to produce).

As far as I know the ability to produce these three fields is not in Recorder wizards/snapshots and so each LRC must devise their own methods of getting records into GIS. I bet there’s a lot a wheel-reinventing by busy LRC staff which could instead be provided centrally by R6 developers or best-practise shared by LRC super-users.

Another example - I’m considering linking JNCC’s latest (11th Oct) spreadsheet of species designations (http://www.jncc.gov.uk/page-3409) to R2002 and MapInfo so that we can display records of any designation(s) against layers of Sites, Habitats, Soil types, Districts, 1:10000 maps, MasterMap, etc. We increasingly want to interrogate records in GIS where they are most useful.

Ideally GIS tables would be produced from “live” Recorder data such that the MapInfo user wouldn’t have to do anything (or much – just a button press) to get the latest records on their screen. (If we had MapBasic and I knew how to program it then such things might be possible. I'm seeing what I can achieve via DBMS, but as its all self-taught experimentation in snatched moments, I could easily be heading up a cul-de-sac).

On a philosophical point I would suggest though that R6 should be kept as simple as possible, following the KISS principle (Keep It Simple (and) Stupid) so that it is robust, maintainable and efficient, and that fancy features, such as GIS-aids, should be kept separate for the smaller subsets of folk that need them.

Regards,
Keith Balmer
Beds and Luton LRC

30

Re: Possible enhancements to Recorder 6 - please contribute your ideas

Hello Charles

Keith has written my reply for me!  We are effectively doing the same thing to create our GIS layers.  I have likewise experimented with snapshots and DBMS linking, but I am also learning as I go along and my time for such things is severely limited.  Something within R6 that could generate these three fields when outputting to a GIS format, would be a good starting point in saving a lot of time on a repetitive task, but I would also like to see something along the lines of the 'MapInfo user accessing the latest records' scenario that Keith describes.

Alison Stewart
Environmental Database Manager
Dorset Environmental Records Centre

Alison Stewart
Dorset Environmental Records Centre

31

Re: Possible enhancements to Recorder 6 - please contribute your ideas

Here are some ideas that could help facilitate these requirements (which are very similar to requirements here in Sussex too):

1. Add eastings (& eastings centred), northings (& northings centred) and grid resolution as report wizard attributes.

1.1 As an addition to this, it would be useful if Recorder could output a GIS layer (a .shp for instance) showing a grid square at the appropriate resolution (see how the NBN Gateway plots records for an example of what I mean). If this were implemented, each grid square would need to be able to refer back to the data that created it, so one of the attributes of the layer would be a sample_key or taxon_occurrence_key, thus allowing it to be joined to the snapshot or database.

2. Filtering in the report wizard is improved to include filtering by group and designation(s), and to allow specification of filters through control lists (as when entering a record) instead of free text. This would make reports/snapshots much more effective and flexible.

3. Enable XML Reports to be output to a snapshot.

4. Allow for snapshots to be run on a schedule (overnight, several times a day, etc.)

5. Allow for a snapshot to be triggered externally on demand; i.e., from the command line or another application (like MapInfo or ArcGIS).

The good news is that no. 1 can be done already (we have those attributes in our report wizard). They were added by Mike Weideli, so if you're desperate for them, do get in touch with him. He can also add other attribs; just ask and he'll tell you if something is possible or not.

In regards to the GIS accessing the latest data live, that, I'm afraid, is something that could only be accomplished in the GIS itself really. As Keith pointed out, it would need to be done in MapBasic on MapInfo and on ArcGIS there are various ways of doing it (as a VBA extension, or .NET extension or similar), but it's stuff that would need to be built into the GIS rather something that could be built into Recorder. Exegesis are probably you best best for that sort of work.

Having said this, it is possible to build a View in SQL Server and then use MapInfo to connect to that view. I've done this in ArcGIS, but because views are queries that run on-the-fly, it's very slow unless you build highly constrained views that pull out only relatively small chunks of data. I think a better solution (for now) to viewing live data is the facility as described above to run snapshots on a regular schedule. If you needed completely up-to-date data, you could always go into Recorder and run a snapshot manually and thus give yourself the latest data.

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

32

Re: Possible enhancements to Recorder 6 - please contribute your ideas

Sadly I don’t have the time to keep up to speed with the forum detail by detail, so I bow to those with greater understanding (specifically CR) who have got a full picture of the issues and problems.

However, as someone who is flitting across the lake of R6, like a dragonfly laying its eggs, I have my own ‘snapshot’ views of the difficulties of the program and offer them up in a broad way. I make no apologies for repetition.

Point of principle.
Recorder is a database, involved with Input, Maintenance and Output. At this stage, I think that is all we should ask of it.


Recording cards
– could be more customisable (i.e. The data manager can create reporting cards that emulate R3 more closely so that volunteers can be comfortable with the input format.)


Filters
– more terms available.
– Filters once applied should allow reporting from them, or exporting of.  R6 has a graphical interface – why not use it properly – report/export what you see.  Select/deselect what you see.

Batch Edit
- all been said before. One by one is too painful.


Reporting
- One gets the sense from reading the forums that you have to be able to wield msxml with aplomb before you can get anything useful out of recorder.
- The reports that come from the wizard are stunted and don’t feel very robust (i.e. I don’t immediately trust what the output tells me).
- The wizard should be more adaptable. (i.e. Advanced tab that takes you into the full (ish) dtb structure to select /deselect elements, change output form layout etc.)
- Personal information. Can this be kept seperately from other data so that one must specify whether to include personal information in an export.  Address information is a real data management issue and needs to be treated differently. (i.e. confidential tab on people).


Import procedure
– as Charles mentioned, be able to import to the entire dtb (again Advanced tab/ administrator privileges)

Checklists
– make them more navigable.
– Way too many available in one go. A Sub menu possible with these checklists 'supra' categorised.
Is the Recorder 3.3 list the most general list available? Very unsure about what they all mean /their relevance. Is metadata available about these checklists – numbers of taxons within them, geographical coverage, legislation that produced them etc etc. for guidance? Confused and confusing.

Drop downs
– make bigger. The checklist dropdown is so small for such a long list.


Rucksacks
- These SEEMED to be the place where one could actually specify checklists or reporting lists of ones own, giving one some control. But to populate them you have to drag and drop one by one! A little long winded.
It was going to be my way of dealing with the fact that I didn’t trust the designations applied within Recorder, and wanted to have my own checklist of local BAP spp. (There seemed to be no way of reporting designations out, for me to validate.)
- I HAD thought that when the updated BAP species lists were published, with the current NBN Taxon version key, I was going to be able to drop those keys into a file and import through the rucksack.  No no no.
I then tried mapping the keys, then the names, onto Taxon Key, taxon version key etc – each time the import missed out a whole load. 
The point would be – Nowhere tells me how to create rucksacks from scratch, nor can they be easily populated. What I thought was going to be a solution seems to be a bigger problem. I generally end up reporting everything and filtering in excel.
If they are going to exist then they should be adaptable - again an import procedure from text files?

Customisation
- Yes please. Lots of tabs - all well and good but most of them wont be used most of the time - confusing and scary for volunteers to navigate. Administrator sets tab structure for each user? OR specifies own tabs/forms with drag and drop type customising.

Geography…
- I don’t use the mapping bit of Recorder, in my books Recorder is not a full GIS.
- I don’t want recorder to try and be a GIS – most LRC’s will run a GIS anyway.
- I don’t want money to be spent on developing something we have already paid for elsewhere.
- Having said this – recorder does on the fly distributions seemingly quite well. How the distributions are calculated is another issue. 
So either Recorder allows its distribution points to be exported directly to shp/tab/dxf etc or it allows the basic observation data to be exported or accessed more easily from the GIS. (see reporting points above).

- Importing locations to Recorder.   In the same way that we have import files for observations, I would love to see Recorder be able to import a dbf and/or a shp/tab with set fields that define locations (name, code, xmin, xmax, centroid, hierarchy level etc).  This could go through the map window with polygon boundaries, or as a table, without the need for a taxon observation attached.  But maybe this is the step towards GIS’ing too far.

- Lots of sites are already maintained in GIS, and they need matching up with the locations in Recorder. The two systems have to communicate, and that either occurs once records are export from recorder, or we try and bring datasets together within recorder.

- On a related point – as I currently understand it… if one connects to the DTB through Arc or MapInfo – we connect to the individual tables, after which we have to specify the connections that we require. Can we not have a summary table /query table in the dtb that offers up all the observations with associated data. The GIS connects to this and we query it from the GIS in the same way as we would any other table.  The GIS programs are perfectly capable of handling several million records as point files. Can we not connect to queries in the NBN DTB?


The general point; a lot of folk using Recorder don’t have the time, skills or interest to unravel the ‘how to do various things’ from watching the forums techno speak.
Folk need functionality in an understandable form that forces the NBN data model through its wizards and structures.
As Keith Balmer mentioned – ‘snatched moments’ is all we have available to us.

How much did they say we had available - £25k?

MAtt



Where did all the hours go?

Cumbria Biodiversity Data Centre
Tullie House Museum

33

Re: Possible enhancements to Recorder 6 - please contribute your ideas

Hi

I would like to add my small contribution to this thread:

- The import wizard should check that the records to be imported do not duplicate records already in the system (not just do a check for duplicates in the import file - not especially helpful to me).

- A 'search and replace' facility to move all records of species 'a' in Checklist 'A' to the relevant species name in any new Checklist.

- Improved reporting options (eg I wish to get a list of observers who have contributed records in a year - I cannot find a way of doing that with the Report Wizard). Perhaps a set of instructions to prepare queries in Access to check the R6 database - currently, I have no idea how to tackle this).

Cheers, Ian

34

Re: Possible enhancements to Recorder 6 - please contribute your ideas

Hi,

I agreed with Ian. We have the similar problem here where the data that entred are from different sources and there's no where to current takle this issue in Recorder 6 import wiz. we have built duplication check into our input system to solve the problem for the time being.

cheers

luck

IT Officer
rECOrd (A Biodiversity Information System for Cheshire, Halton, Warrington and Wirral)
www.record-lrc.co.uk