Steve: separate databases are fine, but it should be clarified how the might best be implemented. The purpose for a new level above surveys would be to:
a) Enable more efficient organisation of data
b) To allow sets of data (many surveys) to be excluded/included from reports, exports, etc. on an as needed basis
To give you a use case, we sometimes receive data from other LRCs in order to carry out cross-border, or regional, projects. This data needs to fit with our existing data (for reporting) and thus wouldn't ideally be put into a completely separate database (it would need to be integrated with our core db). But with the current survey structure, these data get "mixed in" with our core Sussex data, which causes long term problems (lots of locations and recorders we don't want or need) and are hard to get rid of.
Another use case is that we need to exclude a set of about 10 surveys from all exports. Currently there is no practical way to do this unless we export surveys individually. If were were able to dump these surveys into a "folder" that enabled reporting, but excluded exporting, that would be ideal. Each "folder" or category would have a set of access controls that would enable/disable, for example, exporting, reporting, data entry.
I suppose that separate databases could be used to implement this with a facility in Recorder could turn them on and off and apply the access controls. The key is to be able to cleanly integrate and separate multiple datasets on an as-needed basis.
Gordon: good point regarding duplicates. I way to detect and manage them would indeed be a great time saver. Like you, we simply ignore them for now in the absence of any practical way of dealing with them. This also relates to items 2.1.1 and 2.2.1 in my list; once duplicates were detected, we'd need a way of (batch) marking them as invalid quickly and easily.
I would also add to this a way of detecting and managing invalid data. Currently, the only way to detect invalid data is to export it, then re-import it, which is a huge time waster.
Here are some points regarding reporting:
3. Reporting and Output
3.1 Geospatial reporting
3.1.1 Currently, a lot of time is wasted exporting data into GIS, converting it into grids, then reporting against those grids using polygons. The fact that Recorder places points in the bottom-left corner of a grid square means that it cannot reliably be used to do geospatial reporting, thus necessitating the use of a GIS.
3.2 Report Wizard
The report wizard has come a long way, but still needs further improvements in order to save time doing tasks which should be unnecessary
3.2.1 When saving a report, also save the order of the columns. Rearranging the columns consumes a lot of time.
3.2.2 Enable exporting to GIS formats, such as SHP and TAB. Also allow exporting as NBN Gateway style grids, in addition to points
3.2.3 Improve the filters; allow for filtering by taxon group and other filter items that might be missing (could do with some input from others on this - Darwyn?). Also improve the filter dialog to allow for selecting from terms lists. For example, if we are to filter on designation, allow for the designation(s) to be selected from a list. Currently it is very difficult to select by designation because the precise wording of the designation has to be typed in. Things like recorders, locations, designations, measurements, etc. should be selectable from a list.
3.2.4 Allow for an occurrence to be invalidated/verified from the report wizard output table (see 2.1.1)
3.2.5 Allow for editing of values in-place (see 2.2.2)
3.2.6 Improve reporting on designation; the current solution is flawed as many people have noted. Providing a concatenated list of designations (or perhaps designation kind would be a suitable addition/alternative) in one cell would be one solution, as would adding designation columns dynamically; i.e., if a taxon is returned in a report and it has xyz designation, append an xyz column and a boolean TRUE value to the row.
3.3 Exporting
3.3.1 Improve the export filters so that they are as flexible as the report wizard. It is currently impossible the export with anything like the level of granularity needed with the current export filter facility. Perhaps link export filters to report wizard saved reports, complete with their improved filtering (3.2.3).
3.3.2 Filter exports through the validation library and provide a means of fixing invalid data quickly and easily (see 3.3.3 below).
These next two should really go in the data management section, but I'll put them here for now:
3.3.3 Provide facility for detecting and managing invalid data, and provide a simple interface for fixing the invalid data
3.3.4 Provide a facility for analysing import rejects; provide a report to send back to the data custodian so they can fix the problems.
I've personally spend days and days worth of man hours wrangling with these last two. Finding invalid data is a large part of the task, but then once they are found, it can be very time consuming fixing those same data. For instance, if a set of records are somehow outside the date range of their survey, the date must be changed in at least THREE places, and usually more if there are multiple occurrences under one sample (one date change for each taxon determination). The changes must also be made in a particular order. The process is very prone to error and is quite excruciating if a lot of these changes have to be made. I spend over a day recently updating about 50 records in this fashion. With the added ability to batch process items (in this case, batch change the determination dates), it would have taken perhaps half an hour.
Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital