1

Topic: Report constraint end date/start date?

We have very recently added a large (420,000) flora atlas dataset to Recorder 6 where every record is a tetrad with a date range of 1974-1997. We'd really like rare and protected species records from this survey to be returned in our standard consultant data requests in future, because often this is the best information available for flora species in most parts of the county.

We use the report wizard for our standard data requests and a constraint of the event date being post 31/12/1979 to remove historical data, of which we have quite a lot, being a museum based records centre.

On testing, this doesn't pick up any of these flora records, so it must work on the start date rather than the end date of a date range. Does anyone have any suggestions for a work around? So far my colleagues and I have thought of:

- Changing the date range on all the records - this would be bad data management so I am reluctant to do this

- adding duplicates of the rare and protected species records only in a separate survey for data requests and changing the date on those (still bad data management)

- no longer using the date constraint in Recorder and adding an extra step to our excel macros to delete historical records (guess this would work ok, but we prefer doing as much in Recorder as we can before exporting results)

- no longer using a date cut-off, or changing it to 1969 (whereas really we were thinking it was time to increase the cut-off, perhaps to 1989)

Does anyone have any other suggestions or work arounds?

-----------------
Teresa Frost | Wetland Bird Survey National Organiser | BTO
Other hat  | National Forum for Biological Recording Council
(Old hats  | NBN Board, ALERC Board, CBDC, KMBRC)

2

Re: Report constraint end date/start date?

You could always export the records using a SQL script instead of using the report wizard.  This will give you much more control over which records are included/excluded and can even perform some of the actions that are typically done afterwards in Excel (e.g. reformatting fields, summarising records, excluding records, search for specific text like 'roost' or 'nest', etc.)

Better still, run a SQL script that will extract all the records you need and insert them into a new table in the R6 database.  The data can then be spatialised (plotting records as points or polygons as required) and then accessed directly as a spatial GIS layer in ArcGIS or MapInfo.

This is how several LERCs in the south-east that I work with do it.  The SQL script give huge flexibility to select, format and enhance the record details as they are queried in R6, and the spatialisation saves having to plot records using Easting/Northings in GIS.

Andy Foy
Systems Manager
Greenspace Information for Greater London (GiGL) CIC
www.gigl.org.uk

3

Re: Report constraint end date/start date?

Thanks for the idea Andy, an interesting one. Moving to SQL reports is something we've considered in the past but never felt the need for. One of my issues is that there are relatively few people that can confidentally write the R6 XML/SQL reports - with the skills we have we can only make minor modification in-house, and I don't want CBDC to have to go to someone else all the time when a minor change is needed, e.g. a client requests something slightly different. The report wizard is so flexible! However there are one or two other things it can't do yet (e.g. give the spatial ref qualifier).

I do use XML reports that already exist or write basic SQL queries in SQL Server Management Studio, but I guess it is partly just because I feel much more confortable in MS excel and access for manipulating data so often I just dump data out and then process it - Access in particular being able to switch between query design and SQL view is really handy for getting things done quickly, and pivot tables in excel I find a really speedy way of summarising data on the fly. Getting a bit off-topic now. :)

I've never felt a need for getting a spatial GIS layer setup although I know others use it - probably because we put all our data on the Gateway, so we use that when we want to see the records quickly ourselves -  it is easy to use the IMT to change the resolution when viewing. For standard requests we just use the report wizard to select spatially after importing a polygon, and then use excel to give the distance of the record from the polygon centroid for the client - so the species records themselves rarely get into a GIS format unless the client has requested a shapefile as output. Perhaps I need to see the speed of report flow with someone using that setup to see if it would bring any benefits.

-----------------
Teresa Frost | Wetland Bird Survey National Organiser | BTO
Other hat  | National Forum for Biological Recording Council
(Old hats  | NBN Board, ALERC Board, CBDC, KMBRC)

4

Re: Report constraint end date/start date?

Hi Teresa

The benefit of using a GIS based approach (and integrating it with SQL Server, Access, Word and Excel) is that much of the process can be automated (from extracting the data from R6 to generating a final Word/PDF report).

Using a SQL script is certainly much easier to understand and maintain than a R6 XML/SQL report, and by exporting all your species records in to a new 'flattened' table you can then select and extract any sub-set of records for any client very quickly and easily using GIS tools written in MapBasic (for MapInfo) or Python (for ArcGIS).

I could certainly demonstrate the process for you, and discuss the pros and cons, although a long-distance demo might be tricky.  Are you planning on coming south any time or is there anywhere to meet part way?

Out of interest ... what GIS are you using?

Andy Foy
Systems Manager
Greenspace Information for Greater London (GiGL) CIC
www.gigl.org.uk

5

Re: Report constraint end date/start date?

I attach SQL which you can be run in Management Studio which will set up a new Report Wizard filter called Sample End Year. Access this via 'Additional Filters'. Select greater than  and enter a year. Only records after this year will be picked up  based on the vague date end of the sample. It will pick up vague dates such as seasons and unknown which recorder holds with an end year of 9999.  If this is a problem  then the SQL can be adapted to remove these.

Post's attachments

FilterOnEndYear.sql 1.02 kb, 2 downloads since 2015-08-05 

You don't have the permssions to download the attachments of this post.
Mike Weideli

6

Re: Report constraint end date/start date?

That's brilliant, thanks Mike! We will test and hopefully it will sort our problem and then we can think about whether to change our report processes or not. We do have a few records with seasons only/unknown, but I don't think it matters greatly if they are included.

Andy - maybe there will be time for a demo at the ALERC conference in the autumn if you are going, I will ask Tom? CBDC doesn't produce PDF or word reports, we supply records as an excel worksheet with a lot of designation columns and distance from site, so clients can easily filter to the designations/see nearest records as they wish. (We get the designation columns from Recorder 6 too, we've added local designations.) We've got a report wizard template set up so it doesn't take too much time, importing the polygon and adding the constraint is the main chore, since the saved report wizard templates don't remember additional constraints. However being able to do that without opening Recorder may save some time? We've already automated most of the work in Excel using VBA/macros, apart from taking out personal addresses from locations of course, which is unfortunately not something I can see how to ever automate - too many house names rather than numbers, and numbers that relate to roads or compartments rather than addresses...

We're using ArcInfo. We do export the records twice a year to flattened tables to supply as grid square polygons to partners of course so there may be benefits there too. We have an ancient server so anything that can be processed on the PC rather than the server is better.

-----------------
Teresa Frost | Wetland Bird Survey National Organiser | BTO
Other hat  | National Forum for Biological Recording Council
(Old hats  | NBN Board, ALERC Board, CBDC, KMBRC)

7

Re: Report constraint end date/start date?

It is likely that Unknown will be ignored but that Seasons will end up being included. I have worked out how to eliminate these. However, I will wait to see if you encounter any other problems.

Mike Weideli

8

Re: Report constraint end date/start date?

Hi Teresa

Sorry for the delay.  I hope to be going to the ALERC conference (whenever and wherever it is) so I would be happy to do a quick demo of how some LERCs extract data from Recorder6 and use it in GIS.  If there isn't room in the agenda I can do a quick demo during one of the breaks.

Thanks
Andy

Andy Foy
Systems Manager
Greenspace Information for Greater London (GiGL) CIC
www.gigl.org.uk

9

Re: Report constraint end date/start date?

Andy - I'll be at ALERC next week and would be keen to see your demo of this if you are doing one.

Teresa - I see you've mentioned the issue of personal addresses above and elsewhere on the forums.  DERC requested the new fields of Private Location and Private Code to deal with this issue and Mike introduced then in the V6.23.2.276 Release Candidate.  Since then I have imported house numbers, names and postcodes into these fields and also done some ad hoc manual editing during checking of other data.  Before generating the dataset on which the majority of our datasearches will be based for the next few months, rather than editing these fields during Excel post-processing as usual, I instead created tables of the revisions that were needed and used them to update the SAMPLE table from within Management Studio.  All our addresses are in SAMPLE.LOCATION_NAME and in a fairly consistent format so house numbers, and '... House/Cottage' were fairly easy to pick up.  Even with checking against maps that other formats I suspected to be properties actually were, this took just over a days additional time.  Now that I've been through the learning curve, I'm starting to look at the rest of the database and estimate that it could take three to four days, but that will be considerably more time saved in future post-processing.  It's also a good opportunity to tidy up those irritating typos that I've been meaning to get around to for years!

Thanks to Mike for getting these fields included.

Alison

Alison Stewart
Dorset Environmental Records Centre