Hi Jim,
   so, the new atlas does not support the old 'vague' notion of a recording's time stamp? In the previous version it was possible to say that a reading occurred sometime within a range of two dates, since the new version requires a single date, is this data lost? I will post this on github to alert the developer (https://github.com/nbnuk/nbnatlas-issues/issues/190). I wonder, if it is central to the database design to have a single date for each recording, whether the old range might be calculate to a +/- Error (and the single date used to record the center of the range)?
e.g.
[Start: 2017-01-01, End:2017-01-31] -> [Date: 2017-01-16, Error: 15] (i.e. +/- days)

   All the best,
      -Duncan.

p.s. For interest: the raw and processed versions of data is available via the api, e.g.
https://records-ws.nbnatlas.org/occurre … 6c06a6e465
which is json and can by copy/pasted into a viewer to make it easier to read (e.g. http://json.parser.online.fr/)

Yes, I had also noticed that the atlas weren't working correctly with the NBN vague date system and included this in direct feedback to the NBN team. Thanks for raising here as it seems important everyone is aware of the issue until (hopefully) the NBN data model is fully implemented to cope with YY etc, particularly as it makes the month charts very misleading for many species.

Presumably a short-term improvement would be for the date type Ys to have the value 1 in the month field removed and DDs to be dumbed down to month/year or year depending on the variation between the start and end dates etc. This would at least go some way to making the month charts/maps more reflective of reality.

I realised we can easily check the entire database to see if all the records with a year or year range have been given dates in January.

First we can get a count of the different date types that have been imported using the query

https://records-ws.nbnatlas.org/occurrence/facets?facets=date_precision

I am taken by surprise by what I find. The results are as follows

| Date Precision   | Count    |
|------------------|----------|
| Day              | 19636121 |
| Day Range        | 3109621  |
| Month            | 1195614  |
| Month Range      | 139013   |
| Year             | 2148104  |
| Year Range       | 2057207  |
| Before Year      | 692342   |
| No date          | 18052    |
| Unknown          | 16376    |
| Publication Date | 13471    |
| Circa            | 12       |
| D                | 3403238  |
| DD               | 91553    |
| O                | 158077   |
| OO               | 342      |
| Y                | 705973   |
| YY               | 1206230  |
| -Y               | 98787    |
| U                | 385      |

Until now I had only seen the descriptive precision values, not the old style date type abbreviations.

The next thing we can do is, for a particular date precision, find out the distribution of records across the months with a query like

https://records-ws.nbnatlas.org/occurrence/facets?facets=month&q=date_precision:"Year"

Again, some unexpected values but overwhelming confirmation that these records with year precision have been placed in January.

| Month   | Count   |
|---------|---------|
| January | 2147053 |
| May     | 2       |
| June    | 6       |
| August  | 11      |
| TOTAL   | 2147072 |

Note that the total does not match the number of records with a date precision of year. These will be records without a month value I expect.

Jim Bacon.

Copying the pertinent github thread here for info: djtfmartin says,
"We store a start and end date, and we've recently added a date precision field.
We do need to look into indexing these fields better. Apparently the newer versions of SOLR has some support built in.
So there isnt anything inherent about the Atlas that prevents it, its just something to add at some stage."
"FYI - i enabled a date precision filter yesterday, and date precision in downloads.
This is currently just the raw value provided in the original data, no parsing or interpretation."
See image: https://cloud.githubusercontent.com/ass … faf568.png

p.s. when the record only has a year and no month or date it defaults to January, we are aware of this and working on it - the full information is in the data download, so while not ideal the information is still available.

Keith Balmer posted about a problem searching for the peacock butterfly, Aglais io. He observed that he could find no species records by this name although a lot of records existed under the genus Aglais.

The web services allow searching against some of the original values for data imported from the gateway so I tried the following:

https://records-ws.nbnatlas.org/occurrences/search?q=raw_taxon_name:"Aglais io"

This finds 404374 records.

Picking one of these at random I can look at it in detail.

https://records-ws.nbnatlas.org/occurrence/05375aa6-508c-478c-b77b-0b467f669e79

The raw object in the returned JSON includes the following classification information that came from the Gateway:
scientificName: Aglais io
vernacularName: Peacock
taxonID: NHMSYS0021143568
There is a warning about a missing taxonRank.

The processed object, which the Atlas website will be using, has the following:
taxonRank: genus
scientificName: Aglais
taxonConceptID: NHMSYS0000515936

Let us just do a quick comparison with a record submitted against the synonym Inachis io. My search returns 489 records and my random example has uuid 7aacc88c-f3bd-49fa-92e9-b5fe2bd572a9

The raw object includes the following classification information:
scientificName: Inachis io
vernacularName: Peacock
taxonID: NHMSYS0021143568
There is a warning about a missing taxonRank.

The processed object has the following:
taxonRank: species
scientificName: Inachis io
taxonConceptID: NHMSYS0000502940

A faceted search shows that every record which was imported with species name Aglais io has ended up as a genus record.

https://records-ws.nbnatlas.org/occurrences/search?q=raw_taxon_name:"Aglais io"&facets=rank

A further faceted search to look for genus level records which might have been recorded at species level can be attempted.

https://records-ws.nbnatlas.org/occurrences/search?q=rank:genus&facets=raw_taxon_name

Aglais io is top of the list but others (with record counts) include Spilosoma lutea (67961), Campaea margaritaria (48737), Anarta trifolii (22068), Anania hortulata (15531), Pennithera firmata (8076), Deltote pygarga (7110). One thing these all have in common is that they are absent from the UK Species Inventory. Would I be correct in guessing that species names were matched against UKSI on import?

Jim Bacon

Hi Teresa

This is not the same problem but it may be related.

The original TVK which this record had in the Gateway was NHMSYS0021144070
I think this TVK is for a synonym which is not being returned by the UK Species Inventory: Pasiphila chloerata (Mabille, 1870).
The imported occurrence records have been given the TVK NBNSYS0100004488 which matches the UKSI recommended name: Pasiphila chloerata Mabille, 1870.
So far so good.

Now here is the problem. If we search the Atlas for the accepted name for the species, it returns the synonym and links to records with the TVK of the synonym. There are no such records as they were assigned the TVK of the recommended name.

You can access the records you want at https://species.nbnatlas.org/species/NBNSYS0100004488.

It appears that, on importing the occurrence records, TVKs were updated to be consistent with UKSI. Could it be that the species list in the Atlas was imported from the Gateway and was not updated in the same way or that it failed to update this particular species because the synonym was not in the inventory?

I hope that by trying to characterise the problem it will assist the NBN in fixing it.

Jim Bacon

Hi Jim

Are you saying that it is possible that, when datasets were imported containing TVKs for junior synonyms, they might have been "updated" to the TVK for the recommended scientific name?!  That would be utterly horrific as a vast amount of meaning would have been lost.

I have spotted a few situations where asking for a map using the TVK of a junior synonym has not resulted in a map but using the TVK of the recommended name gives a map. I'm a bit worried that there has been some confusion around the technical difference between "names" and "taxonomic concepts".

Chris R.

The fact that Aglais io is being interpreted as Aglais suggests that it's the Scientific name rather than the Taxon id that's being used as the source of the taxon concept.

However, on some records (Oligia strigalis agg.) it seems to be using the Taxon id, perhaps as a fallback

Taxon GUID match 
The match was based on the supplied taxon concept ID rather than the scientific name.

I would have thought that all records should be matched based on the Taxon id ?

[and certainly preferred over the situation with Aglais io where "The match is based on the higher level classification."]

The authority doesn't seem to appear in the original data?

Basis of the taxon match - Name match metric under Taxonomy

Ah, yes that field is entirely missing from Oligia strigalis agg (perhaps I was expecting it would be blank rather than missing altogether?). However, I would still guess that it's not using that field - processed data is blank and even when it is supplied for e.g. Oligia strigalis ss, the "match was based on the supplied taxon concept ID rather than the scientific name": https://records.nbnatlas.org/occurrence … f3e349d8f5

I think it's looking at the Scientific name, found two matches in Oligia strigalis agg and Oligia strigalis - panicked, and fallen back to the Taxon id.

But as we are supplying definitive taxon concepts (or whatever the right term is) there should be no panicking? It should dive straight for that and ignore everything else (that's why they were invented...?)

In the UKSI the Nameserver gives a "recommended_tvk" (the recommended scientific name for this taxonomic concept) for every name in the database. You can use this TVK as the taxonomic concept ID but this might cause problems if names are added or old names raised to be the scientific name for the concept. The Organism table does have a unique, (relatively) stable key for the taxonomic concept. But It is just a parent/child relationship table with the recommended_tvk of that concept recorded on each row. I say it is relatively stable because of course concepts can be merged when they are synonymised and currently one row is just marked deleted, rather than being forwarded to the other row.

Hi Christine,
Putting aside any peculiarities in the search I wonder if you have uncovered any new aspects of content in the database that might be considered erroneous?
So far we have

• species records converted to genus records e.g Aglais io,

• a difference between the accepted name/TVK in the species search compared to occurrences, e.g. Pasiphila chloerata

• an alteration of a species name, e.g. Oligia strigalis agg

A species search for Arran Carpet suggests Dysstroma truncata subsp. concinnata (Stephens, 1831) is the preferred name with TVK NHMSYS0021157753. Attributed to UKSI but not a name I can find through the UKSI website.

This brings back no records as, you tell me, any record imported with that TVK has been changed to NBNSYS0000005844. I suggest that is because the match on import couldn't find the subspecies or TVK so, as we are informed, it reverted to a match on the higher taxa, lopping off the subspecies name. UKSI confirms that this is the TVK for that name.

The conclusion I seem to be coming to is that occurrence records have been processed to match the current version of UKSI while the species dictionary that the Atlas uses for searching is based on a different version of UKSI with a different set of names and TVKs

Jim Bacon.