So to test whether part of the problem was related to recent taxonomic changes I looked at some fungi where thesome species had been moved from Hygrocybe to Gliophilus. This change was not made in the BMS list which I use on Recorder until fairly recently and was not on the Gateway until last year.

Fisrt I tried two Hygrocybe species which had not changed - the search work fine but again if I searched on a trinomial I only got the trinomial records, but if I searced on the binomial I all the records were returned.

When I tried two species which had changed from Hygrocybe to Gliophorous if I searched under the new species name although it appeared on the results lists there were no records. If I searched under the vernacular name, records appeared under related searches and if I looked at the original versus processed data the taxon concept id and species id were missing - hence no match. If I searced under the old name there were no results, although in one case Hygrocybe irrigata returned two records both showing matching taxonomic ids in the original and processed data, but these were the old TVKs. For some reason these two records had not been up-dated.

taxon id    taxon concept    species id   
Hygrocybe glutinipes glutinipes    NHMSYS0001484471    NHMSYS0001484471    NHMSYS0001484470    Taxon GUID match
Hygrocybe glutinipes rubra    NHMSYS0001484472    NHMSYS0001484472    NHMSYS0001484470    Taxon GUID match
Hygrocybe glutinipes rubra    NHMSYS0001484470    NHMSYS0001484470    NHMSYS0001484470    Taxon GUID match
               
Hygrocybe virginea virginea    BMSSYS0000008392    BMSSYS0000008392    NHMSYS0001484537    Taxon GUID match
Hygrocybe virginea ochraceopallida    BMSSYS0000008389    BMSSYS0000008389    NHMSYS0001484537    Taxon GUID match
Hygrocybe virginea fuscescens    NHMSYS0001484539    NHMSYS0001484539    NHMSYS0001484537    Taxon GUID match
Hygrocybe virginea    NHMSYS0001484537    NHMSYS0001484537    NHMSYS0001484537    Taxon GUID match
               
               
Gliophorus psittacinus    BMSSYS0000045739           
               
Gliophorus irrigatus    BMSSYS0000045736           
Hygrocybe irrigata    NHMSYS0001484476    NHMSYS0001484476    NHMSYS0001484476    Taxon GUID match

This would suggest that there is a problem where there have been recent taxonomic changes.

Christine

I can't see any way of telling whether this a problem caused in migration to the Atlas or was a pre-existing problem. Presumably you think this is new, Chris, but I can't for the moment think how that might have arisen.

Hi Jim

Yeah, like you I could understand if the link was just broken complete but to establish a new, erroneous link seems weird. The UKSI database is fine and we've never had issues with the taxa that currently have problems so I doubt that anything was wrong with UKSI as a source of their data. The export was requested in an odd format (Text CSV) so it's possible that something happened during the export but I can easily reissue the files OR just give them an original Access copy which they can use to extract what they need.

I will see what David wants to do.

Chris R.

A query

https://records-ws.nbnatlas.org/occurrences/search?q=*:*&facets=occurrence_status

suggests that there are no absence records at all in the Atlas. Be alert if that may be relevant to your datasets.

Jim Bacon

PS. Wow! BTO records in the Atlas. What great news!

I'm reeling slightly at the size of those numbers... are there really 217 million records in there?

Sounds about right - 152 million in the new BTO dataset, and there were over 100 million on the Gateway (included records with no public view which mostly haven't been published on the atlas, accounting for the "missing" records).

Thanks again for your detective work into the taxa matching issues everyone and particularly Jim, I hope it helps the team solve the issues quickly. There seem to be a lot of lepidoptera being affected - could it have anything to do with the new lepidoptera checklist (see threads in the species dictionary forum here and here)?

I don't see any assertions about the taxon matching process but I can get some more information about the results. The query

https://records-ws.nbnatlas.org/occurrences/search?q=*:*&facets=name_match_metric

returns eight different results for the name matching method and the number of records to which it was applied. They are

+-------------------+-------------------+
| Name match metric | Number of records |
+-------------------+-------------------+
| taxonIdMatch      | 213427156         |
+-------------------+-------------------+
| exactMatch        | 2438552           |
+-------------------+-------------------+
| higherMatch       | 675574            |
+-------------------+-------------------+
| noMatch           | 514150            |
+-------------------+-------------------+
| canonicalMatch    | 59858             |
+-------------------+-------------------+
| fuzzyMatch        | 11790             |
+-------------------+-------------------+
| vernacularMatch   | 2820              |
+-------------------+-------------------+
| Unknown           | 2080              |
+-------------------+-------------------+

I'm reeling slightly at the size of those numbers... are there really 217 million records in there?
We can do a bit of digging in to any of these, for example with this query

https://records-ws.nbnatlas.org/occurrences/search?q=name_match_metric:noMatch&facets=raw_name

This reveals, top of the list, 32k records of Korscheltellus lupulina, NHMSYS0021142343, which have not found a match. Since neither this species nor this genus can be found in UKSI, and consequently not the TVK either, it fits my theory that there should be no match. Example record. There is no name you can search on to access these records. Meanwhile, according to the Atlas species list, this name is accepted in UKSI.

Jim Bacon.

The UKSI recommended TVK for the moth is NHMSYS0021143278 and for Ulmus glabra x minor x plotii = U. x hollandica it is NBNSYS0000162111. I can't see any way yet for the two to have become conflated...?

Keith

I see what you are saying about the synonomy of Alophora barbifrons.

Likewise I see Tephrochlaena oraria is down as a synonym of Syntormon pseudospicatum not Tephrochlaena halterata.

I can't see any way of telling whether this a problem caused in migration to the Atlas or was a pre-existing problem. Presumably you think this is new, Chris, but I can't for the moment think how that might have arisen.

I guess we have to wait for David Martin to be back on the scene to look in to it all.

Jim Bacon.

However, it seems as if that copy of UKSI did not contain some of the TVK/names that the Gateway had employed meaning the the code could not be used.

I wonder if that information is stored anywhere (I can't see it) - i.e. see the steps it has taken (failed using Taxon Id, failed on Scientific name, using Genus). We only seem to be able to see the "last" step. Although if there are only three steps, then you can work it out from the Name match metrics (scientific name -> taxon id -> higher level ?)

Christine

I don't mind what matching algorithm is used so long as the results are correct.

Neither do I - but as we've seen matching on the name isn't and since we have a cast iron code why not use it and bypass any potential slip-ups?

[The agg isn't missing from Oligia strigalis - just not displayed in the processed information]

My example was that "Alophora barbifrons" seems to be linked as a junior synonym of "Parapiophila vulgaris", which it isn't - it's the junior synonym of "Phasia barbifrons".

I've been in touch with David Martin but he says he is travelling this week so I doubt we will get a quick fix for it.

Wondering why I'd been given a tree as a search result I noticed "Larch Tortrix" in the list of names. Isn't this a moth? (http://www.ukmoths.org.uk/species/zeiraphera-griseana). Thus we have something that appears to be simultaneously a tree and a moth? I know it says it is a hybrid, but!

The link to occurrences takes me to records of the tree.

The taxon details page https://species.nbnatlas.org/species/NB … 2111#names gives both Dutch Elm and Larch Tortrix as "UK Species Inventory preferred".

A search for the moth species "Zeiraphera griseana" gives

Note the accepted name for this moth is the Elm hybrid.

Clicking on the "Occurrences: 217" link actually takes me to 465 records for the Elm hybrid!

I've now tried to search for only two species on the Atlas and had two sets of bizarre results.

Keith

• species records converted to genus records e.g Aglais io,

• a difference between the accepted name/TVK in the species search compared to occurrences, e.g. Pasiphila chloerata

• an alteration of a species name, e.g. Oligia strigalis agg

A species search for Arran Carpet suggests Dysstroma truncata subsp. concinnata (Stephens, 1831) is the preferred name with TVK NHMSYS0021157753. Attributed to UKSI but not a name I can find through the UKSI website.

This brings back no records as, you tell me, any record imported with that TVK has been changed to NBNSYS0000005844. I suggest that is because the match on import couldn't find the subspecies or TVK so, as we are informed, it reverted to a match on the higher taxa, lopping off the subspecies name. UKSI confirms that this is the TVK for that name.

The conclusion I seem to be coming to is that occurrence records have been processed to match the current version of UKSI while the species dictionary that the Atlas uses for searching is based on a different version of UKSI with a different set of names and TVKs

Jim Bacon.