The sort order is now derived from the Organism table. Before it was the Sort_Order from the Taxon Group table followed by the Sort_Code from the Taxon_List_Item table for the Recommended_Taxon_List_Item key. This wasn't meaningful as it mixes sort orders from different lists.   The original key can still be derived as the base data is still there. If it helps we can provide a UDF to obtain it and provide the field as an option in the Report Wizard.

Thanks Mike, yes that would be a useful UDF. Regarding the Organisms table, is there any background documentation on it? E.g., why we need it, how to use it, etc.

I have always used this to put records from our collections into systematic order and have just tried to do this for our beetle collections but all it has done is arrange the records alphabetically by family then genus then specific names. Not much use to me when I want a list in the same arrangement as the collection. Is there a way of reinstating this valuable indexing tool? A biodiversity database that doesn't report in systematic order is not much use to me.

Mike - I think it is likely that there are multiple dictionaries/lists that have been used for inputting especially as many of the older records will have been input in Recorder 3 and now a mixture of Recorder 3 and the lists in CAPITALS (they are the preferred ones I think) plus a few others where they are not in any of the main lists. NB a single Tullie House Recorder 6 database is used for Steve's Collections records and our CBDC Cumbrian records.

If there were lookups available I could sort the list for Steve in Excel by including the dictionary list used in the report?

As S says the order is very important to R6 users - both for CBDC doing data requests and providing spreadsheets of thousands of records so these are comprehensible to an ecologist, and Steve printing out lists of records and tying those to the Collections that are arranged in taxonomic order.

I think the old sort order is possibly preferable, as at least you noticed the odd plant appearing among birds or whatever... i.e. Neither works properly but it was easier to spot issues with the old one and you didn't get this situation of whole taxonomic groups not being sorted? But ideally of course the new organism table would include a sort order too, perhaps the next big job for NHM?!

Now I am confused Theresa

There is a single, unique sort order, derived from the organism table, but located in Index_Taxon_Name and there are no taxa in the index which don't have a value assigned. From what Mike says, the sorting within any given higher taxon is alphabetic, rather than truly taxonomic, but this is a deficiency of the dictionary, not of Recorder. It merely needs updating with proper systematic taxonomic ordering, as and when the relevant experts come up with definitive lists.

This is a vastly superior situation to the old version which was truly taxonomic, except for where it was broken, in which case it produced unpredictable results, some of which stood out like a sore thumb.

I agree the current implementation is not perfect, but systematically it is far superior. The data structure used to be poor, because the same taxon could sort differently depending on which list it came from. Maybe that was never an issue with coleoptera, (because all the lists were identically sorted?) Or maybe the differences were small enough to be rarely noted.
Now there is a data structure which ensures there is no ambiguity in sort order, but, as yet the sort orders have not been populated in a taxonomic fashion.

Depends on the user I guess. I am not a taxonomist and I am happy if a list I produce comes out with all the birds together and all the plants. In addition it is nice if all the members of a single genus come out together. I am not precious about whether Orchidaceae is the last family of plants in a list (which is taxonomically correct) or just appears between Naiadaceae and Poaceae in alphabetical order. The crucial thing for me is that the current ordering is systematically correct (groups closely related taxa), if not strictly taxonomically.

Reading back, I think Steve's use of systematic is probably what I am calling taxonomic. My mstake, I told you I am not a taxonomist. However the point remains valid. For most users a coherent list which makes sense is probably adequate, but I can see what someone working on a museum collection might not feel the same

First I need just to make sure, there isn't something going wrong. The sort should be taxonomically accurate in the sense that records will come out in Phylum/Class/Order/Family/Genus and alphabeticaly within these, irrespective of the list used for the input. If this isn't happening then we need to find out why and fix.

The problem we had before was that not all recommended lists had sort orders. Also and some recommended taxa came from the list of additional names or were not on the preferred  list and these could come out just about anywhere.

However, you can go back to the old list. The attached Batch update run after an index build will do this immediately. Version 6.20 of R6 will include this legacy sort order in the Report Wizard and there will be a UDF you can use in XML reports to get this sort order.   

I can get the new method accurate in R6 as the mechanism exists, but  not the data. Using the existing lists I can get fairly close on most species, but I will get stuck with assigning some of the more obscure taxa. I am prepared to have a go as far as I can as long as someone who understands the taxonomy can sort out the rest.  Having had a quick look there are about 300 taxa (mostly species) of the 6800 beetle taxa which will not be in the correct order in the old system.

Yes you will lose the existing sort order if you run the batch update, but you will be able to get it back simply by rebuilding the taxon group index.  The ability to get  Order/Family/Genus etc. is associated with the sort, but will not be affected by the batch update.    Don't think the gateway or indicia  use sort order in this way anywhere.

I attach a revised sort order for the beetles, based on the Organism table weighted with information from the preferred beetles lists. There are a number of entries where there is a blank in the OriginalSortOrder. This means that they are not on the preferred list and it is these that need to be slotted into the correct order, by changing the weight. Probably, if the families are  correct the order below this will not be that important. With other lists the problem has often been with taxa which are on obscure lists and perhaps not even Uk species.

The butterflies/moth can  be separated very easily, or it may be possible to apply the same approach as used for the bettles.

I am reluctant to try and apply any blanket solution to all the lists, because some do not have any meaningful sort order and the issues with each list are different. Each group needs to be looked  at individually and the best approach found.

Many thanks for your efforts on this Mike. It is a great relief to know that it is still possible to report on data in a systematic rather than alphabetic way. I am a great fan of Recorder and have been using it since 1992, which is why I was so dismayed at the loss of functionality in sorting. Would it be possible for you to repeat your work for other taxon groups please? Diptera, Heteroptera and Aculeates we have an immediate need for but others are also required. I am sure that many users of biodiversity data will want to be able to sort it according to the order of the standard national checklists (where they exist) – it can’t just be me. I do think it would be sensible for the organism table to have a systematic coding system for all taxa. Perhaps this could be addressed as a priority in a future upgrade? Many thanks

Fantastic! Thanks again Mike.