1

Re: Associated species

We have been trying to add a large number of fungi records each of which has an associated species. We thought that the way to do it would be to use the 'associated species' field. Unfortunately when we used this it meant that each associated species became a new record. e.g. for a particular site there might be 20 fungus record each associated with beech trees. This resulted in  20 records for beech tree appearing for the same site.
Is there any way around this. It seems a shame to lose the associated species data and the only other way we can think of doing it is to add it in as a comment.

2

Re: Associated species

Recorder can relate mutiple records to just one record. That is to say a number of differnt fungi records could be related to just one beech tree record, however, while this is possible via the Organisation hierarchy using drag and drop,  the Import Wizard isn't sophisticated enough to deal with the situation.

I don't think anything can be done to get around this in the import wizard, but a batch update which sorts it out after import would be a possibility.

Mike Weideli

3

Re: Associated species

Thanks for that Mike,

I have been thinking about how to go about creating such an update, but haven't got round to it as yet. Any pointers you can supply would be very useful/

The main problem we have observed with this is that we share data with county recorders who use MapMate, in particular with our vascular plant recorder. MapMate will not allow the importation of duplicate records and our dataset now includes multiple records of tree species created as associates of fungi records.

Logically speaking of course batch updating all such records is not strictly correct as it might be taken to imply that all fungi records on (e.g.) Beech in a given sample are all on the same tree while this may not be the case. However, life is probably too short to worry overly about that.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

4 (edited by RobLarge 12-04-2011 15:18:44)

Re: Associated species

A bit more consideration Mike

I have just looked at the data and fortunately the worst examples are all in a single survey of fungus records. There are nearly 17,000 survey events in this survey, but if i create a query which groups the events by LOCATION_KEY, LOCATION_NAME, SPATIAL_REF and the three VAGUE_DATE fields, this figure is reduced to 1094 distinct events (most of which contain multiple samples, which include both duplicate tree species as associates and duplicate biotope records as well), which give an indication of the scale of the problem. Ideally I would also like to group by COMMENT, but my test query in the Access db will not allow grouping by text fields (I think this is probably a restriction of SQL Server or of ODBC).

We are doubly fortunate since all of this data derives from a single recorder so that is at least one complication I do not have to worry about. I am wondering if you have any idea of a cunning way I could overcome the difficulty in the possible case where there were multiple recorders for any of the samples. I can only think that I would need to step through each of the suspected duplicate survey events ensuring that each has exactly the same event recorders associated, or perhaps create a function which sorts and concatenates recorder name keys and group on that as well.

After that I guess I would have to step through the events, checking the samples within for similar correspondences and where I find duplicates would then need to transfer all the occurrences to the first sample within the first event of each group.

After that I would then check each resultant sample for duplicate occurrences, check each occurrence to see if it is recorded as a related occurrence and reassign all the related occurrences of a given taxon to a single example, marking all the others for deletion (perhaps setting the determination/validation fields appropriately?)

Finally I would need to ensure that each of the marked occurrences now no longer features in any related occurrence, before deleting them and any associated determination records etc.

Writing this message has just reminded me why I haven't attempted it yet. And none of this answers Hilary's question as to what she should do with new data of this type. As far as i can see, the only sensible option for importing large datasets of fungus or lichen data is to relegate  the associations to a comment field. This is a major deficiency in the import wizard, since it gives the impression that it can import related occurrences, but what it actually does is hugely overcomplicate the organisation of data imported.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

5 (edited by MikeWeideli 14-04-2011 13:49:39)

Re: Associated species

The issue with the MapMate import will take some thought and sorting, because somewhere it looks as though it is creating new keys when you would expect otherwise.   

In a normal input situation with Associated species the problem should  be confined to Taxon_Occurrences and Taxon_Occurrence_Relation  within a  sample. In these cases the batch update could identify mutiple entries for a taxa within a  sample where the Taxon_Occurrence_Key is  in  Taxon_Occurrence_key_2 in the Taxon_Occurrence_relation table. This should reliably identify what are unwanted duplicated entries for just the Associated specie, leaving alone any duplicated species which may be there for other reasons. Taxon_Occurrence_relation can then be updated so that    Taxon_Occurrence_Key_2  for all the entries is changed to lowest Taxon_Ocurrence_Key. Then the now unwanted Taxon_Determinations and Taxon_Occurrences could then be deleted.  I will have a go if anyone thinks they will use it.

Mike Weideli

6

Re: Associated species

Thanks again Mike

I know that we would, but if you don't have time I am sure I will get round to it eventually.

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

7

Re: Associated species

Is there a confusion between using the import wizard to import an Excel list and using it to import MapMate records? I use the import wizard all the time to import lists of fungi records with associated species in Excel and do not have duplicate associated species within the same sample. Hence if the one beech tree has five fungi associated with it, in Recorder I get five fungi records and one record of Beech with the fungi associated to the Beech. If I have a site list in Excel with lots of different grid references for my fungi all with associated species, I can get a number of different records of Beech but as they are different samples with different grid references, that is Recorder behaving as it should. Hence relegating associated species to a comment is a bad idea.

I don't use the Import Wizard to import MapMate records so can't comment on that. I am using the current version of Recorder.

8 (edited by RobLarge 18-04-2011 08:41:37)

Re: Associated species

Thanks David, we are not talking about importing MapMate records, the problem with MapMate comes when we export data to MapMate users (since MapMate rejects what it considers to be duplicate records - those having the same location, spatial ref, date and recorders).

Import of fungi records from excel has generated multiple tree species records within a survey event, but interestingly, following Mikes comment above and yours, I suspect there may be something else going on.

Looking at the data as it has imported, the import has generated multiple duplicate samples within a given survey event, each of these samples contains no duplicate tree species, but because there are many samples, each with (say) a record of Beech, we end up with records which look, to MapMate like duplicates.

I seem to recall a discussion on here of the tendency to produce multiple unnecssary samples and that the problem was addressed in an earier upgrade (can't find the relevant discussions right now though. We do have a number of known problems with add-ins on the workstation used to import these records, so it may be that the fault is machine- (or possibly user-) specific and not directly related to the asociated species data.

I will investigate further and see what I can learn...

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

9

Re: Associated species

Ahhhh, think I have got to the bottom of the original problem now.

The imported dataset had a column called ALT, which contained altitude values. I had not appreciated that this was the cause of the multiple samples, because I had not chosen this field to be imported, Recorder auto-selected it.

As a result we have (correctly) multiple samples with multiple associated tree records. Unfortunately we still have a problem because MapMate apparently doesn't check altitude when it is looking for duplicates (or maybe we don't export altitude to our VP recorder)

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre

10

Re: Associated species

OK , following a re-install of the workstation all of these problems seem to have been corrected. Thanks

Rob Large
Wildlife Sites Officer
Wiltshire & Swindon Biological Records Centre