I think it may be more than a simple (possible cosmetic) alteration of the observations windows to include top level folders. To me, it's an extension to the data model - a top level "DATASET"?

This would have the advantage of being able to export a group of surveys which have a common theme. For example a data provider like CCW or one particular recorder who is running concurrent surveys.

I think is also important to be able to allow one survey to be a member of many "datasets".

I see a dataset as a set of pointers to surveys. I would think the modifications to Recorder would be another table to hold a list of grouped surveys and an iterative method of exporting all surveys within that group in one go. If the output format is zipped MDB would it make sense to have all of those exported surveys in one file? It would make the distribution and re-import of the data easier.  The import methods would need to be modified to handle multi surveys imports as well.

So, yes please - I'd like to vote for this feature too if it can be afforded.

Kind Regards.

Yes, my original idea was for a simple folder hierarchy to hold/organise Surveys in the window. Exportability (of the hierarchy) was not something I had considered as I would want to fit any imports into my own hierarchy. Exportability would then require importability and presumably multiply the cost/time required but would probably be useful, so I would say yes if it doesnt make the cost rocket. Or simple version first, extras maybe later?

Looking at Dave's suggestions, could queries/reports/exports be done via the Report Wizard>Restrict report to Sources>Surveys dialogue boxes? Possibly with a drop down there for "Survey Group" or whatever they might end up being called?

Gordon

Yes, this sounds ideal John. I had a sneaking suspicion you might suggest the Thesaurus in regards to this, I just haven't used it and can't be sure of what it does and how it does it just yet. It's starting to make more sense now. So to expand on your examples and just mull them over a bit:

1. A top-level folder for containing surveys can be created by specifying one or more tags. Surveys with those tags attached to them get included in the folder. These folders can be then be exported, i.e. the surveys within them get exported.

2. I don't see the use of a tag cloud; do you have an example of what you envisage? Would the size of the tag be dictated by number or records? Or number or surveys? Or what?

3. Option to search by tag would be handy. Filtering, i.e. a persistent 'set' of tags, would be achieved via the tag folders idea mentioned above.

4. The report wizard should be able to output data based on one or more tags, but it should also be able to exclude data based on one or more tags too. This is crucial.

It would also be very useful if the export filter tool could include/exclude based on one or more tags.

Do others have any ideas on how sets of data, i.e. groups of surveys, could be used?

Given that the Thesaurus is a complex beast (at least it looks that way), is it going to make it much more difficult to write manual queries (for XML Reports and such) that make use of it?

A tag cloud would be a fun way of looking into a dataset and seeing what is there and how much of it.  This would become more interesting if tags were enabled on multiple data types, including surveys, references but also locations and so forth.  Its more of a data exploration tool than anything of real use!

The Thesaurus is actually probably a simpler beast to use than the collection of dictionaries and term lists that it replaces in the Collections Module.  Firstly, there is only 1 model rather than 4 to learn (taxon, biotope, admin area and term lists).  There are views to flatten the data so getting the actual term or common name of anything is simple (see VW_ConceptTerm and VW_ConceptTermCommon).  The data model is also designed so that tables such as Term_Version and Concept_Group_Version (roughly equivalent to Taxon_Version and Taxon_List_Version) only come into play when they actually need to, so are rarely involved in queries.  Therefore the path through the model from a to b tends to be much shorter.  Finally, there are no large index tables to maintain except the Concept_Lineage table (which replaces Index_Taxon_Group) - the simplicity of the model means Index_Taxon_Synonym and Index_Taxon_Name are not needed or are replaced by dynamic views.  It does get a bit complicated to use when you start looking at fact and relationship inheritance (e.g. fish have scales, therefore blennies have scales as well - you only need the single relationship at the higher level).

If you have a play with the Reference Keywords editor this gives you a very simple editor for a thesaurus list.  This doesn't support many of the really powerful features of the Thesaurus though.

John, Charles et al,

Some interesting ideas for sure. I'm not familiar with the thesaurus in 6, but are we talking about a form of full text search? (on "tags", "keywords").

Being able to tag (via user defined words) data is a very powerful technique, as is of course adding in the actual data itself. In postgreSQL we use the TSearch2 module to provide this facilty. Sorry if this is going a bit of topic, but as we seem to be talking about data exploration could I illustrate with some code. (Which is all work in progress BTW).

Once a table has a column of type tsvector;

CREATE TABLE search ( vector tsvector)

and is indexed;

CREATE INDEX search_index ON search USING GIST(vector)

it is then just a case of "pouring in" data/keywords etc:

INSERT INTO search VALUES (to_tsvector('Some data you want to find later'))

The example below is a query based on the INDEX_TAXON_NAME table which I've migrated to PostgreSQL and also made into a seperate
full search table. The full text search returns a set of keys to join back to the index_taxon_name table in PostgreSQL.

SELECT
    DISTINCT actual_name, common_name, preferred_name, abbreviation, rank(vector,q) * 10 AS rank
FROM
    index_taxon_name, index_taxon_name_fullsearch, to_tsquery('Fox&Moth') AS q
WHERE
    vector @@ q
AND
    index_taxon_name.id = index_taxon_name_key
ORDER BY
    rank(vector,q) * 10 desc, preferred_name

Data Returned;
actual_name, common_name, preferred_name, abbreviation, rank
===========================================
"Fox Moth","Fox Moth","Macrothylacia rubi","fomot",0.99103219807148
"Macrothylacia rubi","Fox Moth","Macrothylacia rubi","marub",0.99103219807148

The above query for "Fox" and "Moth" returns two records in 1.4secs, a query on just "Moth" returns 453 in 12 secs. The vectored search table has actual_name, common_name, preferred_name and abbreviation contained within it and has 1,250,380 rows.

If this is the sort of technology John mentioned as the thesaurus then it would be great to see it extended in Recorder.

(Edited to remove typos due to heat stroke!)

Regards,

I did wonder if SQL Server had FTS as well, John. Thanks for the clarification on tagging. Just a couple of questions/thoughts;

How do other users know what tags you have assigned - Would it depend on a visual representation Tag Cloud - which is quiite nice and would there be a place in the system there the tag ranks could be found. I'm thinking here of external software that links to Recorder.

Is it be possible to do bulk tagging based on a search criteria?