1 Reply by Stuart (edited by Stuart 15-05-2006 09:30:12)

  • Stuart
  • Guest

Re: Different results between SpeciesList and OneSiteData services

Following responses last week, I had a further go this weekend. You can try some of it for yourselves at: http://www.hoverfly.org.uk/NBN_SpList_Test.php

As you will see, this page prompts for a grid ref, distance, overlap type and Taxon group and attempts to get a species list for this buffer.

If I enter the grid ref of my garden (TL203998) and leave everything else to default, the resulting species list I get back is:

Cheilotrichia cinerascens (Meigen, 1804)
Limonia chorea (Meigen, 1818)
Pedicia immaculata (Meigen, 1804)
Limnophila ferruginea (Meigen, 1818)
Ormosia hederae (Curtis)
Molophilus griseus (Meigen, 1804)
Oxycera pygmaea (Fall?n, 1817)
Oxycera trilineata (Linnaeus, 1767)
Oplodontha viridula (Fabricius, 1775)
Stratiomys singularior (Harris, [1776])
Syntormon pallipes (Fabricius, 1794)
Tipula marmorata Meigen, 1818
Tipula nigra Linnaeus, 1758
Volucella inflata (Fabricius, 1794)
Dixella autumnalis (Meigen, 1838)
Hydrophorus praecox (Lehmann, 1822)
Hydrellia griseola (Fall?n, 1813)
Dixella attica (Pandazis, 1933)

NBN Gateway Datasets from which information came.

Invertebrate Site Register - England.     English Nature
Cranefly Dataset                                 Biological Records Centre

[Note that I don't get any Hoverfly Recording Scheme records coming up, although there are plenty of records from mine and Alan Stubbs' gardens, because the public access to that dataset is "1km square", not "full"].

What I then tried to do was to use the OneSiteData service to get the records for an individual species for the same Buffer (the page publicly available on the Hoverfly Recording Scheme web-site doesn't do this at the moment - if you click on a species name it will show the 10km GridMap for that species).

If I click on Cheilotrichia cinerascens (TVK: NBNSYS0000008206) - which is a Cranefly and the data is presumably coming from the "Cranefly Dataset" - then it works and I get a map showing two records on the edges of my buffer zone and their related data.

If I click on Oxycera trilineata (TVK: NBNSYS0000007818) - which is a Soldier Fly (Stratiomyiidae) and the data is presumably coming from the ISR - then it doesn't work and I get an error in the $client->Response saying that the query failed to find anything. Presumably, at least one record must exist for this Buffer otherwise the species would not appear in the list!

The only reason I can think of is a difference in the "public" access rights to these two datasets. However, looking at the metadata for them on the Gateway, I find the access rights I am shown look identical. Both say:

Your access to this dataset     

    * You can view records at 100m resolution
    * You can view sensitive records
    * You are able to download raw data
    * You can view attributes, where they exist, for these records
    * You can view recorder names, where they exist, for these records

Why can I get detailed records from the Cranefly Dataset, but apparently not from the ISR?

2 Reply by Richard Ostler

  • Richard Ostler
  • Member
  • Offline
  • From: UK
  • Registered: 11-05-2006
  • Posts: 21

Re: Different results between SpeciesList and OneSiteData services

Stuart,

I'm assuming you're not adding any dataset filtering. I don't think there is a technical problem here, rather I think this is down to where the data is coming from. The web services are not running of the same database as www.seaarchnbn.net, instead they are running off our loading database. Access to datasets on the loading database is not the same as on the live database (its either the same or worse), so looking to see what access you have on the website will not tell you what access you have for the web services.

The grid squares and known site services use warehouse tables to speed up querying whereas the user defined polygon queries directly query the raw data tables. The warehouse tables are only rebuilt at the end of a data loading cycle, so the data in them will fall out of sync with the raw data during data loading.

Why are we in this situation
There are two reasons:
1. To protect the live site. The web services are still in development - we do not want to run them on the live database and risk a dodgy query crippling www.searchnbn.net. We still need to do more work on load testing.
2. Lack of resources. At the moment we do not have the hardware, however, we have specced and are putting an order in for a new db server dedicated for web services. Once its up and running it will have the same data as www.searchnbn.net. I can't give you an exact timetable for this, but it should be no more than a month.

I hope this clarifies the situation a bit.

Richard

[b]Richard Ostler[/b]
NBN Developer

Richard Ostler's Website

3 Reply by wilkin_s

  • wilkin_s
  • Administrator
  • Offline
  • Registered: 31-03-2006
  • Posts: 87

Re: Different results between SpeciesList and OneSiteData services

Stuart/Richard
I just had a quick go with what you have produced. The grid ref SO100100 (rest of parameteres remain the same as their default values) returns a list of species but no datasets. Is this a problem at the Gateway end or the client end?
Steve

4 Reply by Richard Ostler

  • Richard Ostler
  • Member
  • Offline
  • From: UK
  • Registered: 11-05-2006
  • Posts: 21

Re: Different results between SpeciesList and OneSiteData services

Steve,

Can you post your query - it would help us find the problem.

Richard

[b]Richard Ostler[/b]
NBN Developer

Richard Ostler's Website

5 Reply by Stuart

  • Stuart
  • Guest

Re: Different results between SpeciesList and OneSiteData services

I have had quite a lot of problems with extracting stuff like the Data sets and the Species details out of the associative arrays returned from a web-service request. The nesting within the data structures does not seem to be consistent. I have been meaning to post something about this here - but haven't found the time to nail it down properly! I think that is why it is not always successful at tabulating the Dataset metadata.

Not sure where the problem lies - is the structure of the XML returned varying or is PHP having problems parsing it into an array? One way or another - I suspect the problem lies in the way missing values / empty elemnts are dealt with!

6 Reply by Richard Ostler (edited by Richard Ostler 16-05-2006 15:04:25)

  • Richard Ostler
  • Member
  • Offline
  • From: UK
  • Registered: 11-05-2006
  • Posts: 21

Re: Different results between SpeciesList and OneSiteData services

I've experienced similar problems with PHP. The XML structure for datasets does not (should not) vary.

I've found if only 1 dataset is returned in the list and I try and loop over using a foreach loop I get an error - it only works if there are > 1 datasets. So I've ended up writing code like the following:

print '<table border="1">
$DatasetSummaryList = $response['DatasetSummaryList'];
// php foreach does not seem to handle arrays with just one element so... test size.
if (sizeof($DatasetSummaryList['DatasetSummary']) > 1) {
    foreach ($DatasetSummaryList['DatasetSummary'] as $DatasetSummary) {
        $ProviderMetadata = $DatasetSummary['ProviderMetadata'];
        print '<tr><td bgcolor="#FFFFFF">'.$ProviderMetadata['DatasetTitle'].'</td>';
        print '<td bgcolor="#FFFFFF">'.$ProviderMetadata['DatasetProvider'].'</td></tr>';
    }
} else {
    $DatasetSummary = $DatasetSummaryList['DatasetSummary'];
    $ProviderMetadata = $DatasetSummary['ProviderMetadata'];
    print '<tr><td bgcolor="#FFFFFF">'.$ProviderMetadata['DatasetTitle'].'</td>';
    print '<td bgcolor="#FFFFFF">'.$ProviderMetadata['DatasetProvider'].'</td></tr>';
}
print '</table>';

I don't have much php experience, but I haven't encountered problems with Java or c#.net, so I'm thinking this is a problem with PHP arrays rather than the XML.

[b]Richard Ostler[/b]
NBN Developer

Richard Ostler's Website