1

Topic: REST taxonObservation Polygon WKT

I've just started to put together a bit of .NET code to pull in data from the new REST web services, and have run into problems using the 'polygon' WKT parameter under /taxonObservations.

When I use the gridRef parameter it works fine and returns results in a couple of seconds, but when I put the polygon WKT that is equivalent to that grid square instead, I get a very long wait and no results.

ie. this works...

https://data.nbn.org.uk/api/taxonObserv … Ref=SH3393

and this doesn't...

https://data.nbn.org.uk/api/taxonObserv … on=POLYGON ((53.4073159895508 -4.51376745766249, 53.4162958678331 -4.51429736135726, 53.4166115845473 -4.49926621884018, 53.4076316035263 -4.49873947465632, 53.4073159895508 -4.51376745766249))

Am I missing something?

Thanks,

Tim

2

Re: REST taxonObservation Polygon WKT

Update - I now get the error message {"status":"Must Supply at least one type of filter; dataset (key list), spatial(featureID, gridRef or polygon) or taxon (PTVK list, Output Group, Designation or Organisation Supplied List)","success":false} even though I have supplied a polygon filter.

Is there a specific method of supplying the WKT that I'm not using?

3

Re: REST taxonObservation Polygon WKT

I tried adding datasetKey and the message gone away but still no result ...

IT Officer
rECOrd (A Biodiversity Information System for Cheshire, Halton, Warrington and Wirral)
www.record-lrc.co.uk

4 (edited by BDeed 12-12-2013 15:37:41)

Re: REST taxonObservation Polygon WKT

Hi Tim,

Also unable to get this to work, tried replacing spaces which is known to cause problems with REST and also variations on the parameter naming. Adding additional filters just ignores the polygon.

Interestingly using the gridRef function and a load of crap works but brings back a nil result. If the problem was with the pairs then i would still expect it to run which leads me to suspect the polygon search isn't yet active.

Have you contacted the team to see if the polygon function is active?

Ben

Natural History & Biodiversity Data Enthusiast

5

Re: REST taxonObservation Polygon WKT

Thinking a little more about this i wonder if this has something to do with being unable to query using complex polygons on the interactive map. The same services are used? In which case we should still be able to do a bounding box...

Natural History & Biodiversity Data Enthusiast

6

Re: REST taxonObservation Polygon WKT

Query https://data.nbn.org.uk/api/taxonObserv … ervations? with the gridRef parameter it returns some taxon occurrences with dataset information but when search with polygon parameter it doesn't return error but taking longer to response ... ?

IT Officer
rECOrd (A Biodiversity Information System for Cheshire, Halton, Warrington and Wirral)
www.record-lrc.co.uk

7

Re: REST taxonObservation Polygon WKT

accidentally put in the incorrect wkt and getting error below. Could this just be timeout issue some where?

org.springframework.jdbc.BadSqlGrammarException:
### Error querying database.  Cause: com.microsoft.sqlserver.jdbc.SQLServerException: A .NET Framework error occurred during execution of user-defined routine or aggregate "geometry":
System.FormatException: 24111: The well-known text (WKT) input is not valid.
System.FormatException:
   at Microsoft.SqlServer.Types.WellKnownTextReader.Read(OpenGisType type, Int32 srid)
   at Microsoft.SqlServer.Types.SqlGeometry.GeometryFromText(OpenGisType type, SqlChars text, Int32 srid)
.
### The error may exist in uk/org/nbn/nbnv/api/dao/warehouse/TaxonObservationMapper.java (best guess)
### The error may involve uk.org.nbn.nbnv.api.dao.warehouse.TaxonObservationMapper.selectObservationsByFilterOrderedByDataset-Inline
### The error occurred while setting parameters
### SQL: SELECT obs.*, obs.id AS observationID, f.label AS location, f.resolutionID, r.label AS resolution, si.providerKey AS siteKey, si.name AS siteName, pt.name AS pTaxonName, pt.authority AS pTaxonAuthority, rr.name AS recorder, rd.name AS determiner FROM (SELECT o.*, 1 AS fullVersion FROM TaxonObservationDataEnhanced o INNER JOIN UserTaxonObservationID utoa ON utoa.observationID = o.id  INNER JOIN FeatureData ftd ON ftd.id = o.featureID WHERE ((utoa.userID = ?) AND ftd.geom.STIntersects(geometry::STGeomFromText(?, 4326)) = 1 AND ftd.geom.STTouches(geometry::STGeomFromText(?, 4326)) = 0 AND sensitive = 0) UNION ALL SELECT o.*, 0 AS fullVersion FROM TaxonObservationDataPublic o INNER JOIN FeatureData ftd ON ftd.id = o.featureID WHERE (o.id NOT IN ( SELECT utoa.observationID FROM UserTaxonObservationID utoa WHERE utoa.userID = ?) AND ftd.geom.STIntersects(geometry::STGeomFromText(?, 4326)) = 1 AND ftd.geom.STTouches(geometry::STGeomFromText(?, 4326)) = 0 AND sensitive = 0)) obs INNER JOIN FeatureData f ON f.id = obs.featureID INNER JOIN Resolution r ON r.id = f.resolutionID INNER JOIN TaxonData pt ON pt.taxonVersionKey = obs.pTaxonVersionKey LEFT OUTER JOIN SiteData si ON si.id = obs.siteID LEFT OUTER JOIN RecorderData rr ON rr.id = obs.recorderID LEFT OUTER JOIN RecorderData rd ON rd.id = obs.determinerID ORDER BY obs.datasetKey
### Cause: com.microsoft.sqlserver.jdbc.SQLServerException: A .NET Framework error occurred during execution of user-defined routine or aggregate "geometry":
System.FormatException: 24111: The well-known text (WKT) input is not valid.
System.FormatException:
   at Microsoft.SqlServer.Types.WellKnownTextReader.Read(OpenGisType type, Int32 srid)
   at Microsoft.SqlServer.Types.SqlGeometry.GeometryFromText(OpenGisType type, SqlChars text, Int32 srid)
.
; bad SQL grammar []; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: A .NET Framework error occurred during execution of user-defined routine or aggregate "geometry":
System.FormatException: 24111: The well-known text (WKT) input is not valid.
System.FormatException:
   at Microsoft.SqlServer.Types.WellKnownTextReader.Read(OpenGisType type, Int32 srid)
   at Microsoft.SqlServer.Types.SqlGeometry.GeometryFromText(OpenGisType type, SqlChars text, Int32 srid)

IT Officer
rECOrd (A Biodiversity Information System for Cheshire, Halton, Warrington and Wirral)
www.record-lrc.co.uk

8 (edited by BDeed 12-12-2013 16:39:41)

Re: REST taxonObservation Polygon WKT

Hi Luck,

I have tried something similar and Chrome is listing it as a timeout.
It actually ran using the client i am using, the result as far as i can tell was just a list of every dataset on the NBN! (hence the timeout).

Ben

Natural History & Biodiversity Data Enthusiast

9

Re: REST taxonObservation Polygon WKT

Thanks Ben, both old and new service seem to have all gone down. More testing tomorrow I think!

IT Officer
rECOrd (A Biodiversity Information System for Cheshire, Halton, Warrington and Wirral)
www.record-lrc.co.uk

10

Re: REST taxonObservation Polygon WKT

Hi, I have been going through some of these issues myself, it looks like you have your coordinates mixed up in this case, your WKT should look like this in your example

POLYGON ((-4.51376745766249 53.4073159895508, -4.51429736135726 53.4162958678331, -4.49926621884018 53.4166115845473, -4.49873947465632 53.4076316035263, -4.51376745766249 53.4073159895508))

We have tightened the restrictions on the taxonObservation calls such that a polygon no longer counts as a 'spatial' filter for this call as you could have a massive polygon encompassing the whole of the UK for example, we will see what we can come up with in the future but its going to stay that way for now unfortunately until we can come up with a sensible and workable solution to that issue.  I will change the text to reflect this change in the error message as well, thanks for spotting that.

The call is unfortunately slow as spatial queries are just that, slow and intensive processing, we can get around it for features that we already know about and increase the speed there i.e. features and grid references, but for user defined polygons such as this, the process is slow and unlikely to get faster quickly.

Hope that helps!

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

11

Re: REST taxonObservation Polygon WKT

Hi Matt,
Thank you for your reply.

Just to clarify, does this mean that those of us that need to query the gateway using a user defined polygon (e.g. point and buffer or site with buffer) can no longer do so under the new service?

Would it not be possible to test the polygon area before performing the query and so restrict polygons over a certain size?

Ben

Natural History & Biodiversity Data Enthusiast

12

Re: REST taxonObservation Polygon WKT

You can query with a polygon, you just need either a valid dataset filter or a valid taxanomic filter (ptvk, designation, output group, organisation supplied list) alongside it to make it work for now, the part about restricting polygons over a certain size is on the list of things to do, however Im not sure when we will get to it right now.

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

13

Re: REST taxonObservation Polygon WKT

Ok cheers Matt.

Natural History & Biodiversity Data Enthusiast

14

Re: REST taxonObservation Polygon WKT

Thanks Matt for the info.

IT Officer
rECOrd (A Biodiversity Information System for Cheshire, Halton, Warrington and Wirral)
www.record-lrc.co.uk

15

Re: REST taxonObservation Polygon WKT

Sorry, from Paula's email I assumed I wouldn't be getting a reply until next year so I hadn't checked!

I hadn't realised I'd reversed the lat-long - I converted it from the Easting/Northing and hadn't thought which way around the coordinates should go.

Thanks for clarifying the issue with polygons. I'll have to think about how best to extract the data we need then - our searches (and most people's I imagine) are generally a circular buffer around a point, so don't easily fit a grid square search.

Cheers,

Tim

16

Re: REST taxonObservation Polygon WKT

Hi Tim,

You should be able to generate coordinates for a circular polygon in most GIS, so have that system do the work for you then read in the coordinates. I am not sure what you use at COFNOD?

Natural History & Biodiversity Data Enthusiast

17

Re: REST taxonObservation Polygon WKT

No, that's not a problem, except for the maximum 10,000 character length of the URL string and the number of points in a large circle - I've written a search program in .NET that uses SQL Server 2008 spatial queries, and can easily create a WKT string from there (particularly now I've got the lat/long the right way around!).

The problem is that another filter (dataset, species group etc.) has to be specified as well, and we want to download everything so I can apply our own species categories at this end.

Tim

18

Re: REST taxonObservation Polygon WKT

Hi Tim,

We also have the same issue. Both Matt and Paula have said it's on the to-do list. Would the old combined JNCC designation list be useful? It may be worth you contacting Paula on this.

Natural History & Biodiversity Data Enthusiast

19

Re: REST taxonObservation Polygon WKT

One quick question, before I get lost somewhere else, what would be a viable maximum area size that you could work with, I may have to reduce the maximum size for performance reasons, but I would like to get some feedback of the sort of areas you are considering using so I can see if it causes any issues or not.

Thanks,
Matt

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

20

Re: REST taxonObservation Polygon WKT

For us 80% of our enquiries are either 1km (3,100) or 2km (12,500km2) point and buffer and the biggest one we run is the whole county of Cheshire and 10km point and buffer.

Thanks,
Luck

IT Officer
rECOrd (A Biodiversity Information System for Cheshire, Halton, Warrington and Wirral)
www.record-lrc.co.uk

21

Re: REST taxonObservation Polygon WKT

Just a note that a simple bounding box datasets query is very slow or not working at all.

https://data.nbn.org.uk/api/taxonObserv … on=POLYGON ((-2.676420445063187 53.378815657299505, -2.6192334599467686 53.379125980347332, -2.619755819813232 53.415094098408538, -2.6769910369200143 53.414783370617975, -2.676420445063187 53.378815657299505))

IT Officer
rECOrd (A Biodiversity Information System for Cheshire, Halton, Warrington and Wirral)
www.record-lrc.co.uk

22

Re: REST taxonObservation Polygon WKT

BDeed wrote:

You should be able to generate coordinates for a circular polygon in most GIS, so have that system do the work for you then read in the coordinates.

Via the old web-services we provide an online circular search facility via a simple html form, asking for OS grid ref for the centre, plus the radius, which is what the old web-service accepted as input, and was ideal.

If the javascript/php/whatever now has to calculate a rough circle, and do all the OS->WGS84 conversions itself, that isn't nearly so convenient, and certainly not 'restful' for every poor webmaster that has to work out the code. (Has anyone written a javascript function to do this please that we can all use?).

Surely the NBN can re-instate this lost piece of functionality?

Keith

23 (edited by kbalmer 20-01-2014 13:43:26)

Re: REST taxonObservation Polygon WKT

In case anyone needs it this javascript will produce a rough WGS84 circular polygon from OS easting, northing, radius in metres and the number of points to form the circle.

<script type="text/javascript" src="http://www.bnhs.co.uk/focuson/grabagridref/html/BNHS_coords.js"></script>

<script type="text/javascript">
function CircularPolygon(centre_x, centre_y, radius, points) 
  {
  var step  = 360 / points; // note that 360 must be exactly divisible by points!
  var angle = 0;
  var polygon = "POLYGON((";
  for (var i=0; i<=points; i++) // the first point must be calculated twice to close the polygon
    {
    var x = parseInt(centre_x) + Math.round(Math.cos(angle * (Math.PI/180)) * radius);
    var y = parseInt(centre_y) + Math.round(Math.sin(angle * (Math.PI/180)) * radius);

    var os = new OSRef(x,y);
    var latlng = os.toLatLng(os);
    latlng.OSGB36ToWGS84();

    polygon = polygon + latlng.lng + "%20" + latlng.lat;
    if (i < points)
      {
      polygon = polygon + ",";
      }
    angle = angle + step;
    }
  polygon = polygon + "))";
  return polygon;
  }
</script>

// Useage:
var polygon = CircularPolygon(512350, 245650, 1000, 12); // A 12-sided circle with radius 1km centred on TL123456

Keith

24

Re: REST taxonObservation Polygon WKT

To follow on from Keith I agree that since buffer requests will be common it would be great to be able to submit a point and buffer distance rather than try and work how to create one myself

25 (edited by dan.jones 19-05-2014 11:20:08)

Re: REST taxonObservation Polygon WKT

Hi Matt.

Has anything moved on this restriction?

I've just tried running a query with a gridRef (1km) and with a WKT polygon equivalent to a 1km square and both returned the

{"status":"Must Supply at least one type of filter; dataset (key list), spatial(featureID, gridRef or polygon) or taxon (PTVK list, Output Group, Designation or Organisation Supplied List)","success":false}

error.

Any thoughts? Thanks. Dan