1 Topic by Gordon Barker

  • Gordon Barker
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 344

Topic: First attempt with web services

Hi

Having my first go at using web services and running into some problems Using Geocortex Workflow Designer which may not mean anything to anyone.

Query string is "https://data.nbn.org.uk/api/taxonObserv … ip=overlap" which works in a browser but gives
"Error: Unhandled exception: 'One or more errors occurred.' in activity '1.12: WebRequest'.
  One or more errors occurred.
    The remote server returned an error: (401) Unauthorized." when put in a WebRequest.

Possibly the login isn't working properly, but it seems to process. Using "https://data.nbn.org.uk/api/user/login? … ember=true"

Any suggestions?

Thanks

Gordon Barker
Biological Survey Data Manager
National Trust

2 Reply by Gordon Barker (edited by Gordon Barker 23-06-2015 15:45:08)

  • Gordon Barker
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 344

Re: First attempt with web services

Hopefully this will make a bit more sense to someone:

I have copied what I think is the relevant piece of code from the xaml file:

 <Assign.To>
        <OutArgument x:TypeArguments="x:String">[string3]</OutArgument>
      </Assign.To>
      <Assign.Value>
        <InArgument x:TypeArguments="x:String">https://data.nbn.org.uk/api/taxonObservations/322459760</InArgument>
      </Assign.Value>
    </Assign>
    <gwa1:WebRequest Headers="{x:Null}" ResponseHeaders="{x:Null}" sap:VirtualizedContainerService.HintSize="339,100" Result="[byteArray2]" Uri="https://data.nbn.org.uk/api/user/login?username=gordonbarker&amp;password=*********&amp;remember=true" />
    <gwa1:WebRequest Headers="{x:Null}" ResponseHeaders="{x:Null}" sap:VirtualizedContainerService.HintSize="339,100" Method="GET" Result="[byteArray1]" Uri="[string3]" />
    <gwa1:GetTextFromBytes Bytes="[byteArray1]"

This should return the details for record 322459760, but gives a 401 authentication error. I suspect that although I can get a response from the first web request to log in to the web services, I don't know how to pass that back again with the request for data. Any ideas?

Gordon Barker
Biological Survey Data Manager
National Trust

3 Reply by matt.debont

  • matt.debont
  • Veteran
  • Offline
  • From: Peterborough
  • Registered: 04-11-2013
  • Posts: 145

Re: First attempt with web services

Hi Gordon, I don't know about the ins and outs of this particular package, but it dose look like its not including the cookie from the login request with subsequent requests, you would need to look into how to include that cookie with requests as the authentication is cookie based.

Also you may want to remove the password from that xaml file

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

4 Reply by Gordon Barker

  • Gordon Barker
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 344

Re: First attempt with web services

Thanks Matt,

Do you have any examples of how that is done in other packages? I might be able to work it out from that.

Gordon Barker
Biological Survey Data Manager
National Trust

5 Reply by matt.debont

  • matt.debont
  • Veteran
  • Offline
  • From: Peterborough
  • Registered: 04-11-2013
  • Posts: 145

Re: First attempt with web services

Hi Gordon the only example I have written up is in Java, I don't know if that will help you here however https://data.nbn.org.uk/Documentation/W … ogging_In/

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

6 Reply by Gordon Barker

  • Gordon Barker
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 344

Re: First attempt with web services

Should I return the token value from the login response as a header?

Gordon Barker
Biological Survey Data Manager
National Trust

7 Reply by matt.debont

  • matt.debont
  • Veteran
  • Offline
  • From: Peterborough
  • Registered: 04-11-2013
  • Posts: 145

Re: First attempt with web services

Yes, I think you should be returning it as part of the cookie in the header, i.e. Cookie: nbn.token_key=token or something similar to that I think should do it

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

8 Reply by Gordon Barker

  • Gordon Barker
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 344

Re: First attempt with web services

I can't work out how to do this correctly. I have options to pass back data as Header, Parameter or Value but can't get any of these to work. The other field is Credentials, might there be a way for me to use this?

Gordon Barker
Biological Survey Data Manager
National Trust

9 Reply by matt.debont

  • matt.debont
  • Veteran
  • Offline
  • From: Peterborough
  • Registered: 04-11-2013
  • Posts: 145

Re: First attempt with web services

Credentials is to do with Basic HTTP authentication which I don't think is supported through the API.

If you continue to have issues it might be worth seeing if you can get any support from the creators of this package as I can't see anything obvious but the general form for setting cookies is in the request header, the server sends the requestor a

Set-Cookie: name=value

In the response header, to keep the cookie in the connection, the requestor has to respond with the following in its header;

Cookie: name=value

That is the basic way in which these cookies are supposed to work, also make sure you are using HTTPS as well rather than HTTP as I don't know how this package would respond to the redirect that goes on for HTTP->HTTPS.

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

10 Reply by BDeed

  • BDeed
  • Veteran
  • Offline
  • From: Bolton
  • Registered: 01-03-2011
  • Posts: 451

Re: First attempt with web services

Hi Gordon,
I don't know if it will be of any use for your requirements but i found this example VB code connecton script particularly useful.
http://forums.nbn.org.uk/viewtopic.php?id=4987

Natural History & Biodiversity Data Enthusiast

11 Reply by Gordon Barker (edited by Gordon Barker 02-07-2015 11:22:33)

  • Gordon Barker
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 344

Re: First attempt with web services

Thanks Ben,
I think I'm getting closer. I can use the vb script but the json serialiser causes an error so i took it out just to return the web response. Probably not what should be done becausethat has caused a Error: Unhandled exception: 'Exception has been thrown by the target of an invocation.' . Currently I have

  Dim cc As New CookieContainer()
        Dim LoginRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/user/login?username=" & UserName & "&password=" & Password & "&remember=false")
        LoginRequest.CookieContainer = cc
        Dim LoginResponse As HttpWebResponse = LoginRequest.GetResponse()
        If LoginResponse.StatusCode = HttpStatusCode.OK AndAlso LoginResponse.Cookies.Count > 0 Then
            'Log in
            Dim c As Cookie = LoginResponse.Cookies(0)
            Dim DataRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/taxonObservations/322459760")
            Dim cc2 As New CookieContainer()
            cc2.Add(New Cookie(c.Name, c.Value, c.Path, c.Domain))
            DataRequest.CookieContainer = cc2
            'Get data
            Dim wr As HttpWebResponse = DataRequest.GetResponse()
            'Log out
            Dim LogoutRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/user/logout")
            Dim cc3 As New CookieContainer()
            cc3.Add(New Cookie(c.Name, c.Value, c.Path, c.Domain))
            LogoutRequest.CookieContainer = cc3
            Dim LogoutResponse As HttpWebResponse = LogoutRequest.GetResponse()
            Return wr
        End If

        Return Nothing
Gordon Barker
Biological Survey Data Manager
National Trust

12 Reply by BDeed

  • BDeed
  • Veteran
  • Offline
  • From: Bolton
  • Registered: 01-03-2011
  • Posts: 451

Re: First attempt with web services

Hi Gordon,

The wording of your error is new on me and its been a while since i played with this but i do remember having a lot of issues handling the json. If i remember rightly it's not formed quite as you'd expect (chuck your url into Chrome to see the response).

If you can give me some time i'll see if i can dig out the code we use.

In the meantime there seem to be some useful suggestions here: http://stackoverflow.com/questions/6481 … r-mscorlib

Natural History & Biodiversity Data Enthusiast

13 Reply by Gordon Barker

  • Gordon Barker
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 344

Re: First attempt with web services

If i can get the response out as a string that would be fine because I can convert later on (at least I can for a single observation, I'll work out why I can't do multiples later). I have a suspicion I might be trying to pass the response back in a form different to how it is expected, or in a way that ins't possible and need to put something in to replace the json serialiser I took out

Gordon Barker
Biological Survey Data Manager
National Trust

14 Reply by BDeed

  • BDeed
  • Veteran
  • Offline
  • From: Bolton
  • Registered: 01-03-2011
  • Posts: 451

Re: First attempt with web services

It is possible as i remember outputting as text a lot while i tried to work out how to do the de-serialization. You can't be far out as i worked heavily of Tim's code to start.

Natural History & Biodiversity Data Enthusiast

15 Reply by BDeed (edited by BDeed 02-07-2015 12:57:03)

  • BDeed
  • Veteran
  • Offline
  • From: Bolton
  • Registered: 01-03-2011
  • Posts: 451

Re: First attempt with web services

Gordon,

The below is some rough working code for grabbing records for a grid reference. It can output to xml or csv. It should be fairly simple (..) to adapt for your purposes. The key point is the TaxonObservation class we use to manage the json (see end).

    
Public Function GetTaxonObservation(ByVal UserName As String, ByVal Password As String, ByVal GridReference As String) As List(Of TaxonObservation)
        Dim cc As New CookieContainer()
        Dim LoginRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/user/login?username=" & UserName & "&password=" & Password & "&remember=false")
        LoginRequest.CookieContainer = cc
        Dim LoginResponse As HttpWebResponse = LoginRequest.GetResponse()

        If LoginResponse.StatusCode = HttpStatusCode.OK AndAlso LoginResponse.Cookies.Count > 0 Then
            'Log in
            Dim c As Cookie = LoginResponse.Cookies(0)
            Dim DataRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/taxonObservations?gridRef=" & GridReference)
            Dim cc2 As New CookieContainer()
            cc2.Add(New Cookie(c.Name, c.Value, c.Path, c.Domain))
            DataRequest.CookieContainer = cc2

            'Get data
            Dim wr As HttpWebResponse = CType(DataRequest.GetResponse(), HttpWebResponse)
            Dim wrStreamd As StreamReader = New StreamReader(wr.GetResponseStream(), Encoding.UTF8)
            Dim wrString As String = wrStreamd.ReadToEnd
            Dim objStreamWriter As New StreamWriter("C:\tmp\nbn5.xml")

            'Works
            Dim js As New JavaScriptSerializer
            Dim recs As List(Of TaxonObservation) = js.Deserialize(Of List(Of TaxonObservation))(wrString)

            'outputs records as xml
            Dim x As New XmlSerializer(recs.GetType)
            x.Serialize(objStreamWriter, recs)
            objStreamWriter.Close()

            'Log out
            Dim LogoutRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/user/logout")
            Dim cc3 As New CookieContainer()
            cc3.Add(New Cookie(c.Name, c.Value, c.Path, c.Domain))
            LogoutRequest.CookieContainer = cc3
            Dim LogoutResponse As HttpWebResponse = LogoutRequest.GetResponse()
            'Return recs
        End If
        Return Nothing
    End Function

and the class;

Public Class TaxonObservation
    Public Property observationID As Integer
    Public Property fullVersion As Boolean
    Public Property datasetKey As String
    Public Property surveyKey As String
    Public Property sampleKey As String
    Public Property observationKey As String
    Public Property siteKey As String
    Public Property siteName As String
    Public Property featureID As Integer
    Public Property location As String
    Public Property resolution As String
    Public Property taxonVersionKey As String
    Public Property pTaxonVersionKey As String
    Public Property pTaxonName As String
    Public Property pTaxonAuthority As String
    Public Property startDate As String
    Public Property endDate As String
    Public Property recorder As String
    Public Property sensitive As Boolean
    Public Property absence As Boolean
    Public Property dateTypekey As String
End Class
Natural History & Biodiversity Data Enthusiast

16 Reply by BDeed (edited by BDeed 02-07-2015 13:42:01)

  • BDeed
  • Veteran
  • Offline
  • From: Bolton
  • Registered: 01-03-2011
  • Posts: 451

Re: First attempt with web services

Gordon,

Just had a quick play and this works to bring back and output your json into a straight text file.

*just to note this is very rough. You still need to create your form and some parts of the code are superfluous!

Imports System.Net
Imports System.IO
Imports System.Text
Imports System.Configuration
Imports System.Runtime.CompilerServices
Imports System.Xml
Imports System.Xml.Serialization
Imports System.Xml.Xsl
Imports System.Xml.XPath

Public Class Form1

    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load

    End Sub

    Public Function GetRec(ByVal UserName As String, ByVal Password As String) As List(Of TaxonObservation)
        Dim cc As New CookieContainer()
        Dim LoginRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/user/login?username=" & UserName & "&password=" & Password & "&remember=false")
        LoginRequest.CookieContainer = cc
        Dim LoginResponse As HttpWebResponse = LoginRequest.GetResponse()

        If LoginResponse.StatusCode = HttpStatusCode.OK AndAlso LoginResponse.Cookies.Count > 0 Then
            'Log in
            Dim c As Cookie = LoginResponse.Cookies(0)
            Dim DataRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/taxonObservations/322459760")
            Dim cc2 As New CookieContainer()
            cc2.Add(New Cookie(c.Name, c.Value, c.Path, c.Domain))
            DataRequest.CookieContainer = cc2

            'Get data
            Dim wr As HttpWebResponse = CType(DataRequest.GetResponse(), HttpWebResponse)
            Dim wrStreamd As StreamReader = New StreamReader(wr.GetResponseStream(), Encoding.UTF8)
            Dim wrString As String = wrStreamd.ReadToEnd

            Dim file As StreamWriter
            file = My.Computer.FileSystem.OpenTextFileWriter("C:\tmp\nbn5.txt", True)
            file.WriteLine(wrString)
            file.Close()

            MsgBox(wrString)

            'Log out
            Dim LogoutRequest As HttpWebRequest = HttpWebRequest.Create("https://data.nbn.org.uk/api/user/logout")
            Dim cc3 As New CookieContainer()
            cc3.Add(New Cookie(c.Name, c.Value, c.Path, c.Domain))
            LogoutRequest.CookieContainer = cc3
            Dim LogoutResponse As HttpWebResponse = LogoutRequest.GetResponse()

        End If
        Return Nothing
    End Function

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        If TextBox1.Text = "" Then
            MessageBox.Show("Please enter NBN username")
        ElseIf TextBox2.Text = "" Then
            MessageBox.Show("Please enter NBN password")
        Else
            Dim rec = GetRec(TextBox1.Text, TextBox2.Text)
        End If
    End Sub
End Class

Public Class TaxonObservation
    Public Property observationID As Integer
    Public Property fullVersion As Boolean
    Public Property datasetKey As String
    Public Property surveyKey As String
    Public Property sampleKey As String
    Public Property observationKey As String
    Public Property siteKey As String
    Public Property siteName As String
    Public Property featureID As Integer
    Public Property location As String
    Public Property resolution As String
    Public Property taxonVersionKey As String
    Public Property pTaxonVersionKey As String
    Public Property pTaxonName As String
    Public Property pTaxonAuthority As String
    Public Property startDate As String
    Public Property endDate As String
    Public Property recorder As String
    Public Property sensitive As Boolean
    Public Property absence As Boolean
    Public Property dateTypekey As String
End Class
Natural History & Biodiversity Data Enthusiast

17 Reply by Gordon Barker

  • Gordon Barker
  • Veteran
  • Offline
  • Registered: 15-05-2006
  • Posts: 344

Re: First attempt with web services

Brilliant Ben, that seems to have worked. I swapped in the Get data section into what i already had, as it already functioned without that section, and it has output a text file with a lot of records in it, which I at least partly know how to do the next stage with...

Gordon Barker
Biological Survey Data Manager
National Trust

18 Reply by BDeed

  • BDeed
  • Veteran
  • Offline
  • From: Bolton
  • Registered: 01-03-2011
  • Posts: 451

Re: First attempt with web services

If you're getting a lot of records with what i just posted then something's gone wrong! It should only bring back the record you were interested in. Still so long as it's helpful.

Natural History & Biodiversity Data Enthusiast

19 Reply by BDeed

  • BDeed
  • Veteran
  • Offline
  • From: Bolton
  • Registered: 01-03-2011
  • Posts: 451

Re: First attempt with web services

Sorry just read your post properly. It'd depend on the api (see datarequest section) you are using. If there are any specific things your trying to achieve let me know and i'll see if i can help.

Natural History & Biodiversity Data Enthusiast