1

Topic: REST filter not working

I am in the process of preparing a report to funders which must include details of the use of our datasets on the Gateway.

I want to see the 'View and REST services Log' since 1 April 2014.  When I look at the 'filtered' result, I find it is exactly the same as the unfiltered result, including items that pre-date 1 April 2014.

Once again, we find that a core function of the Gateway (almost the only thing in G5 that can be said to be an advance on G4 IMHO) fails to work correctly when it is needed.

In keeping with the inconsistency of function that pervades G5, inconsistency being one of the few consistent features of the site, if I choose the 'Download log' and filter it in an identical fashion the results are correct.

Please fix it.

Murdo

2

Re: REST filter not working

Hi, thanks for reporting this error, I cant replicate an issue with the list itself when looking at the list on the page itself, I have found a few issues with the download as CSV link however (it tries to download several times with what looks like a history of parameters put into the filter) which might explain what your are seeing here.

So to clarify are you seeing this happen on the page or in the CSV download?

Matt

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

3

Re: REST filter not working

On the page, repeatedly, with different datasets and different dates, while the Download log performed correctly.  I did not try the csv link - why would I when the filter was clearly not working?

Inconsistency on G5 is one of the few things we can rely on, like death and taxes, so it comes as a surprise that right now it it doing exactly the same as it did on Saturday.

The most worrying thing, of course, though this time *not* surprising, is that after I reported my problem you find even more problems lurking unnoticed.  All this coming close to a year and a half after G5 went live.  Why should anyone have any confidence in *anything* that we see here, as it is obvious that the elementary testing that should have been done 2y or so ago and for every relevant change in coding has not happened.

Murdo

4

Re: REST filter not working

Would you be able to let me know what datasets are not working in your experience, and would you mind if I could make myself a dataset administrator to try and find the cause of any bugs that may be lurking on this page. The REST logs are not a well travelled section of the sites so it can be very easy to miss any issues, especially when it appears to not be a specific error and related to data as it seems to be in this case.

Matt

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

5

Re: REST filter not working

Thanks, Matt.  I have just checked the situation with the Badger and FLLP sets.  It still fails, and the .csv download is also not filtered.

I am not preparted to make you an administrator.  If this turns out to be a problem only with HBRG sets, I will eat my boots on live TV.  The data have passed your own validation checks (speak to Dario, who dealt with our updates in the past couple of weeks), so if there is a problem related to data, you have just uncovered another problem in your system.

What I fail to comprehend is why two apparently identical interfaces (Download log and View/REST) which should be using the same filtering routine should give such different results.

In passing, the delay in replying is down to a failure of the Forum system to notify me of your response, despite the fact I am flagged as subscribing, and not for the first time recently.  I also had no notification of responses to the item posted by Chris Johnson the other day, nor of yesterday's posts in the UKSI forum.  So yet another malfunction for someone to investigate.

Murdo

6

Re: REST filter not working

Hi Murdo, when I say its related to the data I mean that there may be an issue with the logs themselves and not your data coming into the gateway, the issue may just be there are a lot of logs, a specific number is being ill formatted at some point, it have passed some limit, etc... that cases a bug. I will keep looking and I may find the issue in the mean time, I will deploy a fix for the bug I found later on today, it may or may not affect your issue however.

With regards to the forum it is not under our control and you would need to contact Mike using the email under the help link on the forums.

Matt

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

7

Re: REST filter not working

I have just had to resort to downloading the .csv files for our datasets.  Naively, I assumed that I would be able to filter for the past 12 months myself, Excel being well endowed with functions and commands that are designed to make life easy for users.

Then, of course, reality intruded.  I find that the download date in the .csv is of the format 'Wed Feb 18 2015 13:41:03 (GMT)'.  That, of course makes any use of the Excel date-manipulation functions to covert text to date utterly useless.

Yes, I could spend an hour or so writing (and *testing*) my own formulae to extract the relevant bits of the string, and format a useable date value, and then filter the rows, but I do have other rather more important calls on my time, and once again I ask myself if anyone involved in designing the G5 interface gave any thought at all to how real people in the real world would want to use it effectively and efficiently.

Keep a dog and bark yourself? Someone obviously thinks that is a sensible option.

M.

8

Re: REST filter not working

I have changed the date output format there so it should be more useful to you, if you are having issues with the date filter on that page still however you can use the API directly using the call

https://data.nbn.org.uk/api/apiViews/{D … yyyy-mm-dd

Matt Debont
Application Developer
Joint Nature Conservation Committee, Monkstone House, City Road, Peterborough PE1 1JY, UK

9

Re: REST filter not working

Thanks, Matt, but I have spent so long on this already that I am reporting the unfiltered views.

M.

10

Re: REST filter not working

Murdo,

Our developers cannot replicate the issue you are seeing. Without administrator access, or a step by step detailed account of the issue there is little more we can do. I am sorry that we currently cannot help you any further on this issue.

Paul Gilbertson,
Technical Project Manager, JNCC

JNCC Technical Project Manager

11

Re: REST filter not working

You have (or someone has) apparently helped inadvertently, Paul.  I have just tried the View/REST filter exactly as I did last week, and the filter now works as one would expect.  That is, when I ask for results since 1 April 2014, that is exactly what I get.

I do not believe that this problem is (was) specific to the HBRG sets, and the fact that the problem seems to have been fixed without admin privileges supports that view.  The filter on dates should not have anything to do with the contents of our datasets.  Only two fields would seem to be relevant - the dataset key and the date of access.  Neither of these has anything to do with files that I can access as administrator - they are in your log.

Murdo

PS - once again, I got no notification of Paul's post.

12

Re: REST filter not working

Murdo,

We issued two patches on the 11th and 12th, but as we were unable to replicate your issues it is unknown if these would have helped.

Requesting administrator access to datasets is not necessarily to do with the specific raw records, but also to with the access patterns that are logged, and many other variables which are specific to a user or organisation. Without being able to replicate the conditions that your issues occur in it is very difficult for us to track down specific and often subtle problems. Hand parsing through thousands of log entries rather than being able to trace the data through all the code paths is an inefficient use of highly constrained developer resource.

I am glad that the issue seems resolved.

Paul Gilbertson

JNCC Technical Project Manager