1

Topic: How to fine tune output from Dynamic Report Explorer

I have some basic data capture working against the test warehouse on Drupal 7 with the iForms module. I am using the Dynamic Report Explorer to show the latest records on the website and have configured the columns as described on p.130 of the 'Flight Manual', to show only what I need.

Is there a way to restrict the number of records that are extracted? I really only want to show the last 'n' records, or maybe records for the last 'n' days.

What is the best way to manipulate the Species name, that is being shown like 'Neomys fodiens | Eurasian Water Shrew'. All I really want is the English name.

Is the formatting/layout being done on my webserver, or at the warehouse? If the former, I am happy to over-ride the PHP, if someone can direct me to the appropriate code.

Maybe one of the other 'Report types' (e.g. Report grid) would be more useful, but I can'tsee where that is documented.

Thanks, Chris

Chris Dee
Garganey Consulting

2

Re: How to fine tune output from Dynamic Report Explorer

Hi Chris

First and foremost, output is determined by the report that is being run on the warehouse. This determines the columns being retrieved from the database and the parameters it supports. You can see this at https://code.google.com/p/indicia/sourc … e_list.xml

To show the last 'n' records you need to introduce a 'limit' clause in to the query. As far as I can see, this is not possible in this case as it is overriden by the pager code. I don't seem to be able to turn the pagers off. There might be a lingering bug if this is a use-case that has not been tested till now.

To show dates for the last n days needs a 'where' clause in the query. The report does not contain a date parameter but we might be able to supply one. The report_grid function can take a filters parameter which you could provide in the Form Structure.
Theoretically, I thought

@filters={"date_start":"2015-02-04"}

would select records from that one date or

@filters={"date_start":["2015-02-04","2015-02-05","2015-02-06"]}

would select from all those dates. I haven't got this to work in the course of trying to give you an answer here but this is an option I have never tried before so I might be doing it wrong. As for a date which is of the form "date > now - offset", I think that is beyond what we can do through the  Form Structure. Writing a custom report would be a solution.

Regarding the species name, you can see from the report that it is returned by the query snippet

CASE WHEN o.preferred_taxon=o.default_common_name then o.preferred_taxon ELSE o.preferred_taxon || COALESCE(' | ' || o.default_common_name, '') END

I.e if the preferred name is the same as the common name then you just get that; if there is no common name you just get the preferred name; if there is a preferred name and a common name then you get both.

Since this is your starting point, there are no built in options I can think of in the client code to manipulate this. Again a custom report would get you what you want. Alternatively, there could be another report in the library which is better suited to your needs.

The warehouse just provides the data needed and it is formatted by the code in the client_helpers library on your server, notably in the functions of the report_helper file.

Sorry this does not really include any easy answers. The way forward may depend on just what you want to achieve. You could
- go back to the simpler Report Grid form, perhaps with the library/occurrences/occurrences_list.xml report which incudes a common name column and date parameters.
- try to use a new filtering feature which is mentioned in passing at http://indicia-docs.readthedocs.org/en/ … orts.html. 
- create a custom report.

I'm going to have to leave it at that as I had so much else planned to do today.

Jim Bacon.

3

Re: How to fine tune output from Dynamic Report Explorer

Hi Jim

Thanks, that's helpful and sorry if this is all very obvious to you, but even as an experienced web developer, this is not very clear from the documentation. There is obviously plenty of functionality built into Indicia, and what I am trying to do doesn't seem particularly ground-breaking, yet doesn't seem to be well-supported (or maybe just not well-documented).

Basically I want to add a basic records capture form to an existing Drupal 7 website, offering a pre-defined list of species (mammals, reptiles and amphibians) with a pre-defined list of sites or the ability to select a location from the map, and of course other standard fields (observer, date, count, etc.). On another page, I want to display the latest 'n' submitted records from that survey, possibly with some built-in species filtering. Ideally, all this needs to work without any user login being required.

Regarding your three options:

- Report Grid Form I could only get that to generate a 'Run report' button. I want the records to be rendered directly in the content, like the Dynamic Report Explorer does. I don't want the visitor to have to click 'Run report'. Have I missed an option?
- Filtering feature Do you mean the [standard_params] | @filter-date_age=... ? That demands 'Easy login' which is something I was hoping to avoid.
- Custom Report I'm happy to have a go at this, but how do I install a custom report. Is there a UI on the Warehouse to load up the XML, or do I need FTP access to the test warehouse (which I don't have). How would that then get moved to the live instance?

Thanks, Chris

Chris Dee
Garganey Consulting

4

Re: How to fine tune output from Dynamic Report Explorer

Hi Chris.

Believe me, it is little more obvious to you than me. In attempting to answer your question I was combining a bit of knowledge with trying to read the code. Some of the time I was using a debugger to inspect what was going on. I always struggle when I try to do something a bit different. I spent several hours on the answer which is why I had to quit in the end.

A run report button may imply that the report is needing some parameters to be supplied before it runs. If that is the case and the parameter values are known in advance then the configuration of the form allows you to supply those values so that the report can run without operator input. I'm ad-libbing a bit there and haven't done a test.

Custom reports cannot be installed without server access which we do not grant to anyone not employed by CEH. You would have to work with me to arrange that which, of course, I am happy to do.

Jim Bacon.

5

Re: How to fine tune output from Dynamic Report Explorer

Thanks Jim

Yes, the Report Grid Form does come up with parameters - "Survey", "Vague date processing" and "Download", so I provided those in the Preset parameter values: box and now the results are displayed immediately.

For future reference for others finding this posting, and because I couldn't find them documented anywhere, the names of the parameters that needed to be supplied were not the HTML form field names (report-xxx), but merely the xxx (in my case survey=, vagueDate= and download=). I'm not sure if that is the general case, but it worked for these three fields.

I'll be in touch separately about installing a custom report.

Thanks, Chris Dee

Chris Dee
Garganey Consulting

6

Re: How to fine tune output from Dynamic Report Explorer

Good work.

If you look at the report you are using, which can be found in the code repository (now on Github https://github.com/Indicia-Team/warehou … /reports), you will see there is a <params> section which lists all the parameters including their names. Since there can be any number of reports and the parameter names are arbitrarily chosen by the report writer, the source is the most reliable place to obtain this information.

For info, I am on leave next week.

Jim Bacon.

7

Re: How to fine tune output from Dynamic Report Explorer

Hi

I am using the report named 'Freeform report' in Form category 'Reporting', but cannot find that in the GitHub location mentioned.

Where can I find its definition ?

Thanks, Chris

Chris Dee
Garganey Consulting

8

Re: How to fine tune output from Dynamic Report Explorer

Hi

You are mixing up two different things there. When you select 'Freeform report' you are choosing  a prebuilt form (https://github.com/Indicia-Team/client_ … report.php) which you can then configure to use any of the report files you want from the location mentioned previously.

Jim.

9

Re: How to fine tune output from Dynamic Report Explorer

Thanks, I now understand that. I am using Survey details and do indeed see survey_details.xml in the repository.

Rich though it is, if the Freeform Report "Bands framework" lacks all the functionality I need to produce a smart report, what is the recommended way to incorporate my own PHP report generation code, using the data provided by survey_details?

Thanks, Chris

Chris Dee
Garganey Consulting

10

Re: How to fine tune output from Dynamic Report Explorer

You would be looking at creating your own pre-built form I would say.
See http://indicia-docs.readthedocs.org/en/ … index.html

If the Freeform Report is nearly what you want then sub-classing it can be a very economical approach. Alternatively you could copy and modify it.

Jim Bacon

11

Re: How to fine tune output from Dynamic Report Explorer

Excellent, thanks - I've made substantial progress with this information. Good documentation too. Just one niggle - sentence 2 of the first paragraph of http://indicia-docs.readthedocs.org/en/ … -node.html, is presumably for Drupal 6 (with which I am unfamiliar). For Drupal 7 it should read something like "On the admin menu, select Content > Add content > Indicia pages to access the page used to add and configure an Indicia form."

I am using the report Library / Occurences / Occurences download to extract the data. I can't find a generic report that a) sorts the output by occurence date (i.e. date_end) and parameterises a limit clause. I can write one, but don't want to duplicate something that is already there that I've just not found. If I write a custom report, do I just email it to you to install on the test warehouse?

Thanks, Chris

Chris Dee
Garganey Consulting

12

Re: How to fine tune output from Dynamic Report Explorer

Thanks for the feedback. I've updated the documentation.

I don't know of a report that meets the criteria you have set out. I note that in many cases a user can sort tables by clicking on the column heading and that the site admin can alter the number of rows in output tables by a configuration setting called "Items per page".
I also have a recollection that you can pass a limit value through to a report request in the extraParams part of the options so that any report can be limited without needing to write that in to the report.

If you do write a custom report then send it to me and I can install it on the test warehouse.

Jim Bacon.

13 (edited by cwdee 13-10-2015 06:40:10)

Re: How to fine tune output from Dynamic Report Explorer

The data presentation is not tabular, so clicking on the table headers isn't an option. I'd like to go the custom report route.

Are there two formats of the report XML for custom reports? In library/occurences/occurences_download.xml I see an abstracted form of the query with XML constructs like:
<table>
  <subTable>
    <tabColumn />
  </subTable>
</table>
with
<mergeTabColumn /> and <attributes /> elements,

whereas in others (such as library/occurrences/occurrence_list.xml) and documented here, I see a less-abstracted form of the SQL like:
<query>...</query>
<order_bys>...</order_bys>
<columns>...</columns>

Should I be using the latter?

Is this difference in syntax also the reason why Occurences download provides columns named like lt0_location_name, lt0_entered_sref, lt0_end_date, whereas Occurence list provides location_name, entered_sref, end_date? If not what do the lt[0-9]_ prefixes indicate?

Are the database table structures documented somewhere? I couldn't find a schema on the indicia-docs website. Specifically, I need to know how to join the taxon group name (taxon_group ?).

Thanks for your ongoing help - I think we are nearly there, Chris

Chris Dee
Garganey Consulting

14 (edited by Jim Bacon 13-10-2015 15:51:31)

Re: How to fine tune output from Dynamic Report Explorer

I've only ever seen and used the documented form of the report files so I'd have to recommend going with that. Your second example is also 2 years more recent than the former.

There is a very old schema diagram at https://code.google.com/p/indicia/wiki/DataModel. I am investigating whether I can generate something more up to date.

Given an occurrence_id of xxx there are two ways to get the associated taxon group. The long, slow way

SELECT tg.title 
FROM occurrences o
JOIN taxa_taxon_lists ttl ON ttl.id = o.taxa_taxon_list_id
JOIN taxa t ON t.id = ttl.taxon_id
JOIN taxon_groups tg ON tg.id = t.taxon_group_id
WHERE o.id = xxx

or using the cache

SELECT taxon_group 
FROM cache_occurrences 
WHERE id = xxx

Jim Bacon.

15 (edited by Jim Bacon 28-10-2015 09:09:56)

Re: How to fine tune output from Dynamic Report Explorer

To get the value of a single-value, text, sample attribute with id of 458 for a sample_id of xxx the join looks like the following.

SELECT sav.text_value
FROM samples s
JOIN sample_attribute_values sav ON sav.sample_id = s.id AND sav.sample_attribute_id = 458
WHERE s.id = xxx

To get the value of a single-value, integer, occurrence attribute with id of 77 for an occurrence_id of xxx the join looks like.

SELECT oav.int_value
FROM occurrences o
JOIN occurrence_attribute_values oav ON oav.occurrence_id = o.id AND oav.occurrence_attribute_id = 77
WHERE o.id = xxx

However, this hard codes the attribute id in to the report which can be a pain. For better flexibility and portability, you can specify the attributes you want as parameters, see http://indicia-docs.readthedocs.org/en/ … attributes

Jim Bacon