1

Topic: Records not removed from Dynamic Report Explorer form

Experimenting with different approaches, I've got the  "Browse Records" feature enabled, and I've also built an "Explore Records" form as per the advanced tutorial, using the Dynamic Record Explorer.

Now, I had entered a number of dummy records for testing which I've now deleted at the Warehouse end.

These have accordingly disappeared from my "Browse Records" grid, and, incidentally, from my Distribution Map page.  But they remain visible in the "Explore Records" form, both on the map and in the grid.

I've cleared caches.

Cheers

Alan

Alan Hale
Aberystwyth

2 (edited by Alan Hale 13-03-2013 17:30:27)

Re: Records not removed from Dynamic Report Explorer form

Ah - just seen the solution given by Jim in the posting earlier today entitled "Data deletion".

So I've rebuilt the cache as suggested and that has done the trick.

Teach me to check other postings first.

Thanks Jim

Alan

Alan Hale
Aberystwyth

3

Re: Records not removed from Dynamic Report Explorer form

If you delete them via the warehouse or a "proper" client, then they should be removed from the cache for you. If you delete records manually using pgAdmin, then you can simply delete the records with matching Ids from cache_occurrences.

Cheers
John

John van Breda
Biodiverse IT

4

Re: Records not removed from Dynamic Report Explorer form

Hi John - I did in fact delete the dummy records via the warehouse, but they still persisted until I rebuilt the cache.

What do you mean by a "proper" client?

I've just noticed something else odd - in pgAdmin, my cache_occurrences table appears to have no columns!

Cheers

Alan

Alan Hale
Aberystwyth

5

Re: Records not removed from Dynamic Report Explorer form

Hi Alan,

When John says "delete via warehouse" he means by the warehouse website interface and by a "proper client" he means, for example, a client website that deletes records via the warehouse web services. PgAdmin, on the other hand, is not a "proper client" although, with the correct query, it could mimic one.

For the cache to behave, deletion must set the deleted field to true and the updated_on field to the time of deletion.

Sounds like your cache_occurrences needs attention. Note my comment from yesterday about that.

Jim Bacon.

6

Re: Records not removed from Dynamic Report Explorer form

Hi again,

My last comment yesterday may already be out of date. John wrote in an email to me yesterday

another significant change is that the cache_occurrences table is now updated as soon as any changes are made and it is no longer updated by the scheduled tasks – so record edits are now visible immediately in the reports.

This change has been commited to branch 0.8.2, the trunk, and warehouse1 and testwarehouse at indicia.org.uk have been patched.

Jim Bacon

7 (edited by Alan Hale 16-03-2013 16:16:19)

Re: Records not removed from Dynamic Report Explorer form

Thanks for that update Jim.

Alan

Alan Hale
Aberystwyth

8

Re: Records not removed from Dynamic Report Explorer form

.... but I've just updated my 0.8.2 warehouse from SVN, and tested this out by deleting some records (occurrences and corresponding samples). They are still showing in my dynamic explorer form grid. And they remain present in the cache_occurrences table.

I concede it's likely I've misunderstood something here!

Cheers

Alan

Alan Hale
Aberystwyth

9

Re: Records not removed from Dynamic Report Explorer form

As an interim measure - can I safely delete the records from the cache_occurrences table via pgAdmin?

Cheers

Alan

Alan Hale
Aberystwyth

10 (edited by Alan Hale 18-03-2013 15:13:58)

Re: Records not removed from Dynamic Report Explorer form

I now find I can't add new samples (Sample with occurrences form) and get this error:

There was an SQL error: ERROR: column "location_id" of relation "cache_occurrences" does not exist at character 648 - insert into cache_occurrences ( id, record_status, downloaded_flag, zero_abundance, website_id, survey_id, sample_id, survey_title, date_start, date_end, date_type, public_entered_sref, entered_sref_system, public_geom, sample_method, taxa_taxon_list_id, preferred_taxa_taxon_list_id, taxonomic_sort_order, taxon, authority, preferred_taxon, preferred_authority, default_common_name, search_name, taxa_taxon_list_external_key, taxon_meaning_id, taxon_group_id, taxon_group, created_by_id, cache_created_on, cache_updated_on, certainty, location_name, recorders, verifier, images, training, location_id ) select distinct on (o.id) o.id, o.record_status, o.downloaded_flag, o.zero_abundance, su.website_id as website_id, su.id as survey_id, s.id as sample_id, su.title as survey_title, s.date_start, s.date_end, s.date_type, case when o.confidential=true or o.sensitivity_precision is not null then null else coalesce(s.entered_sref, l.centroid_sref) end as public_entered_sref, case when s.entered_sref_system is null then l.centroid_sref_system else s.entered_sref_system end as entered_sref_system, reduce_precision(coalesce(s.geom, l.centroid_geom), o.confidential, o.sensitivity_precision, case when s.entered_sref_system is null then l.centroid_sref_system else s.entered_sref_system end) as public_geom, tmethod.term as sample_method, cttl.id as taxa_taxon_list_id, cttl.preferred_taxa_taxon_list_id, cttl.taxonomic_sort_order, cttl.taxon, cttl.authority, cttl.preferred_taxon, cttl.preferred_authority, cttl.default_common_name, cttl.search_name, cttl.external_key as taxa_taxon_list_external_key, cttl.taxon_meaning_id, cttl.taxon_group_id, cttl.taxon_group, o.created_by_id, now(), now(), case when certainty.sort_order is null then null when certainty.sort_order <100 then 'C' when certainty.sort_order <200 then 'L' else 'U' end, case when o.confidential=true or o.sensitivity_precision is not null then null else coalesce(l.name, s.location_name) end, s.recorder_names, pv.surname || ', ' || pv.first_name, images.list, o.training, s.location_id from occurrences o left join cache_occurrences co on co.id=o.id join samples s on s.id=o.sample_id left join locations l on l.id=s.location_id and l.deleted=false join surveys su on su.id=s.survey_id left join cache_termlists_terms tmethod on tmethod.id=s.sample_method_id join cache_taxa_taxon_lists cttl on cttl.id=o.taxa_taxon_list_id left join (occurrence_attribute_values oav join termlists_terms certainty on certainty.id=oav.int_value join occurrence_attributes oa on oa.id=oav.occurrence_attribute_id and oa.deleted='f' and oa.system_function='certainty' ) on oav.occurrence_id=o.id and oav.deleted='f' left join users uv on uv.id=o.verified_by_id and uv.deleted=false left join people pv on pv.id=uv.person_id and pv.deleted=false left join (select occurrence_id, array_to_string(array_agg(path), ',') as list from occurrence_images where deleted=false group by occurrence_id) as images on images.occurrence_id=o.id where co.id is null and o.id=77

Checking via pgAdmin there is no location_id column in cache_occurrences.

Alan

Alan Hale
Aberystwyth

11 (edited by Jim Bacon 18-03-2013 16:26:45)

Re: Records not removed from Dynamic Report Explorer form

Hi Alan

Did you remember to visit home/upgrade to run database scripts?
(Refer to documentation on upgrading)

Jim Bacon.

12

Re: Records not removed from Dynamic Report Explorer form

Ah! - well spotted Jim. I didn't (there was no prompt upgrade and I forgot to do so).

But I've just tried to run the upgrade and get this error:

There was an SQL error: ERROR: relation "spatial_systems" does not exist at character 26 - SELECT "id", "code" FROM spatial_systems

Alan Hale
Aberystwyth

13

Re: Records not removed from Dynamic Report Explorer form

... and just seen in the developer notes that there will be no prompt to upgrade the database if updating from SVN.

I can't see any problem reported with the database upgrade in the application log by the way. (And I do now have a location_id column).

Alan Hale
Aberystwyth

14

Re: Records not removed from Dynamic Report Explorer form

Hi Alan

I can see this is a result of changes in r5427

The script 201303131119_occurrence_sensitivity.sql in this revision adds the spatial_systems table.

However, the upgrade code was also modified in this revision and now executes a function, populate_spatial_systems_table(), which is the source of the error you are seeing.

I would hazard a guess that the function is being called before the table has been created. I am not clear why that would be. It suggests that script hasn't run. What is the last run script for Indicia in the system table?

Jim Bacon.

15

Re: Records not removed from Dynamic Report Explorer form

One thought that crossed my mind was that you had not spotted my forum post about script sequences at http://forums.nbn.org.uk/viewtopic.php?id=4512, but I see that is not the case.

The code to populate the spatial systems table is indeed the very last thing in the upgrade, so if we are getting that far without the table existing it would indeed imply that the script  201303131119_occurrence_sensitivity.sql  has not been run for some reason.

Best wishes

John van Breda
Biodiverse IT

16

Re: Records not removed from Dynamic Report Explorer form

Hi John - yes, I did run that update system query  (through pgAdmin).

Jim - the last script run on the upgrade was "201303111529_cache_occurrences_location_id.sql"

Cheers

Alan

Alan Hale
Aberystwyth

17

Re: Records not removed from Dynamic Report Explorer form

Hi Alan

If you look in the system table you will find a record for Indicia plus each module that has been updated. The last run script you quote is for the cache_builder module not the Indica core. Could you have another look for us and see what is recorded as the last run script for Indicia?

Jim Bacon.

18

Re: Records not removed from Dynamic Report Explorer form

Ah - I see.

It looks like it was "201304031008_occurrence_training.sql".

Alan

Alan Hale
Aberystwyth

19

Re: Records not removed from Dynamic Report Explorer form

Thanks, Alan.

The answer to the problem is there to see. That script is dated 3rd April which is a date in the future. I have posted a query for you to execute with pgAdmin in the other thread, "Updating warehouse from SVN trunk". Once you have done that, visit /home/upgrade again and the upgrade process should complete successfully.

Finally, try adding and deleting some records and check the cache to see if they are being added and deleted correctly.

As far as cleaning up your cache_occurrences is concerned, I don't think a visit to scheduled_tasks?force_cache_rebuild will work any more as it sets a last_run_date and that cache is no longer built on schedule. It could be made to work if, instead, it dropped the cache_occurrences table and the variable, populated-occurrences, was deleted. Perhaps John has an opinion on that.

Jim Bacon

20 (edited by Alan Hale 19-03-2013 15:43:38)

Re: Records not removed from Dynamic Report Explorer form

Thanks Jim. OK, I've executed that new query and visited home/upgrade again. But I get the same error:

There was an SQL error: ERROR: relation "spatial_systems" does not exist at character 26 - SELECT "id", "code" FROM spatial_systems

.... and the last run script is still "201304031008_occurrence_training.sql"

Am I doing something wrong here? pgAdmin is telling me the query has run OK.

Alan

Alan Hale
Aberystwyth

21

Re: Records not removed from Dynamic Report Explorer form

Ah, now I've refreshed in pgAdmin the system table is giving me "201303031008_occurrence_training.sql"

But the error after /home/upgrade still occurs.

Alan

Alan Hale
Aberystwyth

22

Re: Records not removed from Dynamic Report Explorer form

Well, let's check that the update file is present where it should be.

Have a look in modules/indicia_setup/db/version_0_8_2 and confirm that the file 201303131119_occurrence_sensitivity.sql is there.

Jim Bacon.

23

Re: Records not removed from Dynamic Report Explorer form

I assume you meant ""201303031008_occurrence_training.sql".  No, it's not there - just the   "post-dated" version:                           "201304031008_occurrence_training.sql".

Alan

Alan Hale
Aberystwyth

24

Re: Records not removed from Dynamic Report Explorer form

Hi Alan

This tells us that something has not worked correctly with your sync with the repository. If the sync had worked then, firstly, the date on the occurrence_training.sql would have been corrected and, secondly, the occurrence_sensitivity update would be present.

It was no mistake of mine to ask if 201303131119_occurrence_sensitivity.sql was present as this is the update that comes with  r5427 and creates the table that you are missing.

Are you using TortoiseSVN as your Subversion client as we are?

Jim Bacon.

25

Re: Records not removed from Dynamic Report Explorer form

Hi Jim

Ah, sorry for making that assumption. Just to confirm, 201303131119_occurrence_sensitivity.sql  is not present.

I'm using SlikSVN, running it from the command line on my Windows PC - so I'm checking out to my PC and then uploading to the remote web server. Perhaps not the best way? What would you recommend? I'm only superficially familiar with using SVN.

TortoiseSVN - that's a graphical front-end to SVN isn't it?

What I could do is checkout 0.8.2 again and look for any error messages.

Cheers

Alan

Alan Hale
Aberystwyth