26

Re: Records not removed from Dynamic Report Explorer form

Hi Alan,

So I think we have pinned down the problem and now it is down to you to find out where, in your process, things have gone wrong.

I operate in two different ways on two different servers, both of which I am able to remote desktop. One has TortoiseSVN installed on it and I update the code it runs directly from the repository. This is quick and error free. The other I update by checking out to my PC and then selectively updating files known to have changed. This is slow and error prone.

Why like this? Well the former is used for development so updating from and committing to the repository is frequent and expected.  The latter? Well, it is the production server and I guess I was just expecting to upgrade according to the instructions on a rather infrequent basis.

TortoiseSVN is, as you say, a graphical Subversion client.

Jim Bacon

27

Re: Records not removed from Dynamic Report Explorer form

Hi Jim

I'm sorry to have taken your time up with this. I've just done another checkout from svn and straightaway I can see that the file count is up by 4 over my previous checkout. On further examination these include "201303131119_occurrence_sensitivity.sql"

So, I'll overwrite my warehouse code with this, run the database upgrade, and report back.

Cheers

Alan

Alan Hale
Aberystwyth

28

Re: Records not removed from Dynamic Report Explorer form

Hi Alan,

So, if this file has only been added to your system now, although it was part of a revision that was included in your previous checkout, that suggests to me that SlikSVN was misbehaving.

Unlucky for you that what should have been a simple upgrade has caused you so much hassle. On the other hand, you've learnt a few more things about Indicia along the way. Fingers crossed it works better this time.

Jim Bacon

29

Re: Records not removed from Dynamic Report Explorer form

Hi Jim

I'm sorry to have taken your time up with this. I've just done another checkout from svn and straightaway I can see that the file count is up by 4 over my previous checkout. On further examination these include "201303131119_occurrence_sensitivity.sql"

So, I'll overwrite my warehouse code with this, run the database upgrade, and report back.

Cheers

Alan

Alan Hale
Aberystwyth

30

Re: Records not removed from Dynamic Report Explorer form

OK, I've successfully uploaded the new 0.8.2 code and the files that were missing before are now there.

I've also run the scripts indicated here

But running the database upgrade gives me this:

There was an SQL error: ERROR: permission denied for relation pg_attribute - -- Hacky but efficient way to increase terms.term field size across all dependencies update pg_attribute set atttypmod=204 where attname='term' and atttypmod=104 and atttypid=1043

Alan

Alan Hale
Aberystwyth

31

Re: Records not removed from Dynamic Report Explorer form

According to the log file the error arises in "201303071000_term_field_size.sql"

Alan Hale
Aberystwyth

32

Re: Records not removed from Dynamic Report Explorer form

Hi Alan

Okay. first question is which update is this coming from? It might be listed in your log file but, a quick search for pg_attribute in the 0_8_2 database updates folder quickly shows it is 201303071000_term_field_size.sql. (We have cross-posted, I see.)

The update is very small

-- Hacky but efficient way to increase terms.term field size across all dependencies
update pg_attribute set atttypmod=204 where attname='term' and atttypmod=104 and atttypid=1043

which I can translate, thanks to the docs, as "change the length of any varchar field called 'term' to 200 characters in any table or view where it is currently 100 characters."

This update was presumably just added to your system so this is the first attempt to run it. It is an odd one as it is modifying the postgreSQL catalogue rather than the tables directly. I haven't seen this before. As the comment in the update says, "Hacky".

The fact that you are getting permission denied for trying to mess with the tables that postgres uses to organise itself is not all that surprising. More surprising is that my system has logged running the update successfully. If I look at the pg_attribute table (which is in the PostgrSQL catalog in the Indicia database) the update appears to have been ineffective, however.

I can suggest a workaround (running the query in pgAdmin as superuser and then updating the last_run_script to pretend it executed correctly) or you could just delete the update file and live with the current limit on the size of terms.

I guess the friendly

ALTER TABLE terms ALTER term TYPE character varying(200);

was rejected because, as the PostgreSQL documentation says

If a table has any descendant tables, it is not permitted to add, rename, or change the type of a column in the parent table without doing the same to the descendants. That is, ALTER TABLE ONLY will be rejected. This ensures that the descendants always have columns matching the parent.

so it may just be a way to avoid updating the views that use it.

I'm not sure it is a good hack either because

select pgc.relname, pga.attname, pga.atttypid, pga.atttypmod from pg_attribute pga
join pg_class pgc on pgc.oid = pga.attrelid
where pga.attname = 'term'

suggests it affects things that are not related to 'terms.term'.

Also the query is trying to update from 100 to 200 characters in size and my system currently has a size of 150 so that wouldn't have worked anyway.

Perhaps John can tell us if there is any reason why we shouldn't do it the long way which would be to drop all the views, update the column and recreate all the views.

Sorry, Alan. I think this one is a problem on our side.

Jim Bacon.

33

Re: Records not removed from Dynamic Report Explorer form

Thanks Jim - I'm learning a lot from all this!

Now, I did think of deleting the update file and running the upgrade again - but when i did I got another error, this time from "201303131119_occurrence_sensitivity.sql".  The error report is very long - here is just the first part of it:

ERROR: "$1" is declared CONSTANT CONTEXT: compilation of PL/pgSQL function "reduce_precision" near line 19 - ALTER TABLE occurrences ADD COLUMN sensitivity_precision integer;

Cheers
Alan

Alan Hale
Aberystwyth

34

Re: Records not removed from Dynamic Report Explorer form

Yup, I'm learning a lot too.
I'd no idea what that error message meant so I Googled it - probably you did too.
I found a post that says

You can't assign to a function's input parameters.

Sure enough you see the reduce_precision function is declared

CREATE OR REPLACE FUNCTION reduce_precision(geom geometry, confidential boolean, sensitivity_precision integer, sref_system varchar (20))

and later the geom parameter is assigned a value

geom = st_transform(geom, sref_metadata.srid);

Now this does not throw an error on my system. Why not? Well I found PL/pgSQL had been changed to allow parameters to be assigned to.

The date of that change, compared to the release dates of PostgreSQL indicates that it would have been introduced with version 9.0. And that is the version I am using. So I am betting you are using an earlier version. Can you confirm that to make sure I am not barking up the wrong tree.

Jim Bacon.

35 (edited by Alan Hale 21-03-2013 11:29:32)

Re: Records not removed from Dynamic Report Explorer form

You're dead right - I'm on v. 8.4. I recall trying to upgrade through yum to version 9.2 a few weeks ago and hitting some problems which led me (due to shortage of time) to temporarily give up. I'll have to revisit that.

Cheers

Alan

Alan Hale
Aberystwyth

36

Re: Records not removed from Dynamic Report Explorer form

Hi Alan

The documentation says that Indicia supports version 8.4 or later and I think I can rewrite that function in a couple of seconds.

Would you like to test the attached for us?

Jim Bacon.

Post's attachments

201303131119_occurrence_sensitivity.sql 4.73 kb, 3 downloads since 2013-03-21 

You don't have the permssions to download the attachments of this post.

37 (edited by Alan Hale 21-03-2013 13:17:18)

Re: Records not removed from Dynamic Report Explorer form

OK, just tried that - I now get a new error:

syntax error at or near "geom" at character 1
QUERY:  geom =  $1
CONTEXT:  SQL statement in PL/PgSQL function "reduce_precision" near line 8 - ALTER TABLE occurrences
   ADD COLUMN sensitivity_precision integer;

And there is a comment:

COMMENT ON COLUMN occurrences.sensitivity_precision IS 'Precision of grid references for public access of records that are sensitive. For example, set to 1000 to limit public access to a 1km grid square. If null then not sensitive.';

Cheers

Alan

Alan Hale
Aberystwyth

38

Re: Records not removed from Dynamic Report Explorer form

Sorry, forgot to declare the variable geom.
Try again.

Post's attachments

201303131119_occurrence_sensitivity.sql 4.83 kb, 3 downloads since 2013-03-21 

You don't have the permssions to download the attachments of this post.

39 (edited by Alan Hale 21-03-2013 15:25:41)

Re: Records not removed from Dynamic Report Explorer form

Yep! That's done it! Thanks for your perseverance Jim.

Alan

Alan Hale
Aberystwyth

40

Re: Records not removed from Dynamic Report Explorer form

Finally!!!
Hopefully you can add and delete samples and see them appear and disappear from cache_occurrences now.

Jim Bacon

41

Re: Records not removed from Dynamic Report Explorer form

Ah, forgot to test that! But, I have now and yes, they are appearing and disappearing as they should.

Great - thanks again.

Alan

Alan Hale
Aberystwyth