1 Reply by Janet Simkin

  • Janet Simkin
  • Janet Simkin
  • Veteran
  • Offline
  • From: Northumberland
  • Registered: 10-05-2006
  • Posts: 260

Re: Problem with export using filter

Since upgrading to R6.13.3 (on Vista) I have tried several exports using export filters to NBN Access zipped format, but they have all failed with the same error:

command line      : RecorderApp.exe /197194
executable        : RecorderApp.exe
exec. date/time   : 2008-11-04 08:45
version           : 6.13.3.182
madExcept version : 2.7h
exception class   : EOleException
exception message : Violation of PRIMARY KEY constraint 'PK__#TempList_________6A30C649'. Cannot insert duplicate key in object 'dbo.#TempList'.

main thread ($544):
00917935 RecorderApp.exe FormActions 1866 TdmFormActions.RevalidateNBNData
008aa150 RecorderApp.exe DataExport   667 TdlgDataExport.ValidateAndExport
008a8f55 RecorderApp.exe DataExport   334 TdlgDataExport.bbOkClick
004b4048 RecorderApp.exe Controls         TControl.Click
004a9160 RecorderApp.exe StdCtrls         TButton.Click
004a9254 RecorderApp.exe StdCtrls         TButton.CNCommand
004b3eb0 RecorderApp.exe Controls         TControl.WndProc
004b6f0f RecorderApp.exe Controls         TWinControl.WndProc
004a9024 RecorderApp.exe StdCtrls         TButtonControl.WndProc
004b3c80 RecorderApp.exe Controls         TControl.Perform
004b7047 RecorderApp.exe Controls         DoControlMsg
004b769f RecorderApp.exe Controls         TWinControl.WMCommand
004d0754 RecorderApp.exe Forms            TCustomForm.WMCommand
004b3eb0 RecorderApp.exe Controls         TControl.WndProc
004b6f0f RecorderApp.exe Controls         TWinControl.WndProc
004ce7ad RecorderApp.exe Forms            TCustomForm.WndProc
004b6b8c RecorderApp.exe Controls         TWinControl.MainWndProc
004528d8 RecorderApp.exe Classes          StdWndProc
76e80af5 user32.dll                       SendMessageW
76e8b9fc user32.dll                       CallWindowProcA
004b6ff3 RecorderApp.exe Controls         TWinControl.DefaultHandler
004b4450 RecorderApp.exe Controls         TControl.WMLButtonUp
004b3eb0 RecorderApp.exe Controls         TControl.WndProc
004b6f0f RecorderApp.exe Controls         TWinControl.WndProc
004a9024 RecorderApp.exe StdCtrls         TButtonControl.WndProc
004b6b8c RecorderApp.exe Controls         TWinControl.MainWndProc
004528d8 RecorderApp.exe Classes          StdWndProc
76e75a27 user32.dll                       DispatchMessageA
004d4de3 RecorderApp.exe Forms            TApplication.ProcessMessage
004d4e1a RecorderApp.exe Forms            TApplication.HandleMessage
004d1989 RecorderApp.exe Forms            TCustomForm.ShowModal
009163db RecorderApp.exe FormActions 1418 TdmFormActions.ExportUsingFilterClick
004c5a96 RecorderApp.exe Menus            TMenuItem.Click
004c6d33 RecorderApp.exe Menus            TMenu.DispatchCommand
004c7aba RecorderApp.exe Menus            TPopupList.WndProc
004b6b8c RecorderApp.exe Controls         TWinControl.MainWndProc
004c7a09 RecorderApp.exe Menus            TPopupList.MainWndProc
004528d8 RecorderApp.exe Classes          StdWndProc
76e75a27 user32.dll                       DispatchMessageA
004d4de3 RecorderApp.exe Forms            TApplication.ProcessMessage
004d4e1a RecorderApp.exe Forms            TApplication.HandleMessage
004d503a RecorderApp.exe Forms            TApplication.Run
0099f832 RecorderApp.exe RecorderApp  317 initialization
7688490f kernel32.dll                     BaseThreadInitThunk

All the filters are set to include all records, with observations, locations and people. Some are for several surveys, another for just one. Export of the same data, survey by survey, without using filters works.

While we are on the subject of exports, does anyone know whether the problem with transferring custody (http://forums.nbn.org.uk/viewtopic.php?id=502) has been fixed yet, or is on the list for the next release?

Janet

Janet Simkin
British Lichen Society

Janet Simkin's Website

2 Reply by Lynn Heeley

  • Lynn Heeley
  • Administrator
  • Offline
  • Registered: 03-07-2008
  • Posts: 345

Re: Problem with export using filter

Janet,

Can you please give us a little more information about this we currently can't get the same error.

For your second question (you refer to Recorder being unable to reassign custodianship on surveys of over 250K records) - can you confirm that this is still an issue with the latest version (you were on 6.10.4 at the time). If so we will definately get this logged, sorry it seems to have fallen between the rails in the changeover from Sarah to myself.

Thanks,
Lynn

3 Reply by Janet Simkin

  • Janet Simkin
  • Janet Simkin
  • Veteran
  • Offline
  • From: Northumberland
  • Registered: 10-05-2006
  • Posts: 260

Re: Problem with export using filter

I can't think of any more information to give you, so I'll try it a few more times and look for clues. Will also try the transfer of custody, but that used to hang the whole system so it will have to wait until a quieter time.

Janet

Janet Simkin
British Lichen Society

Janet Simkin's Website

4 Reply by SimonWood

  • SimonWood
  • Veteran
  • Offline
  • Registered: 09-09-2008
  • Posts: 63

Re: Problem with export using filter

Hi Janet,

We have been able to reproduce this, and we think it is caused by the Validation Add-in. As a short term solution if you uninstall the Add-in the error no longer occurs.

We will investigate this further to provide a long term resolution.

Thanks,

Simon Wood
[url=http://www.dorsetsoftware.com/]Dorset Software Services[/url]

SimonWood's Website

5 Reply by Janet Simkin

  • Janet Simkin
  • Janet Simkin
  • Veteran
  • Offline
  • From: Northumberland
  • Registered: 10-05-2006
  • Posts: 260

Re: Problem with export using filter

Thanks, Simon. I'm wondering about the implications of uninstalling the validation add-in, isn't it also used by the import wizard and other functions?

Also, does this suggest that the validation is finding some problem with my database that I ought to do something about?

I have tested the transfer of custody on export and this does now work, although it is still very slow getting to the screen where you specify the new custodian (half an hour in my test, and it was only selecting a few thousand records to export) so I'm not sure whether it will handle large volumes of data. Curiously this export completed succesfully, but the same export without the transfer of custody failed like all the rest.

Janet

Janet Simkin
British Lichen Society

Janet Simkin's Website

6 Reply by SimonWood

  • SimonWood
  • Veteran
  • Offline
  • Registered: 09-09-2008
  • Posts: 63

Re: Problem with export using filter

The validation addin is now used by all import and NBN data or zipped access export processes so you are right, it would affect other functions. However, the bug you have found appears to occur only using the Export Filters so one possibility would be to uninstall the addin only for the export, then re-install it.

As to whether there is a specific issue in your data, the fact that we were able to reproduce it implies that although it relates to the validation addin, it is not a data problem. My feeling is it’s actually a bug in the code which prepares the list of items to export from an export filter so that they can be validated. That’s why the data exports OK when you do it survey by survey.

Thanks for the information about custody transfer as this may help to track down the problem.

Simon Wood
[url=http://www.dorsetsoftware.com/]Dorset Software Services[/url]

SimonWood's Website

7 Reply by Janet Simkin

  • Janet Simkin
  • Janet Simkin
  • Veteran
  • Offline
  • From: Northumberland
  • Registered: 10-05-2006
  • Posts: 260

Re: Problem with export using filter

In case anyone else hits the same problem, there is an easier way round it. Select the option to transfer custody but then enter your own site ID so that you are transferring custody to yourself. The exports take longer but at least they work.

Janet

Janet Simkin
British Lichen Society

Janet Simkin's Website

8 Reply by Lynn Heeley

  • Lynn Heeley
  • Administrator
  • Offline
  • Registered: 03-07-2008
  • Posts: 345

Re: Problem with export using filter

Janet did you manage to try the transferring of custody issue you refer to above - is it still an issue in 6.13.3?
Thanks, Lynn

9 Reply by Janet Simkin

  • Janet Simkin
  • Janet Simkin
  • Veteran
  • Offline
  • From: Northumberland
  • Registered: 10-05-2006
  • Posts: 260

Re: Problem with export using filter

Hi Lynn,

Export with transfer of custody seems to work on 6.13.3, including the options to transfer only some items or none at all. That is, it works with small volumes of data.

The problem is with larger exports. For the last week I have been trying to export about 70k records, comprising all the records in 20 smallish surveys. That's from a database of 1.1m records organised into 100 surveys. It takes 36 hours to find the information to export before it gets to the transfer of custody screen. That's not a typo, it takes 36 hours and then to make sure it has all been a complete waste of time it falls over doing the actual export with an out of memory error. For all that time R6 can't be used for anything else.

I can't do the export without transfer of custody because of the error previously reported.

So, I've been able to do some of the exports we need but the rest will have to be broken down and done a few small surveys at a time, which is rather tiresome both for me and for the people who will be receiving the exports. This will be a regular requirement for us, and presumably for other national databases where there is a need for several people to hold copies of all or part of the database on their standalone systems, so I hope there will be a resolution in due course. In the meantime, any guidance on the maximum number of records or surveys to include in an export would be helpful.

Janet

Janet Simkin
British Lichen Society

Janet Simkin's Website

10 Reply by Lynn Heeley

  • Lynn Heeley
  • Administrator
  • Offline
  • Registered: 03-07-2008
  • Posts: 345

Re: Problem with export using filter

Hi Janet,

Thank you for your post - I will make sure this gets chased up (obviously funds permitting) - there is clearly an error of some sort here. I'm afraid I don't know what the maximum number of records/surveys to include in the export to ensure that its works - perhaps someone else might be able to help here?

Sorry to not be able to help and thank again for getting back to me with the information.
Best Wishes,
Lynn

11 Reply by charlesroper

  • charlesroper
  • charlesroper
  • Digital Development Manager
  • Offline
  • From: Field Studies Council
  • Registered: 26-03-2006
  • Posts: 906

Re: Problem with export using filter

I don't have an answer to your question I'm afraid, but I can say that the export routines in Recorder do need some TLC. Like Janet, I've also had some problems like this with the Export Filters to the extent that I've given up using them. I never seem to be able to reliably reproduce the problems, though, mainly because it takes so long to generate the problem in the first place!

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

charlesroper's Website