1 Reply by TonyP

  • TonyP
  • TonyP
  • Veteran
  • Offline
  • From: Somerset
  • Registered: 15-05-2006
  • Posts: 306

Re: Import Wizard Validation Failure

Hi, I thought I had raised this one some time ago but could not find it on searching.

I imported a large [circa 250,000] butterflies dataset and got through all the hurdles. When it came to actually doing the import I had it stop at the Validation point. There is no error give or reason. The import just stops.

I'm pretty sure this is not a reasonable way to proceed. Having spent many days getting the data to fit in with Recorder, telling me it's not good enough is not good enough.

I seem to remember this might have something to do with the 'committed matches' tables suddenly becoming unacceptable.

Does anyone know? I really don't want to have to rebuild all those weeks of informed decisions again. I have had to throw it away once before and it is expensive.


Tony

Data Manger
Somerset Environmental Records Centre

TonyP's Website

2 Reply by nmrs

  • nmrs
  • Veteran
  • Offline
  • From: London
  • Registered: 22-11-2006
  • Posts: 387

Re: Import Wizard Validation Failure

Hi Tony,

Are you importing 250,000 records in one hit? I split my imports with a maximum import of 50,000 records at any one time.

The only drawback with splitting imports is that you MUST make sure any single Event is not split over multiple imports! To ensure this, I sort my data in Date, Gridref (or Site), Taxon order and just make the split in the data where it clearly shows a change of Event.

Les Evans-Hill
Senior Data Officer
Butterfly Conservation, Butterflies for the New Millennium and National Moth Recording Scheme

nmrs's Website

3 Reply by johnvanbreda

  • johnvanbreda
  • Administrator
  • Offline
  • From: UK
  • Registered: 27-04-2006
  • Posts: 1,237

Re: Import Wizard Validation Failure

I suspect the problem is performance in the validation library rather than anything to do with committed matches. As Les suggests, reducing the size of each import should help.

John van Breda
Biodiverse IT

johnvanbreda's Website

4 Reply by TonyP

  • TonyP
  • TonyP
  • Veteran
  • Offline
  • From: Somerset
  • Registered: 15-05-2006
  • Posts: 306

Re: Import Wizard Validation Failure

nmrs wrote:

Hi Tony,

Are you importing 250,000 records in one hit? I split my imports with a maximum import of 50,000 records at any one time.

The only drawback with splitting imports is that you MUST make sure any single Event is not split over multiple imports! To ensure this, I sort my data in Date, Gridref (or Site), Taxon order and just make the split in the data where it clearly shows a change of Event.

Oops, too late, well sort of. The offending dataset was split by me into smaller chunks as I thought that might be the only way to find the problem. No error report was made the first time around. I cut the import down to 34,000 records and it failed with the error below.

I have of course managed to import datasets of around 200,000. There are two ways of looking at this process. By attempting to import all in one go I can eliminate all the problems. If then have to stop and complete the import in parts I will not have to resolve any issues in isolation. I have for example found the same people referred to in 5 or 6 different ways and been able to easily resolve them to a single individual.

On Events
I'm a little puzzled though if data from the same event is not imported at the same time how is it considered to be a different event. Surely something that happens on the same day with the same people at the same location is what an event is?

date/time         : 2011-02-22, 15:12:28, 838ms
computer name     : xxx
user name         : Administrator
operating system  : Windows XP Service Pack 3 build 2600
system language   : English
system up time    : 3 hours 53 minutes
program up time   : 3 hours 37 minutes
processors        : 3x AMD Athlon(tm) II X3 435 Processor
physical memory   : 1190/1791 MB (free/total)
free disk space   : (C:) 9.48 GB (U:) 118.17 GB
display mode      : 1152x864, 32 bit
process id        : $ce4
allocated memory  : 71.57 MB
command line      : RecorderApp.exe /65958
executable        : RecorderApp.exe
exec. date/time   : 2010-07-22 16:40
version           : 6.15.4.238
madExcept version : 2.7h
exception class   : EOleException
exception message : Invalid class string.

main thread ($cf0):
009811c1 RecorderApp.exe Import        391  +7 TfraImport.DoComValidation
00980f9e RecorderApp.exe Import        346  +6 TfraImport.PrepareDatabaseComparison
009807c4 RecorderApp.exe Import        159 +21 TfraImport.Execute
00990758 RecorderApp.exe ImportWizard  339 +69 TfrmImportWizard.ChangePage
0099013e RecorderApp.exe ImportWizard  192 +22 TfrmImportWizard.btnNextClick
004cce08 RecorderApp.exe Controls              TControl.Click
004a9f5c RecorderApp.exe StdCtrls              TButton.Click
00470c01 RecorderApp.exe Buttons               TBitBtn.Click
004aa050 RecorderApp.exe StdCtrls              TButton.CNCommand
004ccc70 RecorderApp.exe Controls              TControl.WndProc
004cfccf RecorderApp.exe Controls              TWinControl.WndProc
004a9e20 RecorderApp.exe StdCtrls              TButtonControl.WndProc
00721c39 RecorderApp.exe XPMenu       2928 +10 TControlSubClass.ControlSubClass
7c90eb94 ntdll.dll                             RtlAnsiStringToUnicodeString
004cca40 RecorderApp.exe Controls              TControl.Perform
004cfe07 RecorderApp.exe Controls              DoControlMsg
004d045f RecorderApp.exe Controls              TWinControl.WMCommand
004ccc70 RecorderApp.exe Controls              TControl.WndProc
004cfccf RecorderApp.exe Controls              TWinControl.WndProc
004a9e20 RecorderApp.exe StdCtrls              TButtonControl.WndProc
004464c9 RecorderApp.exe Classes               TList.Delete
004cf94c RecorderApp.exe Controls              TWinControl.MainWndProc
00453460 RecorderApp.exe Classes               StdWndProc
7e4292de user32.dll                            SendMessageW
7e42a993 user32.dll                            CallWindowProcA
004cfdb3 RecorderApp.exe Controls              TWinControl.DefaultHandler
004cd210 RecorderApp.exe Controls              TControl.WMLButtonUp
004ccc70 RecorderApp.exe Controls              TControl.WndProc
004cfc01 RecorderApp.exe Controls              TWinControl.WndProc
004ccc70 RecorderApp.exe Controls              TControl.WndProc
004cfccf RecorderApp.exe Controls              TWinControl.WndProc
004a9e20 RecorderApp.exe StdCtrls              TButtonControl.WndProc
00721c39 RecorderApp.exe XPMenu       2928 +10 TControlSubClass.ControlSubClass
004cca40 RecorderApp.exe Controls              TControl.Perform
004cf94c RecorderApp.exe Controls              TWinControl.MainWndProc
00453460 RecorderApp.exe Classes               StdWndProc
7e4196c2 user32.dll                            DispatchMessageA
004bc12b RecorderApp.exe Forms                 TApplication.ProcessMessage
004bc162 RecorderApp.exe Forms                 TApplication.HandleMessage
004bc392 RecorderApp.exe Forms                 TApplication.Run
00a33a20 RecorderApp.exe RecorderApp   375 +83 initialization

thread $d08:
7c90e514 ntdll.dll                 KiFastSystemCallRet
7c90df48 ntdll.dll                 NtWaitForMultipleObjects
7c80958a kernel32.dll              WaitForMultipleObjectsEx
7e4195f3 user32.dll                MsgWaitForMultipleObjectsEx
7e4196a3 user32.dll                MsgWaitForMultipleObjects
0042aba7 RecorderApp.exe madExcept ThreadExceptFrame
>> created by main thread ($cf0) at:
4ec67b7f gdiplus.dll               

thread $d24:
7c90e514 ntdll.dll  KiFastSystemCallRet
7c90df48 ntdll.dll  NtWaitForMultipleObjects

thread $d28:
7c90e514 ntdll.dll     KiFastSystemCallRet
7c90df58 ntdll.dll     NtWaitForSingleObject
7c8025d5 kernel32.dll  WaitForSingleObjectEx
7c80253d kernel32.dll  WaitForSingleObject

thread $fc4:
7c90e514 ntdll.dll  KiFastSystemCallRet
7c90d218 ntdll.dll  NtDelayExecution

thread $1cc:
7c90e514 ntdll.dll                 KiFastSystemCallRet
7c90daa8 ntdll.dll                 NtReplyWaitReceivePortEx
0042aba7 RecorderApp.exe madExcept ThreadExceptFrame
>> created by thread $e8c at:
77e7d10d RPCRT4.dll               

thread $378: <priority:1>
7c90e514 ntdll.dll     KiFastSystemCallRet
7c90df58 ntdll.dll     NtWaitForSingleObject
7c8025d5 kernel32.dll  WaitForSingleObjectEx
7c80253d kernel32.dll  WaitForSingleObject

thread $350: <priority:1>
7c90e514 ntdll.dll     KiFastSystemCallRet
7c90df58 ntdll.dll     NtWaitForSingleObject
7c8025d5 kernel32.dll  WaitForSingleObjectEx
7c80253d kernel32.dll  WaitForSingleObject

thread $994: <priority:1>
7c90e514 ntdll.dll     KiFastSystemCallRet
7c90df58 ntdll.dll     NtWaitForSingleObject
7c8025d5 kernel32.dll  WaitForSingleObjectEx
7c80253d kernel32.dll  WaitForSingleObject

modules:
00350000 C4DLL.DLL         1.0.0.1          U:
003c0000 MS5User.Dll       5.0.0.4          U:
00400000 RecorderApp.exe   6.15.4.238       U:
01430000 ms5cnv.dll        5.0.0.10         U:
01c00000 odbcint.dll       3.525.1132.0     C:\WINDOWS\system32
02a50000 xpsp2res.dll      5.1.2600.5512    C:\WINDOWS\system32
03a00000 nvLsp.dll         2.2.0.465        C:\WINDOWS\system32
03a60000 SQLOLEDB.RLL      2000.85.1132.0   C:\Program Files\Common Files\System\Ole DB
03ab0000 msadcer.dll       2.81.1132.0      C:\Program Files\Common Files\System\msadc
03e60000 ShowUnchecked.ocx 6.0.0.13         U:\Addins
040c0000 StdValLib.dll     6.15.1.230       U:\Addins
046b0000 Normaliz.dll      6.0.5441.0       C:\WINDOWS\system32
0f9a0000 VBAJET32.DLL      6.0.1.9432       C:\WINDOWS\system32
0f9c0000 expsrv.dll        6.0.72.9590      C:\WINDOWS\system32
10000000 MS5.Dll           5.0.0.12         U:
1b000000 msjet40.dll       4.0.9511.0       C:\WINDOWS\system32
1b2c0000 msjter40.dll      4.0.9502.0       C:\WINDOWS\system32
1b2d0000 MSJINT40.DLL      4.0.9502.0       C:\WINDOWS\system32
1b570000 msjetoledb40.dll  4.0.9502.0       C:\WINDOWS\system32
1b5d0000 mswstr10.dll      4.0.9502.0       C:\WINDOWS\system32
1b800000 msjtes40.dll      4.0.9502.0       C:\WINDOWS\system32
20000000 xpsp3res.dll      5.1.2600.5512    C:\WINDOWS\system32
3d930000 WININET.dll       8.0.6001.19019   C:\WINDOWS\system32
3dfd0000 iertutil.dll      8.0.6001.19019   C:\WINDOWS\system32
4ca30000 msadrh15.dll      2.81.1132.0      C:\Program Files\Common Files\System\ado
4d650000 msadox.dll        2.81.3012.0      C:\Program Files\Common Files\System\ado
4dd00000 sqloledb.dll      2000.85.1132.0   C:\Program Files\Common Files\System\Ole DB
4de10000 msado15.dll       2.81.3012.0      C:\Program Files\Common Files\System\ado
4ec50000 gdiplus.dll       5.2.6001.22319   C:\WINDOWS\WinSxS\x86_Microsoft.Windows.GdiPlus_6595b64144ccf1df_1.0.6001.22319_x-ww_f0b4c2df
5ad70000 uxtheme.dll       6.0.2900.5512    C:\WINDOWS\system32
5b860000 NETAPI32.dll      5.1.2600.5694    C:\WINDOWS\system32
5d090000 comctl32.dll      5.82.2900.6028   C:\WINDOWS\system32
5edd0000 olepro32.dll      5.1.2600.5512    C:\WINDOWS\system32
605d0000 mslbui.dll        5.1.2600.5512    C:\WINDOWS\system32
60e30000 MSDATL3.dll       2.81.1132.0      C:\Program Files\Common Files\System\Ole DB
662b0000 hnetcfg.dll       5.1.2600.5512    C:\WINDOWS\system32
68000000 rsaenh.dll        5.1.2600.5507    C:\WINDOWS\system32
68100000 dssenh.dll        5.1.2600.5507    C:\WINDOWS\system32
69b10000 msxml4.dll        4.20.9876.0      c:\WINDOWS\system32
6bd00000 SYNCOR11.DLL      0.1.2.3          C:\WINDOWS\system32
6d4f0000 DBNETLIB.DLL      2000.85.1132.0   C:\WINDOWS\system32
71a50000 mswsock.dll       5.1.2600.5625    C:\WINDOWS\system32
71a90000 wshtcpip.dll      5.1.2600.5512    C:\WINDOWS\System32
71aa0000 WS2HELP.dll       5.1.2600.5512    C:\WINDOWS\system32
71ab0000 WS2_32.dll        5.1.2600.5512    C:\WINDOWS\system32
71ad0000 wsock32.dll       5.1.2600.5512    C:\WINDOWS\system32
71b20000 mpr.dll           5.1.2600.5512    C:\WINDOWS\system32
71bf0000 SAMLIB.dll        5.1.2600.5512    C:\WINDOWS\System32
71c10000 ntlanman.dll      5.1.2600.5512    C:\WINDOWS\System32
71c80000 NETRAP.dll        5.1.2600.5512    C:\WINDOWS\System32
71c90000 NETUI1.dll        5.1.2600.5512    C:\WINDOWS\System32
71cd0000 NETUI0.dll        5.1.2600.5512    C:\WINDOWS\System32
71f80000 security.dll      5.1.2600.5512    C:\WINDOWS\system32
73000000 winspool.drv      5.1.2600.5512    C:\WINDOWS\system32
73160000 oledb32.dll       2.81.1132.0      C:\Program Files\Common Files\System\Ole DB
74060000 msadce.dll        2.81.3002.0      C:\Program Files\Common Files\System\msadc
74320000 odbc32.dll        3.525.3012.0     C:\WINDOWS\system32
74720000 MSCTF.dll         5.1.2600.5512    C:\WINDOWS\system32
750b0000 RESUTILS.DLL      5.1.2600.5512    C:\WINDOWS\system32
750f0000 MTXCLU.DLL        2001.12.4414.706 C:\WINDOWS\system32
75130000 colbact.DLL       2001.12.4414.700 C:\WINDOWS\system32
75350000 OLEDB32R.DLL      2.81.1132.0      C:\Program Files\Common Files\System\Ole DB
754d0000 CRYPTUI.dll       5.131.2600.5512  C:\WINDOWS\system32
755c0000 msctfime.ime      5.1.2600.5512    C:\WINDOWS\system32
75f60000 drprov.dll        5.1.2600.5512    C:\WINDOWS\System32
75f70000 davclnt.dll       5.1.2600.5512    C:\WINDOWS\System32
75f80000 browseui.dll      6.0.2900.5512    C:\WINDOWS\system32
76390000 IMM32.DLL         5.1.2600.5512    C:\WINDOWS\system32
763b0000 comdlg32.dll      6.0.2900.5512    C:\WINDOWS\system32
765b0000 MSDART.DLL        2.81.1132.0      C:\WINDOWS\system32
76600000 CSCDLL.dll        5.1.2600.5512    C:\WINDOWS\System32
76620000 comsvcs.dll       2001.12.4414.702 C:\WINDOWS\system32
76780000 SHFolder.dll      6.0.2900.5512    C:\WINDOWS\system32
76790000 cryptdll.dll      5.1.2600.5512    C:\WINDOWS\system32
767a0000 ntdsapi.dll       5.1.2600.5512    C:\WINDOWS\system32
767f0000 schannel.dll      5.1.2600.6006    C:\WINDOWS\system32
76990000 ntshrui.dll       5.1.2600.5512    C:\WINDOWS\system32
769c0000 USERENV.dll       5.1.2600.5512    C:\WINDOWS\system32
76b20000 ATL.DLL           3.5.2284.2       C:\WINDOWS\system32
76b40000 winmm.dll         5.1.2600.5512    C:\WINDOWS\system32
76bf0000 PSAPI.DLL         5.1.2600.5512    C:\WINDOWS\system32
76c30000 WINTRUST.dll      5.131.2600.5922  C:\WINDOWS\system32
76c90000 IMAGEHLP.dll      5.1.2600.5512    C:\WINDOWS\system32
76d10000 CLUSAPI.DLL       5.1.2600.5512    C:\WINDOWS\system32
76d60000 iphlpapi.dll      5.1.2600.5512    C:\WINDOWS\system32
76f20000 DNSAPI.dll        5.1.2600.5625    C:\WINDOWS\system32
76f60000 WLDAP32.dll       5.1.2600.5512    C:\WINDOWS\system32
76fb0000 winrnr.dll        5.1.2600.5512    C:\WINDOWS\System32
76fc0000 rasadhlp.dll      5.1.2600.5512    C:\WINDOWS\system32
76fd0000 CLBCATQ.DLL       2001.12.4414.700 C:\WINDOWS\system32
77050000 COMRes.dll        2001.12.4414.700 C:\WINDOWS\system32
77120000 oleaut32.dll      5.1.2600.5512    C:\WINDOWS\system32
773d0000 comctl32.dll      6.0.2900.6028    C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.6028_x-ww_61e65202
774e0000 ole32.dll         5.1.2600.6010    C:\WINDOWS\system32
77920000 SETUPAPI.dll      5.1.2600.5512    C:\WINDOWS\system32
77a20000 cscui.dll         5.1.2600.5512    C:\WINDOWS\System32
77a80000 crypt32.dll       5.131.2600.5512  C:\WINDOWS\system32
77b20000 MSASN1.dll        5.1.2600.5875    C:\WINDOWS\system32
77b40000 apphelp.dll       5.1.2600.5512    C:\WINDOWS\system32
77c00000 version.dll       5.1.2600.5512    C:\WINDOWS\system32
77c10000 msvcrt.dll        7.0.2600.5512    C:\WINDOWS\system32
77c70000 msv1_0.dll        5.1.2600.5876    C:\WINDOWS\system32
77dd0000 advapi32.dll      5.1.2600.5755    C:\WINDOWS\system32
77e70000 RPCRT4.dll        5.1.2600.6022    C:\WINDOWS\system32
77f10000 GDI32.dll         5.1.2600.5698    C:\WINDOWS\system32
77f60000 SHLWAPI.dll       6.0.2900.5912    C:\WINDOWS\system32
77fe0000 Secur32.dll       5.1.2600.5834    C:\WINDOWS\system32
78130000 urlmon.dll        8.0.6001.19019   C:\WINDOWS\system32
7c800000 kernel32.dll      5.1.2600.5781    C:\WINDOWS\system32
7c900000 ntdll.dll         5.1.2600.6055    C:\WINDOWS\system32
7c9c0000 shell32.dll       6.0.2900.6072    C:\WINDOWS\system32
7e290000 shdocvw.dll       6.0.2900.5512    C:\WINDOWS\system32
7e410000 user32.dll        5.1.2600.5512    C:\WINDOWS\system32
7e4b0000 HHCtrl.ocx        5.2.3790.4110    C:\WINDOWS\system32
7e720000 SXS.DLL           5.1.2600.5512    C:\WINDOWS\system32

Data Manger
Somerset Environmental Records Centre

TonyP's Website

5 Reply by johnvanbreda

  • johnvanbreda
  • Administrator
  • Offline
  • From: UK
  • Registered: 27-04-2006
  • Posts: 1,237

Re: Import Wizard Validation Failure

The issue with things being detected as different events is simply an issue of feasibility - if the import had to query the existing data to match your lists of recorders to as well as the imported data, things would grind to a halt. So this issue was considered a reasonable compromise.

Sounds a bit like your validation library addin is not installed properly. What happens if you right click something like a sample and revalidate it?

John van Breda
Biodiverse IT

johnvanbreda's Website

6 Reply by TonyP (edited by TonyP 28-02-2011 12:44:45)

  • TonyP
  • TonyP
  • Veteran
  • Offline
  • From: Somerset
  • Registered: 15-05-2006
  • Posts: 306

Re: Import Wizard Validation Failure

johnvanbreda wrote:

The issue with things being detected as different events is simply an issue of feasibility.

Sounds a bit like your validation library addin is not installed properly. What happens if you right click something like a sample and revalidate it?

Ok thanks for that John. I have tried to revalidate both samples and occurrences without any problems.

Then I had a thought. I was logged in on another machine at the time. There the validation fails. I log into the machine as the administrator and onto Recorder as me. I'm guessing it's a registry error. I will look at this later. Just now I think I will try importing on my machine where I did not get problems.

Tony
PS That seems to be the problem my PC doesn't cause the problem so must have the Addin set up correctly. Hurrah, thanks so much John.

Data Manger
Somerset Environmental Records Centre

TonyP's Website