1

Re: R6 Import Wizard Problems with Concatenated Data

As a trial to decide whether to switch from R2002 to R6 I’ve been doing a few basic trial imports and found that the import wizard in 6.9.3.102 seems to have problems with every 64th line of excel input when columns being imported are concatenations of other columns.

For example I have an Observers column which is the concatenation of two other columns, e.g. A2  is created like ‘= B2 & “ & “ & C2’.

Similarly I also create the Comment column by concatenating various  columns together.

The import wizard reports as being empty (marked as errors in red) the Observers column for lines 3, 67, 131, 195, etc. i.e. every 64th line.

The comment field in the import wizard grid for these rows is also shown as being empty, but are not marked as errors because empty comment fields are fine.

As a test I replaced the concatenated Observers column with the equivalent values of plain text (i.e. paste special - “Values”) and this got around the Observer errors with every 64th line.

When I checked what had actually been imported the Comments were indeed blank for every 64th record. The import wizard had effectively thrown away the comments for every 64th record.

This doesn’t bode well for the 260,300 botanical records that I’ve been given on spreadsheet and need to get through the import wizard... (Is there still a performance issue that means that I will have to chop this up into chunks of around 4000 records per import, i.e. about 65 separate imports?)

(Also somewhere along the migration/upgrade process at least one of our sites became corrupted and lost all of its records.  I don’t know if that was before or after the upgrade to 6.9.3.102 as I only found it afterwards).

Looks like we won't be switching just yet.

Regards, Keith Balmer
Beds & Luton LRC

2 (edited by Sarah Shaw 11-05-2007 13:45:14)

Re: R6 Import Wizard Problems with Concatenated Data

Hi Keith

Is it possible for you to send me the excel file you were using (for testing purposes only)?

I can then look into this and get back to you.

There are still limitations with the import wizard in terms of the number of rows that can be imported, but we are planning to review this functionality over the next few months.

No need to send - Charles Roper has also spotted this problem and has just sent me an example file.

Best wishes,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

3

Re: R6 Import Wizard Problems with Concatenated Data

Keith, this can be worked around by copying the formulas, then Edit > Paste Special > Values to convert them from formulas into values.

Charles Roper
Digital Development Manager | Field Studies Council
http://www.field-studies-council.org | https://twitter.com/charlesroper | https://twitter.com/fsc_digital

4

Re: R6 Import Wizard Problems with Concatenated Data

Hi Keith and Charles

Just to let you know I have created a CCN for Dorset Software to look into this/fix in a future version.

Many thanks,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

5

Re: R6 Import Wizard Problems with Concatenated Data

Sarah Shaw wrote:

There are still limitations with the import wizard in terms of the number of rows that can be imported, but we are planning to review this functionality over the next few months.

Sarah, It would be helpful to know what this limitation is. Could you publish details please. Thanks

Dave Cope,
Biodiversity Technology Officer,
Biodiversity Information Service for Powys and Brecon Beacons National Park.

6

Re: R6 Import Wizard Problems with Concatenated Data

Hi Dave

The limit in terms of the number of rows in excel (or an in Access or text file) appears to vary considerably from user to user.

This is probably in part to do with the performance of individual systems, but I think also depends on the complexity of the data (how many columns are included, how much matching is done in the wizard process).

Charles Roper has reported that he can import 10,000 rows at a time (in an excel file), but other users report around 5,000 - for details see these existing threads - http://forums.nbn.org.uk/viewtopic.php?id=272 and http://forums.nbn.org.uk/viewtopic.php?id=263.

John van Breda from Dorset Software will probably be able to shed some light on this for us.

Kind regards,

Sarah

Sarah Shaw
Biodiversity Information Assistant
JNCC

Sarah Shaw
Biodiversity Information Assistant
JNCC

7

Re: R6 Import Wizard Problems with Concatenated Data

Hi

When using R2002 I liked that the 'sex/stage' column was separate from abundance and had a problem after moving to R6 to find that they needed to be together. I found that concatenating the values from the two columns into a blank column inserted for the purpose, then copying and using Paste Special, Values (as Charles said) into the 'abundance' column, then deleting the newly inserted column and the 'sex/stage' column led to imports that worked as expected. I have only imported up to about 1000 at a time, so cannot comment on maximum number of lines to import at a time.

Cheers, Ian