1 Reply by stevemcbill (edited by stevemcbill 18-05-2010 09:44:04)

  • stevemcbill
  • Veteran
  • Offline
  • From: Runcorn, Cheshire
  • Registered: 21-11-2006
  • Posts: 803

Re: Process of Testing of New Updates

I realise I am exposing my lack of knowledge about how the NBN tests new updates to Recorder here but I feel it would be good, for more than just myself, to obtain clarification.

Since Windows 3.1 Microsoft has often been, accurately, accused of using its User base to undertake testing on its software rather than doing it 'in-house' before release.

Consequently, I am wondering what testing procedures are in-place to test changes to Recorder-6 prior to their release to the Recorder User Community. 

Firstly, does Dorset Software have a testing team/person to undertake structured testing of new changes to the program ?  Are these tests documented ?

Secondly, does the NBN have a testing team/person to parallel test changes before they are formally accepted and signed off by the NBN and then released to Users ??

I suppose the above also prompts a further question of 'How' the testing is undertaken.  The usage of a wide variety of Windows operating systems and underlying hardware architecture (32 and 64 bit) as well as supporting software (database management, etc.) by Users means that extensive testing may be impossible.  However, the NBN and JNCC know ALL users of Recorder-6 and could contact such Users (perhaps on an annual survey basis) to find out what machines, operating systems, etc., are being used and could then target their testing based upon such knowledge to help reduce the amount of Community Testing being undertaken.

I would welcome others thoughts and comments (as well as answers) to any of the above.

Cheers and thanks.

Steve

Steve J. McWilliam
www.rECOrd-LRC.co.uk
www.stevemcwilliam.co.uk/guitar/

stevemcbill's Website

2 Reply by Mary Campling

  • Mary Campling
  • Administrator
  • Offline
  • Registered: 16-04-2010
  • Posts: 101

Re: Process of Testing of New Updates

Dear Steve,

Dorset software conduct a series of in-house testing on all of the changes made to the Recorder system prior to releasing a beta version to JNCC. The testing at Dorset is conducted by a separate testing team and not the actual developers of the code. For small/medium CCNs the system is tested in the specific areas that the CCNs relate to. For larger CCNs which affect core functionality throughout the system, a full regression test of the system may be necessary. As this is a time consuming (thus costly) factor, it is specified on the individual CCN if a full regression test of the system is required. For example, when the system was upgraded to work on Vista a full regression test was carried out.

The beta release is subsequently tested by contractors on behalf of JNCC who know the system extremely well and work with it on a daily basis (ie Recorder experts) – the testing at this stage is conducted on Windows XP and Vista. In addition to this, testing also occurs at JNCC on both XP and Vista and with Steering Group members in Luxembourg and Germany. This period of testing runs for a set length of time and any issues discovered are reported to Dorset– after this there is a subsequent build phase at Dorset where the corrections are implemented followed by a second set of testing which ensures that any issues raised during the first set of testing have been resolved. The version is then released to users.

Occasionally an unexpected change in behaviour occurs in an area of Recorder that is unrelated to any planned changes. For example, in a recent previous release, there was an issue picked up by the user community regarding the Record Card screen. The interesting point with this issue was that the code in this area was not changed for the release. The issue stemmed from the import wizard, which was allowing the importation of null sample comments (and other fields). The Record Card was then subsequently encountering issues when saving to these existing records.

We do not rely on the Recorder user base to undertake testing as we endeavour to only release new versions of Recorder which we believe are stable and functioning correctly (eg we did not release 6.14.6 but fixed the issues and will roll out with 6.15). However, some issues are only detected after extensive use which is naturally the realm of Recorder users.

Please note that the NBN have no role in testing or releasing versions of Recorder.

With regards to the test specification of the machines used at Dorset, the common setup for testing minor changes is now:
Windows Vista Business SP2 x32 bit
SQL Express 2005
Intel Core 2 2.4ghz
3GB Ram

They also identify whether or not the system needs to be tested on:
Windows XP SP3 x32 bit.
MSDE
1GB Ram

Hope this helps,
Mary