Testing

Testing is not considered to be part of the Project Management approach. More often, it is defined within the methodology being used or in a specific testing methodology. However, most Project Managers have to deal with technology development and all should wish to test their solutions before moving into live operations.

In this section we examine testing:

• what do we test?
• faults are inevitable 
• building a solid, scientific proof that the solution is fit for purpose
• three main types of testing 
• test data 
• baseline to test against
• automated test tools 
• regression testing 
• User Acceptance Testing (UAT)
• testing process 
• specific types of testing.

What do we test?

It is common to think of testing as a technical issue - something that is done for technology developments. It is good practice, however, to test all elements of all solutions, whether or not technology is involved. Maybe "test" is not a good word to use. We need to be confident that every element of the solution is ready for operational use.

Here are some examples, starting with some common technical issues but moving onto critical business issues:

• Does the computer application work acceptably?
• Can the network cope with the volumes of transactions and data?
• Do we have adequate provisions for backup, recovery, and disaster recovery of the systems?
• Will the disaster recovery process work, eg staff awareness, instructions, facilities, re-routing telephone numbers, availability of office equipment, stationery, downtime, loss of transactions in progress, etc?
• Can the workforce cope with additional workloads during cutover and early live operation?
• Do the staff have sufficient knowledge and capability to operate the new processes?
• Are the new processes documented accurately in a usable manner and accessible to the staff?
• Have our customers understood the changes that will affect them?
• Will the new product and service offerings be a commercial success?

All these aspects can, and probably should, be tested. In some cases it is possible to test multiple aspects in a single process. For example, if I test the new computer system, using trained end-users, who are following the procedural documentation provided, I am validating the processes, documentation, and effectiveness of the training as well as the correct operation of the system. Trial runs or operational pilots can also test the overall solution.

Faults are Inevitable

There is a naive view that if you do the right amount of testing the end-product will be correct.

There is a scientific / mathematical view that you can logically define a test process which will cover every designed aspect of a solution and thus create a complete proof of the solution. It is a good theory but it is hard to find practical examples.

There is a practical view that the more testing you do, the closer you get to finding all the errors - but to reach a perfect result would require infinite effort. You would choose, therefore, how much effort was justified to achieve an acceptable compromise between effort and quality.

There is a realistic view, as demonstrated by Myers, that you will not approach zero errors, but, instead, approach the limit of faults that would ever be uncovered by a formalised testing process. (The Art of Software Testing by Glenford J. Myers, 1979)

There is a chaos-theory view that once you reach the limit, the disruption caused by further dabbling with the solution will generate more problems than are solved.

What is certain is that there is a balance between testing effort and the degree of quality. The choice should be a business decision, based on optimising benefit. There is no right answer. The manufacturers of a quality product will want to invest more time in testing than the suppliers of a cheap commodity product. A life-critical system, such as an aircraft, will warrant greater confidence levels than a non-critical application such as a typing tutor program.

It is interesting to compare theory with reality. The onboard flight software for the Space Shuttle is a mission-critical, life-critical, national-pride-critical software application. It is hard to imagine any system that would merit greater degrees of testing and quality. (By the way, none of these issues related to any accidents or injuries.) This chart shows the errors found in the code in the period 1988 to 1993: errors that got into released versions failures that occurred failures that occurred during flight (just one in 1988). The pattern of reducing errors does bear an interesting resemblance to the theoretical norms. At first there is a relatively high number of faults, steadily reducing but never reaching zero.

Building a Solid, Scientific Proof

The main goal of formal testing is to produce a sound, logically valid proof that the results meet desired levels of confidence. Testing must be conducted in a carefully controlled manner to achieve this. It must methodically address every component of the solution.

A great deal of testing is normally conducted as part of the development process - testing components of the solution as they are built to ensure they meet their specifications. This is a valuable and valid process, but it does not form part of the formal proof of the solution. Until development is completed there is no guarantee that components have been stabilised. There is also no independent verification that they met the overall needs in the context of the overall solution.

As well as providing a reasonable degree of confidence prior to formal testing, the informal tests can provide useful material and preparation for the subsequent formal tests. They lay a foundation that is the staring point for formal testing.

Testing a complex solution cannot usually be achieved in a single step. Normally the work is broken down into manageable pieces. For example, you might:

• Start by testing the various functions of the solution in isolation. Validate in full detail everything they are supposed to do. Check plausible alternative scenarios as well as the normal situations, for example, a sale might be cancelled, a payment might be rejected, an item might be out of stock. Do not check the implausible, for example, a negative quantity if it is impossible to enter it as a negative value (that is more a job for the developers to consider during their unit testing).
• When all the functions in a particular process have been validated in detail, test them together. This time you are looking for the correct passage of information and transactions throughout the entire end-to-end process. You do not need to repeat the detailed tests you did before. You are only looking at issues that involve the combinations of separate functions.
• When all the processes have been validated you can test the full system. Again, you do not repeat earlier tests. Only focus on issues that involve the overall operation of the system.
• Finally, when you have confidence in the new system, test how it behaves when connected to other systems and external parties.

Three Main Types of Testing

At the end of this section we list many specific types of testing and some variants in the terminology people use. For now, we will simply recognise the three main uses...

Unit Testing

Unit testing validates in detail each component that was developed. It is typically the developer's point of view - does it conform to the specification. It ignores whether that component works in the context of the completed system.

User Testing

User testing, or business testing, looks at the results from a business / user perspective. Does the final product do the things we want it to do? Do all the individual features work properly? Is it fit for purpose?

Technical/Operational Testing and Tuning

Technical testing and tuning examines the capability of the solution to be operated safely, efficiently and dependably at normal and peak levels of usage. It is concerned with the inner operational workings and procedures. It does not concern itself with whether it meets the users' functional needs, but it is concerned about the demands they will place upon it.

Test Data

There is an art to creating good test data. Having a complete, fictitious world in which to follow realistic storylines is a good way to test out the various business scenarios. Remember that all the sub-plots of the story have to co-exist. You will need to follow normal transactions alongside all the abnormal situations - amendments, cancellations, failures, mis-matches. You will also want to simulate the passage of time. Follow the scenarios throughout their natural lifecycle and beyond into consequences such as management reporting and accounting. Good test data requires good preparation. The scenarios should be planned in advance. They should be constructed to explore every aspect of the business solution in a logical manner. Expected results should be predicted alongside the test scenarios so that you are not dependent on the judgement of the tester.

Paint the backdrops Tell a story Divide the story into acts and scenes Any good story has twists and turns Many misadventures will befall our hero But the hero will always find a way to escape and recover And there will be a happy ending?

Baseline

The test data is intended to validate the correct functionality of the solution. But where is that correct baseline defined? Consider these possibilities...

Test Tools

Many test tools are available these days. They offer control, speed, and repeatability of tests. They are particularly useful for testing scenarios that require high volumes of transactions such as peak-volume network loading. Repeatability is useful for running re-tests after problems have been solved and regression testing where a component of the solution has been changed and it is necessary to validate that other functionality has not been affected.

The disadvantages of test tools are the time and cost. Preparation of test data has to be exhaustive and correct (although in some cases you might be able to capture scripts from the initial manual testing). The tools themselves can represent a significant cost.

Regression Testing

Regression means going back. With testing processes we mean that you need to repeat previously successful tests any time there is a chance that subsequent changes could have affected an aspect of the solution.

The testing process involved building a sequence of tests, many of which stood upon the successful results of earlier tests. If a component of the solution has been changed, all other components which relied upon it might have been affected. Similarly, all tests that relied on an earlier test might no long be valid. Consider carefully which solution and testing components need to be re-validated.

One consequence of this issue is that changes during the testing process are bad news. There will always be errors and changes during the testing, but it might be better to defer the correction of some minor problems to make better overall progress. A seemingly innocuous example with a big hidden catch is when a software supplier suggests that a problem you are experiencing would be solved by moving to their next software release in which the problem has been fixed. Moving to a new release probably means that every test you did to date is now invalidated and will have to be repeated.

User Acceptance Testing

User Acceptance Testing (UAT) is a common name for the type of testing that forms the basis for the business accepting the solution from its developers (whether internal or external). It is a common requirement in many organisations and in many contracts. There are two drawbacks - it might not be very reliable and it might be a waste of effort. In fact, you might get a better solution, faster and cheaper if you do not do it...

The developers will need to conduct exhaustive methodical tests regardless of the need for independent User Acceptance Tests. Best quality will be achieved if they include the business and users in these tests to make sure they have not misunderstood the business needs. If, therefore, the developers' formal testing is done with full participation of the business and with the responsible user managers having prime authority over the content, review and sign-off of the tests, the requirement for user acceptance can be met in a single phase of testing instead of two different phases.

Conversely, where the business is left to define and conduct its own independent testing, it is common to find that they do not have the methodical, comprehensive approach of the developers. The reliability of the tests is often questionable.

A single, combined phase of systems testing and User Acceptance Tests can produce the best results in the shortest time. Consider whether it is appropriate and permissible in your circumstances.

Testing Process

There are many detailed approaches to testing. See if there is a specific approach to be used in your case. If not, here is a basic process that you might follow.

Test Objectives First, define a comprehensive set of tests to build the solid testing "wall". Every requirement and plausible scenario should be covered - with no gaps and, preferably, with no overlaps or duplication. Each test may be described initially in terms of its objective, for example, an objective might be "to test that a cancelled order reverses all transaction data, works orders, stock positions and financial postings". Identify who is responsible for conducting this test and name the relevant user manager who is to approve the definition, review and sign-off of the test. At a later stage, you might add scheduling and sequencing information.
Test Definition When the list of test objectives has been reviewed and agreed, detail the test scenario and steps, for example: note stock level for item XYZ note credit available for customer CUS01 create order #1234 for quantity 1 of item XYZ at price £100 for customer CUS01 check stock level for item XYZ check credit available for customer CUS01 cancel order #1234 check stock level for item XYZ check credit available for customer CUS01 etc Alongside these steps, write the expected results - ie how the tester will know that it worked as expected. For example: 100 £1000 Order confirmation message confirms quantity 1 of item XYZ at price £100 for customer CUS01 99 £900 Order cancellation message confirms quantity 1 of item XYZ at price £100 for customer CUS01 100 £1000 etc The remainder of this form is used as a checklist for running the test. For each step the tester either ticks the step to note that it worked or creates an test incident control record. In this approach, we do not allow any other action. In other approaches a variety of other actions might be taken in an attempt to avoid noting the failure. Although this might seem desirable, the lack of control can be dangerous and counter-productive. Encourage the expectation and belief that the purpose of a tester is to find discrepancies and that registering them is a good thing. In fact, you will find a majority of the problems are faults in the test scripts rather than the actual solution.
Test Control Log Progress will be tracked in a test control log. At any point in the process it should be clear what the status is for each test, along with the various incidents that were reported.
Test Incident Control For each discrepancy in the testing a test incident control form notes the details and tracks any remedial action.
Test Incident Control Log Test incidents are logged and tracked to completion.
Test Signoff When a test is completed, it should be reviewed and formally signed-off by the test leader, the responsible user manager and anyone else who is been identified as an authoritative reviewer. In some cases you might note that a test was not entirely satisfactory although the problems were not sufficiently severe that you would wish to delay completing the tests or releasing the system. You should record what future action is required to remedy this problem. Ensure that appropriate remedial actions are defined and agreed for action at an appropriate time.

Specific Types of Testing

Here is a non-exhaustive list showing several types of testing. Consider which of these would be appropriate in your circumstances.

Note also that these types may have different characteristics when applied to different aspects of the overall solution. For example, you could pilot a new computer system, a training course or a new workgroup structure.

Specific methodologies will use their own terminology. You should note that several expressions can mean more than one thing. The best example here is "Conference Room Pilot" where we list five different usages we have heard, each at a different stage in the lifecycle.

Copyright ©  Simon Wallace, 1999-2016