Commercially Available Testing Tools

Once an application has been developed, the developers must demonstrate that it performs the tasks for which it was designed accurately, reliably and with adequate performance. For this to be fulfilled extensive testing must be carried out and tools have been built to assist with this process. Developers have built different types of tool for addressing different aspects of the same general problem. The importance of proper testing to detect as many errors as feasibly possible has been driven by the increase of malicious or criminal intent on the part of developers that produce applications with functions that facilitate fraud or other criminal activity (an especial risk to the financial industry). This problem has been addressed by European Community Legislation, increasing the onus on software developers to show that they took all reasonable steps to ensure an application was free of defects and suitable for the purpose for which it was developed. Failure to do so could leave the developer liable to be sued by anyone have has incurred a loss in any business as a result of software collapse. The main types of tool that have resulted as a partial result of this are described below. There are a large number of testing tools that are available, but they all work in very different ways. The main types of testing categories are described below. Tools that analyse source code without executing test cases, but in deriving test cases for the software to be tested. There are three different types used in industry that are described below: Code based testing tools accept source code as input and perform a number of analyses that result in the generation of test cases. This type of automated tool can broken down in to four further categories. The first are Code analysers that evaluate test modules automatically for proper syntax; statements are then highlighted where the syntax is wrong, if construction is error prone or if an item is undefined. The second category is Structure checkers where modules are submitted as input and a graph generated, depicting the hierarchy of modules and tools check for structural flaws, for example, determining the location of loops and branches and how they are used within the system. The third type are Data analysers which review data structures, data declarations and module interfaces, and notes improper linkage between modules, conflicting data definitions and illegal data usage. The final type are Sequence checkers where sequences of events are checked and marked if coded in wrong sequence. Specialised testing languages enable a software engineer to write detailed test specifications that describe each test case and the logistics for its execution. An example of one of these languages is Prolog, that is specifically used for test case generation. Requirements based testing tools isolate specific user requirements and suggest test cases (or classes of tests) that will exercise the requirements. Tools that analyse source code during execution of test cases by interacting with a program as it is executing and checking the path coverage, test assertions about the value of specific variables and otherwise instrumenting the execution flow of the program. They can be either intrusive or non-intrusive. An intrusive tool changes the software to be tested by inserting extra instructions or ‘probes† that perform the activities mentioned above. A non-intrusive tool uses a separate hardware processor that runs in parallel with the processor containing the program that is being tested. Systems can be difficult to test because several parallel operations are being carried out concurrently, which is especially true for real-time systems. Therefore it is difficult to anticipate the conditions and generate representative test conditions. However, dynamic test tools can capture a state of events during the execution of a program and so are often called program monitors, because they watch and report the behaviour of the program. The functions of the monitor are to list the number of times a submodule is called or a line of code is executed. These statistics tell testers if the test cases have statement coverage. Another function is to report on whether a decision point has branched in all directions, providing information about branch coverage. System performance information is also provided, including statistics about particular variables e. g. their first value, last value, minimum and maximum values. Breakpoints can be defined for the system, so when a variable attains or exceeds a specific value, the test tool reports the occurrence. Some tools will stop when breakpoints are reached so that the tester can examine the contents of memory or specific data items, as it is possible to change values as the test progresses. Any information captured during the test can be used to provide information about control flow. Another automated tool, analysers, are similar to monitors, except that they can also evaluate captured data to prescribed criteria. A test coverage analyser records the number of each statement executed during a test step and notifies us if certain routines or statements are not executed. A timing analyser works with predefined areas or memory or code and tracks the amount of time spent in each area as system functions are performed. This type of tracking can be useful during performance testing when timing requirements are checked. Tools that simulate functions of hardware or other externals by presenting to a system all characteristics of a system or device without actually having the system/device available. This is particularly useful if another company is developing part of a system; this part can be simulated to allow you to test your own part. The simulator can sometimes be more useful than the device itself as all data regarding the devices' state throughout the test can be stored, aiding in error location. Simulators also help with stress and volume testing, since it can be programmed to load the system with substantial amounts of data, requests or users. Generally, simulators give control over the test conditions, allowing you to perform tests that may otherwise be dangerous or impossible. Test management tools are used to control and co-ordinate testing for each of the major testing steps. Tools in this category manage and co-ordinate regression testing, perform comparisons that ascertain differences between actual and expected output and conduct batch testing of programs with interactive human-computer interfaces. In addition to the functions noted above, many test management tools also serve as generic test drivers. A test driver reads one or more test cases from a testing file, formats the test data to conform to the needs of the software under test, and then invokes the software to be tested. The C/S environment demands specialised testing tools that exercise the graphical user interface and the network communications requirements for client and server. This category can be sub-divided into the following functions: Reverse engineering to specification tools which take source code as input and generate graphical structured analysis and design models, ‘where-used' lists and other design information. Code restructuring and analysis tools that analyse program syntax, generate a control flow graph and automatically generate a structured program. On-line system reengineering tools which are used to modify on-line database systems. Many of the above tools are limited to specific programming languages, although most major languages are addressed and require some degree of interaction with the software engineer. Next generation reverse and forward engineering tools will make much stronger use of artificial intelligence techniques, applying a knowledge base that is application domain specific, i. e. a set of decomposition rules that would apply to all programs in a particular application area. The AI component will assist in system decomposition and reconstruction, but will still require interaction with a software engineer throughout the reengineering cycle. Several testing aids can be combined into one automated tool; a test harness is a monitoring system that tracks test input data, passes it to the program or system being tested and records the resulting output. A test harness can also compare actual with expected output and report any discrepancies. Most test harness tools are environment specific by the nature of the process. Test data set generators can generate test data sets derived from the requirements modelling process. Used in conjunction with test harnesses they will provide a formal documented test environment. In most cases a combination of the above tools will improve chances that a delivered application performs the tasks expected correctly and reliably. All testing tools generate large amounts of information about an applications structure. This information must be interpreted and used to detect and rectify subtle logic and structure error. There is a large amount of interest in producing automated support for this interpretation process; to pinpoint possible problem areas and suggest further lines of investigation. With the exception of Interpreters, that are still in development, the above categories of testing tools are available commercially. There are a large number of products available produced by many different companies, so two case studies have been selected to give an impression of the testing tools commercially available. Where possible, the category of testing tools as described above that each product fits into has been added in brackets after the product name. The current products available from this French company are aimed at user interface testing and there are three product lines. The first, UniTest, is designed to perform unit testing of embedded systems. It can develop test scripts that can run on native, simulator, emulator or target platforms. ATTOL's second product, SystemTest, automates the production and exploitation of integration and validation tests for systems. Both of these two products can be integrated with ATTOL's final product, Coverage (test coverage analyser), which is a code coverage tool that is designed to obtain the level of code coverage during the unit or integration testing. TestStudio is one of four products that make up the software development product, Rational Suite. The TestStudio product is itself made up of other Rational Products. Rational Robot provides thorough testing of an entire application, Rational TestFactory automatically detects run-time errors without user assistance and generates optimal scripts for regression testing. Rational Purify locates hard-to-find run-time errors that cause program crashes. Rational Quantify pinpoints performance ‘bottlenecks' in applications and Rational PureCoverage (test coverage analyser) identifies untested code and provides code-coverage analysis. The nature of many products available is that they perform testing to meet user requirements. To do this they are often a combination of several types of testing tool, which makes it difficult to identify specific categories of testing. However, many of the products available did require the system or application being tested to actually be run, whether on a simulator or real-time, suggesting dynamic testing is used more than static testing. There are however, a huge range of testing tools commercially available, combining many different testing methods.