Lecture
The original benchmark for testing of any program is the requirements of the technical specification and/or requirements specification of the customer and potential user, imposed on the programs being created. Such documents must establish the composition, content and values of the results that the user should obtain under specific conditions and initial data. Any deviation of the program's operating results from the requirements imposed on it and the benchmarks formed from them should be classified as an error or defect in the program.
Obtaining reference values of parameters and characteristics of test results for computational programs is relatively simpler, primarily because a significantly smaller number of control values is required. Intermediate result values can be determined by interpolation or omitted altogether, based on the assumption of smoothness of the computed functions, i.e. it is not mandatory to prepare benchmarks for absolutely all combinations of initial data. For the same volume of the program being tested, preparing benchmark values for logical programs becomes more complex, primarily due to their combinatorial nature and the increased number of test values required. In this case, tests must, in principle, be calculated for all possible combinations of initial data, since each of them can completely change the domain and semantic meaning of the resulting values.
Using mathematical models or prototypes implemented on a computer is the most effective way to obtain benchmark values and the required characteristics of complex program operation. Two types of models can be created: based on more complex and more accurate algorithms which, for example, cannot be implemented on the target computer due to limited resources, and based on simplified, generalized models for faster obtaining of benchmark values. The second type of model is naturally characterized by lower potential accuracy of results; however, due to reduced complexity, errors are less likely in them.
The operating results of real programs that are predecessor-prototypes or pilot projects of the version of the software system being developed, in order to be used as benchmarks during testing, require ensuring the identity and repeatability of the initial data used by the programs under test when obtaining benchmark values. At the final stages of software system development, the formalization of benchmark values is in some cases not carried through to completion, and the informal representation of the developer or customer serves as the benchmark. In all cases, it is advisable to record and retain the values used as test benchmarks in order to ensure the repeatability of testing sessions.
When comparing the operating results of the programs under test against the benchmarks, one should use criteria for evaluating tolerable deviations from the benchmarks and decisions on the degree of correctness of the program. The magnitude of the tolerances depends on the type of algorithm being tested, the method, and the stage of program correctness verification. For simple logical algorithms implementing certain decision-making schemes, the analysis of test results usually uses a deterministic criterion of absolute identity between the tested and benchmark solutions given the same initial data. In computational algorithms and when testing complex logical algorithms, comparison with the benchmark has to be carried out statistically.
The selection and sequencing strategy of components for testing in large software systems depends on their architecture and the actual composition of components ready for use. In top-down testing from requirements it begins with the programs that organize the computational process. Initially, the control core of the software complex and the programs for solving functional tasks, located at the higher hierarchical levels, are tested for compliance with the initial requirements of the technical specification. Components of lower hierarchical levels are connected to them sequentially, as they become ready. Such a strategytop-down is effective when there is a sufficiently complete set of ready-made, proven software components and/or modules previously tested in versions of similar or pilot software complexes.
If some of the lower-level programs are not developed or not sufficiently tested, program simulators — «stubs» — may be temporarily used in their place. As a result, at the initial stages of testing, models of functional groups of programs or of the complex, with a certain number of simulators of program components, are checked. The advantage of this testing strategy is the preservation and consistent development of test initial data as components are connected. However, testing groups of programs with stubs may require considerable effort to detect the simplest errors in newly developed and connected modules, if they had not been sufficiently tested autonomously beforehand.
In systematic bottom-up testing, first of all, the program components and/or modules of the lower hierarchical
levels within the functional group of programs, to which the modules that call them are sequentially connected. In these modules, testing also begins with the simplest constructs, variables and information processing routes. Correspondingly, the testing methods used and the types of errors detected become progressively more complex. Sequential build-up of components in the program complex from the bottom up makes it possible to check the operability of such groups in their natural execution, without substituting or simulating lower-level components. The main difficulties with this strategy consist in the need for continuous updating and increasing the number of test sets as each new higher-level component is connected. At the same time, testing of lower hierarchical-level components is deepened, which contributes to a systematic improvement in their quality.
Top-down and bottom-up testing differ not only in the scheme of analyzing modules or small components, but also in the fundamental goals of the entire process of testing large program complexes. The main goal of top-down testing — is to verify the requirements for modules and program components, and to achieve their quality through autonomous testing of each of them outside of real time. Thus, in the process of decomposition, the required quality of the components and their compliance with the initial requirements must be ensured. In bottom-up testing, the main task — is to ensure the consolidation, integration and correct interaction of all components for the complete solution of tasks by the required program complex. This assumes a sufficiently high quality of previously prepared components. The fragments of these basic testing processes presented in this chapter are schematically combined in fig. 13.5. It is useful to keep such a diagram in mind when planning a testing strategy for large software systems.
Taking into account the specifics of applying methods and the technological stages of the life cycle of program components, the tasks of bottom-up testing of the following objects: are considered sequentially below in this section.
— formalized specifications of requirements for software and information modules, for groups of programs, and for program complexes;
software modules programmed and prepared for testing at the level of program source code and at the level of object code of the implementing computer;
autonomous groups of program modules and components solving complete functional tasks;
functional components within software tools.
Requirements for the characteristics of the system
Fig. 13.5
The task of testing specifications consists in verifying the completeness and mutual consistency of the functions prescribed to program and information components by the requirements of different hierarchical levels (see section 13.1). In addition, testing tasks include checking the consistency of descriptions of information at the inputs and outputs of interacting program modules and groups of programs, as well as with the descriptions of information modules in the database. As a result of testing specifications, their correctness and consistency must be ensured within the generalized description of requirements for the functions of the entire software system and the interaction of all its component parts. It is advisable to carry out testing of interconnections starting from the specification of requirements for the complex or group of programs. Sequentially, at each hierarchical level, it should be traced that the functions implemented by lower-level programs, as prescribed by the program specifications, provide for the upper-level programs. At the same time, the completeness of the fulfillment of these functions by the specifications of information modules is checked (see fig. 13.2).
The process of testing program modules consists in verifying the correctness of the processing by the modules of incoming information and of the resulting output data in accordance with the functions presented in the requirements specifications. The correctness of the module structure and of the constructive elements used — loops, blocks, switches, etc. — must be verified. Since the largest number of specialists, often of not very high qualification, are involved at this stage of testing, testing methodologies and the regulation of the use of automation tools become particularly important.
The information processing routes in each module and the correctness of their implementation depending on the initial data are subject to verification. Test completeness is determined by the criteria for selecting routes for testing and by the degree of test coverage of the specification requirements and of the possible program execution routes. On each selected route, the correctness of the computations performed for certain fixed initial data must be verified. This reveals errors of incomplete composition or incorrectness of conditions in the implementation of particular data-processing routes, as well as certain variable-conversion errors. For each selected route, a set of conditions determining its implementation is formed from the program text and is used when creating the corresponding test. Such a representation of routes makes it possible to systematically monitor the level of route verification achieved and, to some extent, guards against the accidental omission of individual untested routes.
Autonomous testing of functional components with the execution of programs is intended for verifying the correctness of solving individual sufficiently large functional tasks. At this stage, the correctness of control and information links between groups of modules is verified, as well as the correctness of the implementation of requirements in the process of processing information within a group of programs. In this case, the complexity of the objects under test increases significantly and, correspondingly — the size and complexity of the tests. As a result, the requirements for test automation and the costs of its execution increase. Deterministic testing should verify the structure of the group of programs and the main information-processing routes. In some cases, results should be obtained using stochastic testing methods.
Integration testing consists in combining program code corresponding to two or more program modules, and testing the resulting code. This should ensure that together they work as required, prior to full integration and testing of the code of each functional component. Since individual modules may include other modules, some part of the integration and testing of modules may take place during module testing. Test cases should cover all design requirements at the level of the software system's functional components. After this, all necessary software system changes related to correcting defects identified during verification should be carried out, along with retesting to the required extent, and the software system's development files and other software products should be modified based on the results of integration testing.
Testing of functional components within software tools during the development of program complexes, and the assessment of testing completeness, are carried out mainly according to the degree of fulfillment of the required functions and according to the characteristics of the correctness and quality of the software system's operation achieved as a whole (see fig. 13.2). Systematizing the types of testing and carrying them out in an orderly manner during development can be of considerable help in improving the quality of complex, critical software systems. These types of testing should be oriented toward differentiated detection of certain classes of defects. For each type of testing, it is advisable to develop a methodology for its execution, indicating the components to be checked, the controlled parameters, and the expected and reference results. In addition, during final tests or certification, integral testing should be carried out with the widest possible variation of tests under conditions corresponding to normal and forced operation.
Testing the completeness of the solution of functional tasks with typical initial data is intended to detect operational defects under normal, standard conditions defined by the requirements of the technical specification for the baseline version of the software system. The primary benchmark is the goals and objectives of creating the software product. Some part of the tests may contain deterministic initial data, for the analysis of which various graphical display systems are often used. Particular attention should be paid to test variants that revealed errors. Additional testing should be carried out for these conditions.
Testing the operation of programs in critical situations according to the conditions and logic of solving tasks {stress testing of fault tolerance)is carried out when running programs in abnormal situations, which rarely occur but are important for ensuring the quality and reliability of the information processing system's operation. To develop such tests, scenarios of critical combinations of initial data values and task-solving conditions are created, under which it is necessary to verify program operation and result distortions and failures can be expected. Such stress, abnormal combinations are prepared manually, or their implementation is provided for as part of the data of real-time stochastic test simulators. The particular importance of verification in critical situations is determined by the danger of such errors manifesting themselves when the software system operates under real conditions. Therefore, external environment simulators that automatically prepare initial data, and monitoring tools that respond to anomalous results of the execution of the tested programs, are actively used during testing.
The main stages of systematic testing and trials of a large real-time program complex and its components are presented in fig. 13.6. During testing and trials of the correctness of the functional components of program complexes, the following stages are distinguished:
— integration of modules and testing of autonomous functional groups of programs statically, without interaction with other components and, possibly, without connection to the real-time operating system;
testing of functional groups of programs statically, taking into account interaction with certain other essential components and with the database;
testing of individual program components in real time in interaction with other functional components and with the main components of the operating system and database.
Fig. 13.6
The complexity of testing components at these stages is largely due to the asynchronous process of their development and debugging by individual specialists in teams. The originally planned logic of interfacing individual components with each other and connecting them to the operating system is not always followed, due to delays in the development and autonomous debugging of some of them. The advisable sequence of testing certain components may be disrupted by the unreadiness of other interacting programs to interface with them.
To simulate external environment objects and other interacting groups of programs, deterministic control tasks or private generators of the corresponding tests are used predominantly at these stages. It is advisable to develop and design these test generators as separate modules or groups of programs, operating on the same technological computer and in the same operating environment as the components being debugged. Together with them, private specialized programs for processing and summarizing individual testing results of the corresponding groups of programs are also implemented and operate.
After the integration of the main functional components, their testing and trials as part of the software system as a whole begin. The following stages of qualification testing and trials of the software system in real time are most characteristic for them (see fig. 13.6):
based on data from a simulation bench or test generators simulating individual external environment objects;
with simulators of individual external environment objects and with real actions from operator-users;
in a fully adequate real or simulated external environment and with real actions from operator-users (see lecture 14).
At all stages of testing, apart from operations of directly verifying program operation, two more important groups of work can be distinguished. The first group — consists of work on the methodological support of testing processes and on the creation of automated test generation tools. The second group of work should ensure the possibility of processing test results and correctly assessing the achieved characteristics of program operation quality.
Test generation and test-result processing tools can be divided into three types (see fig. 13.6). The same static testing automation tools usually provide for debugging groups of programs both autonomously and in interaction with other components. Tools that simulate the external environment in real time are most often oriented toward testing both functional components and the software system as a whole. Yet another type of test generator uses real external environment objects to one degree or another. Initially, such objects are simulation benches involving the real operation of operator-users (see lecture 14). Later, sources of tests may be complexes of real hardware of external objects or their hardware analogues.
The scheme considered is oriented toward the testing and trials of a program complex located on a single hardware platform. When creatingdistributed client-server systems additional, quite complex testing tasks arise. Software tools located on the hardware platforms of clients and servers must first be tested autonomously according to the complete program presented above, and then their interaction must be additionally tested. This interaction of the client and server parts of the software complex must be prepared through autonomous testing of the entire telecommunication system and guaranteeing its quality. After this, the last three stages of comprehensive real-time testing presented in fig. 13.6 should be repeated, but now with full interaction of both functional components of the client-server system.
Comments