Lecture
To evaluate the characteristics of and test software systems at various stages of the life cycle, it is advisable to use the recommendations of the ISO 14598:1-6 — Evaluation of software product — standard as a methodological basis. The standard presents the process of evaluating a software system as a set of actions performed in cooperation between the customer and the evaluator — the tester. Potential customers for the evaluation of a software system may be developers, suppliers, purchasers, users of the software system, manufacturers of information-processing systems, and also independent testing laboratories for software products. To obtain the greatest effect from the test results, it is recommended that the evaluation be, as far as possible:
objective — the evaluation results must be based on real facts, not colored by the feelings or opinions of the testers;
repeatable — repeated evaluation of an identical product against an identical specification by the same tester must give the same results as the initial evaluation;
reproducible — evaluation of the same product against the same specification by different specialists must give the same results as in the previous test;
— unbiased — the performers of the evaluation process must remain unprejudiced toward any particular result.
The general scheme of the processes for evaluating the characteristics of program complexes consists of (Fig. 14.1):
Formalization of the initial requirements for evaluating the software system:
defining the goals of testing and quality evaluation at intermediate and final stages of the software life cycle;
identifying the type of software system;
identifying the consumers of the evaluation results;
identifying the features of the evaluation model and the composition of the required quality characteristics
Formalization of the principles for evaluating characteristics and testing the software system:
selection of the software system's characteristics and classification of requirements for measurements and tests;
establishing priority levels for quality characteristics and attributes;
identifying criteria for the expert review and comparison of quality characteristics against requirements
*
Designing the processes for evaluating and testing the software system's characteristics:
developing plans for evaluating the software system's characteristics in accordance with user needs and the stages of the life cycle;
designing the processes for evaluating characteristics and testing the software system
Implementing the processes and using the results of evaluating the software system's characteristics:
performing tests to evaluate the actual values of the software system's characteristics;
comparing the test results against the criteria for the required characteristics of the software system;
evaluating and summarizing the test results for the software system's characteristics
Fig. 14.1
formalization of the initial requirements for evaluating the values of the software system's characteristics, defining the testing goals, identifying the consumers of the test results;
formalization of the principles and features of evaluation when carrying out expert reviews, measurements and tests of the software system's characteristics, identifying criteria for comparing the obtained characteristics against the requirements;
planning and designing the processes for evaluating characteristics over the software system's life cycle in accordance with the needs of the users of these characteristics;
implementing the processes of testing, measurement and evaluation of the quality achieved by the software product, comparing the test results against the requirements; documenting and using the results.
The first part of the standard presents the concept of planning and managing the processes for evaluating program characteristics, as well as their relationship to the life-cycle management processes of the software system (per ISO 12207). When preparing for testing, it is recommended to structure the technology and procedures for applying the specific software system with the aim of sequential, detailed evaluation of groups of functional characteristics or individual quality attributes at the stages of the life cycle.
The decision to carry out an evaluation of a software system's characteristics may be made during development and throughout the entire life cycle. If such a decision is made at the initial stage of development, it becomes possible to build tests and measurement tools for testing into the development process. This ensures maximum success in satisfying all requirements regarding the results of evaluating the software system's characteristics and minimizes the risk of errors in extreme, unplanned situations. When the customer is the developer of the software system, early contact with the evaluator for testing or expert review can help the developer take into account certain special requirements from the tester. For very large, complex software system projects, it should be advantageous for the developer to have detailed cooperation with the characteristics evaluator throughout the entire product development process, in order to minimize the duration and cost of the testing process. It should be the responsibilityof the customer of the testing to:
— establish the legal rights it needs to perform testing of the software system;
provide the tester with the information necessary to identify and describe the software system and its components;
establish the initial requirements and enter into negotiations with the tester to work out realistic evaluation requirements;
subordinate the testing requirements to the relevant regulatory documents and standards;
establish the confidentiality requirements for information transmitted for testing;
where necessary, provide support to the tester, including training and access to suitable documents;
guarantee timely delivery to the tester of the description and components of the software system, including documentation and other materials;
inform the tester of any possible factors that could devalue the test results.
The tester should analyze the technical constraints relating to the measurements or verifications specified in the evaluation specification, and document a suitable method. An evaluation method — is a procedure describing the action performed by the tester to obtain the result of a specified measurement, expert review or verification for a system, component or software system. When the evaluation method is described on the basis of software tools, that tooling must be identified in the test plan and methodology. Here the responsibilities of the evaluator — the tester include:
verifying the customer's legal rights to the system and software system being evaluated, for which the tester may require the appropriate documents from the customer;
maintaining confidentiality of all information transmitted by the customer, including the software system's texts, records of results and the test report;
providing qualified and trained personnel to perform the tests;
providing the tooling and technology for evaluation;
carrying out the tests in accordance with the evaluation requirements and the customer's specification;
keeping records of any work performed during the evaluation that affects the results;
— ensuring the visibility of the testing process for observation by the customer and the timely delivery of the evaluation report to the customer.
The evaluation requirements must contain a general description of the software system's field of application and consist of a list of requirements for the characteristics. The relative significance (priority) of specific characteristics in the requirements must be indicated. For each item in the evaluation requirements, a specification of the information contained in the software system's components must be presented.
The evaluation — testing specification must define the scope of expert review and measurement of the various components of the product being submitted for evaluation. The level of detail in the test specification must be such that it guarantees the repeatability and reproducibility of the tests. The customer must provide a description of the product being evaluated. The purpose of such a description is to define the scope of the evaluation and to identify those components that are considered part of the product, as distinct from software system components that are only referenced to facilitate understanding of the product's functions. This should make it possible to decompose the evaluation requirements into subcharacteristics. The tester should then specify the measurement and expert-review methods intended for analyzing the characteristics, subcharacteristics and quality attributes of the selected components. The evaluator must verify the evaluation specification against the requirements, establishing that the components listed in the product description contain all the information needed to carry out the evaluation in accordance with the requirements.
The test or expert-review plan for the characteristics of software products must consist of the following sections:
introduction — the statement of the problem and the goals of the evaluation — testing;
the methodology for ensuring the objectivity of the tests of the software system's characteristics;
the characteristics and quality and security attributes of the software system identified for the project under the ISO 9126:1-4 standard and other standards;
the required quality of the software system and the reliability of the testing processes;
the schedule for performing the work on testing the software system's characteristics;
the distribution of duties and responsibilities of the specialists in evaluating the quality characteristics and attributes;
analysis and use of the test results;
the content and format of the reports on the performance of the tests;
additional requirements regarding equipment, tools and methods for servicing the tests, and for compliance with standards and organizational support for measurements and expert reviews.
The tester must provide the input data for the evaluation process: the predefined evaluation specifications; the methods and tools. The evaluation — testing process is recommended in the standard to consist of the following actions:
analysis of the evaluation requirements, during which the customer's real testing requirements are identified;
specification of the processes and possible results, during which the evaluation specification is developed on the basis of the requirements and the description of the software product provided by the customer;
design of the testing processes, during which the plan is developed on the basis of the evaluation specification, the software system's components and the methods proposed by the tester;
the process of carrying out the evaluation plan, which consists of modeling, expert review, measurement and testing of the software system's components in accordance with the plan, using software tools, as well as the actions performed by the tester, which are recorded and whose results are entered into the report;
the evaluation conclusion, which consists of providing a report on the results of testing the components and the software system.
During testing, it may be necessary to use software tools to collect processed data or to carry out the interpretation of intermediate data. The evaluator's staff must be trained to use the appropriate tools. During the evaluation, the following data must be obtained, recorded and presented to the customer:
the requirements, which describe the goals of the evaluation, in particular the required criticality and safety of the product being tested;
the evaluation specification, which defines the entire scope of analysis and the measurements to be performed, and all the software system's components that must be analyzed and measured;
the evaluation plan, which describes the operating procedures needed to carry out the evaluation specification, in particular all the methods and tools used in the evaluation;
the evaluation records, which consist of the evaluation plan and the evaluator's detailed actions while carrying out the plan;
the characteristic evaluation report, which contains the requirements, the evaluation specification, the results of the measurements and analysis, as well as any other information needed for the evaluation to be repeated or reproduced, and also a summary report for the customer.
The purpose of the conclusion is to review the test report and pass the evaluation data on to the customer. A joint review and analysis of the report by the customer and the tester should be organized. The customer should be given the opportunity to make comments on the report. If such comments are made, they should be entered into a special section of the report, after which it must be passed to the tester and the customer.
From the standpoint of different consumers of the results of measuring and evaluating the quality of the software system, the third, fourth and fifth parts of the ISO 14598:1-6 standard — respectively for:
developers — evaluation of internal and external quality characteristics (Part 3);
operational users — measurement of external metrics and metrics in use (Part 4);
customers and testers — determination of metrics in use (Part 5).
In each part, similar sections are identified and detailed: the features and needs of specific users regarding the test results and the range of required software system characteristics; the concept of carrying out testing and quality evaluation; defining the requirements for the processes of testing the program characteristics; identifying the software system's quality characteristics and attributes for specific consumers of the test results.
It is proposed that the results of the characteristics evaluation be reflected from the standpoint of: the life-cycle processes; the products and their components; the operation and application of the software system. It is recommended that the requirements for the evaluation processes be structured into the main (functional), organizational and design requirements, and that internal and external quality metrics and their measurement also be identified, based on the subcharacteristics and their attributes in the relevant parts of the ISO 9126:1-4 standard. The performance of the software product testing processes under the requirements of the standard must be carried out by qualified and certified specialists, independent of the developers of the project and of the processes for creating the software system and its components, though correlated with the stages of the life cycle of the specific project in accordance with the adapted version of the ISO 12207 standard being applied.
Attention is drawn to the advisability of taking into account and using the history of the development and changes in the operational requirements of the customers-consumers for a given project and the functional purpose of the software system. In the test report, it is recommended that the quality characteristics be presented in accordance with the range and measures of subcharacteristics and attributes, adapted to the purpose, functions and features of the specific software system project and of the consumers of the measurement results.
The confidentiality of all components and documents of the software system's testing must be protected by the evaluator. The confidentiality requirements cover many aspects of the evaluation work, including the receipt, management, storage and transmission of all information relating to the product's characteristics. The confidentiality of intermediate test data must be protected, the same as the confidentiality of the original components and documents. The tester must make an effort to prevent any accidental, erroneous or malicious modification of this data.
The quality characteristics of software system operation depend not only on its internal properties, but also on the properties of the external environment in which it is applied (see ISO 12119). To reduce uncertainties and outright errors when evaluating the quality of a software system, it is necessary, before testing begins, to determine the main parameters of the external environment under which the program complex must operate with the required characteristics when its quality and operation are being evaluated.
For this, the customer and the developer must jointly structure, describe and agree on a model of the external environment and its parameters in the average, typical mode of applying the software system, as well as in the most probable and critical modes in which the required quality characteristics of the software system's operation must be ensured. Such a model must reflect and record the characteristics of:
the external information flows, including their distribution by source type, data quality characteristics and the possibility of defects in them;
the intensity and structure of typical messages from operational users and administrators and their required qualifications, reflected in the probability of errors and the quality of the information issued;
possible negative and unauthorized influences from the external environment when the software system is applied;
the required characteristics of the computing equipment on which the program complex is intended to operate with the required quality.
When comparing the results of evaluating the quality characteristics against the requirements of the technical specification and the specifications, the developer or supplier is obliged to satisfy the customers' requirements only within the bounds of the agreed parameters of the external environment model. Evaluation of the software system's quality beyond these bounds must additionally be agreed upon by the testers with the developer. In this case, failure to meet the requirements may be classified as an extension of them beyond the bounds limited by the contract, and not taken into account by the customer when evaluating the software system's quality characteristics, or as additional work subject to corresponding financing by the customer, for the purpose of modifying the programs so as to satisfy these requirements.
Internal qualification tests of the quality of software systems (chief-designer tests), which are often combined with the completion of comprehensive debugging, must be formally documented and serve as the basis for submitting the software system to the customer for qualification testing for the final evaluation of the software product's quality characteristics (see ISO 12207, ISO 15504, ISO 16326). The developer must implement and evaluate the design, the program complex, the tests, the test results and the documentation for the user, taking into account:
the completeness of the testing coverage of all the specification requirements for the components and for the software system as a whole;
consistency with the results required by the customer and the expected results of applying the software system;
the possibility of integrating and testing the software system as part of the system;
the possibility of operating and maintaining versions of the software system in accordance with the requirements of the contract.
Any testing is limited by the permissible number and volume of checks, as well as by the duration of the testing commission's work, and therefore cannot guarantee an absolute verification of the product's quality. To increase the reliability of the determination and improve the evaluation of the software system's characteristics, after the internal tests it is advisable to hand the program complex over to some users for trial operation under typical conditions. This makes it possible to evaluate the operational characteristics of the created complex more thoroughly and to eliminate certain defects and errors. It is advisable for the trial operation to be conducted by the developers with the participation of the customer's testers and certain users appointed by the customer. The results and quality characteristics of the trial operation, following the chief-designer tests, may be taken into account by the customer when conducting the qualification tests, in order to reduce them.
Lecture 13 examined the stages of testing the components and the software system as a whole from the standpoint of a sequential increase in the functional complexity of the tests and interaction with objects of the external environment. However, the organizational stages of testing in accordance with the standards and their accountability to the developers-suppliers and customers were not taken into account there. The stages and processes of qualification testing of the software system for the purpose of formally certifying to the customer the achieved characteristics of the program complex's quality and its components as part of the system are regulated in the standards ISO 12207 and ISO 15504. These identify three main, functional stages for carrying out qualification testing and tests (Fig. 14.2):
— qualification testing of the functional components and of the software system as a whole, outside the system's hardware;
integration and testing of the software system as a whole as part of the system's hardware;
qualification testing and full testing of the system together with the software system.
Fig. 14.2
Qualification testing of the functional components and of the software system as a whole is performed in order to demonstrate to the customer that all the requirements of the contract have been implemented and the necessary quality of the program complex's operation has been achieved. Qualification testing must cover all the requirements for the components in the software system's requirements specification and in the interface requirements specifications. Testing of the components for each configuration must show that the requirements for the components to be implemented in that configuration are fully satisfied. The persons responsible for the qualification testing of the components must not be the persons who carried out the working design or programming of the corresponding component. This does not exclude the possibility of assistance being provided in conducting the qualification testing by the persons who carried out the working design or the programs, for example, by supplying test cases based on their knowledge of the internal implementation of the component or software system.
The developer must define and register the process of preparing for testing, the test cases, scenarios and procedures to be used for the qualification testing of the component, and trace the correspondence between the test cases and the requirements of the contract. It must prepare the test data needed to execute the test cases, and notify the customer's representative in advance of the time and place of the qualification testing. If the testing of a component is to be witnessed by the customer's representative, then before it is conducted the developer must check the test cases and test procedures to ensure that they are complete and accurate and that the software system is ready for testing in the presence of the customer's representative. The results of these checks must be recorded in the software system's development files, and the test cases and procedures modified accordingly to eliminate the defects identified. All necessary changes to the software system should be made, having first notified the customer's representative, retesting should be carried out to the necessary extent, and the software system's development files and other software products should be modified based on the results of the integration and testing of the modules and components.
The results of these actions must be included in the report to the customer on the preliminary tests conducted by the developer.
Integration and testing of the software system as part of the system's hardware consists of combining them, testing the resulting complex in order to determine whether they work together as required by the contract, and this process must continue until integration and testing have been carried out for all the program and hardware components. The test cases must cover all aspects of the system level and the required quality characteristics of the design. The developer must make the necessary corrections to the programs, take part in retesting to the necessary extent, and modify the software system's development files and other software components based on the results of integration testing.
Qualification testing of the system and of the software product as a whole is performed to demonstrate to the customer's representative that all the requirements of the technical specification are satisfied and that the quality characteristics meet the conditions of the contract. It must cover all the requirements in the system and subsystem specifications, as well as the requirements for the interface with the external environment. The tests must include testing on the target computing system or on an alternative system model approved by the customer's representative. The developer must participate in developing and registering the process of preparing for testing, the test cases, scenarios and test procedures to be used for the full system test, and in tracing the correspondence between the test cases and the requirements for the system's quality characteristics. Each requirement being verified must correspond to specific substantiated system characteristics, have an identifier unique to the project, so that testing can be carried out and its fulfillment traced by means of an objective test. For each requirement, qualification methods must be selected for the subsystems and components of the software system, which need to be traced back to the system requirements. The degree of detail of the requirements should be chosen taking into account, first of all, those quality characteristics that are included in the conditions for the system's acceptance, and giving priority to those which the customer requires to be provided without fail.
All the results obtained must be included in the report on testing the software system and the system. If the qualification testing of the system is to be witnessed by the customer's representative, then before it is conducted the developer must check the test cases and test procedures to ensure that they are complete and accurate and that the system is ready for testing in the presence of the customer's representative. The tests must be performed in accordance with the test cases, scenarios and procedures approved by the customer.
Evaluation of the software product's quality during the qualification and acceptance tests is carried out by a customer commission, which includes the head (chief designer) of the development effort and certain lead developers, or a certified certification laboratory. During testing, the commission must be guided by the following documents (see Fig. 14.2):
— the contract, technical specification and requirements specifications for the software system, approved by the customer and agreed with the developer;
the current state and departmental standards on the life cycle and testing of programs, on the technological and operational documentation, as well as de facto standards agreed with the customer for use — the profile of standards and regulatory documents;
the Test Program covering all the requirements of the contract, the technical specification and the specifications;
test procedures covering every section of the requirements of the technical specification, the specifications and the Test Program;
a complete set of adequate operational and technological documentation for the program complex.
The Test Program is a plan for conducting a series of experiments and must be developed from the standpoint of the permissible minimization of the volume of testing during the testing process, in order to verify the fulfillment of the technical specification's requirements and compliance with the submitted documentation. The Test Program, the procedures for carrying it out and for evaluating the results, developed jointly by the customer and the developer, must be agreed upon and approved. They must contain refinements and details of the requirements of the technical specification and specifications for the given software system, and must also guarantee the correct verification of all the specified quality characteristics. The Test Program must contain the following clearly formulated sections:
the object of testing, its purpose and a list of the main documents that determined its development;
the goal of the testing, indicating all the requirements of the technical specification, the characteristics and quality attributes subject to verification, and the constraints on conducting the tests;
the Test Program itself, containing verification of the completeness of the designed software system in accordance with the technical specification and a plan for conducting testing to verify compliance with all sections of the technical specification and the additional requirements formalized by separate decisions of the developers and the customer;
test procedures that unambiguously define all the concepts of the quality characteristics being verified, the testing conditions and scenarios, and the tools used for the tests;
procedures for processing and evaluating the test results for each section of the Test Program.
A test procedure must contain: a description of the organization of the testing process, the test cases, scenarios and procedures used for testing an individual component or the software system as a whole. Each test must have an identifier unique to the given project; instructions for conducting the testing must be provided; a description of the hardware and software system for carrying out the testing, as well as instructions for retesting. In addition, references to the requirements being verified must be given, the execution conditions (hardware and software system component configuration) must be indicated, along with the input data, reference and expected results, criteria for evaluating the quality of the results, the procedure for conducting the testing for each test case, and the assumptions and constraints.
The test plan for the software system must describe the procedure for the qualification testing of the components and subsystems, the test external environment to be used during testing, must identify the tests to be performed, and must specify the schedule of testing activities. For each intended test implementation, the following must be indicated: the hardware versions to be used; interface equipment; additional external devices; test-message generators; test synchronization devices. In addition, the document must present the testing schedule and a matrix tracing the tests to the requirements of the specifications for the software system or its components, as well as the subcontractors taking part in the testing, their role and responsibility.
The large volume of heterogeneous data obtained during the testing of large-scale software systems, and the variety of possible ways of processing, interpreting and evaluating them, mean that the most important factors become the procedures for processing and evaluating the results, as well as the verification protocols for the items of the Test Program. In accordance with the test procedures, the automation tools must ensure the completeness of the verification of the characteristics for each section of the procedures. The test results are recorded in protocols, which usually contain the following sections:
the purpose of the testing and the section of the technical specification's requirements against which the tests were conducted;
an indication of the sections of the procedures in accordance with which the testing, processing and evaluation of the results were conducted;
the conditions and scenarios for conducting the testing and the characteristics of the source data;
the summarized test results with an evaluation of their compliance with the requirements of the technical specification and other governing documents, as well as the technical documentation;
a description of the differences between the test environment and the real operational environment;
a description of the defects and errors found and of the recommended improvements to the software system under test;
conclusions about the test results and about the compliance of the created software system or component with a specific section of the requirements of the technical specification and the original specifications.
The protocols for the entire test program are summarized in a certificate of acceptance, resulting in a conclusion about the system's compliance with the customer's requirements and about the completion of the work with a positive or negative outcome. When all the requirements of the technical specification are fulfilled, the customer is obliged to accept the program complex, and the work is considered complete.
The first baseline version of the software system must be subjected to the most complete and comprehensive testing. During the testing of subsequent upgraded versions of the software system, significant reductions in the volume of testing of proven, reused components are possible. However, the comprehensive and final tests of each new version of the software system are, as a rule, carried out in full, guaranteeing verification of the fulfillment of all the requirements of the modified technical specification. To detect defects during the operation of production units, each of them must be provided with a certain minimum set of tools for checking operation and detecting distortions in the results. These tools must make it possible to record the conditions of incorrect program operation and the nature in which defects manifest themselves when the software system is used. Subsequent correction of errors must be carried out by the specialists responsible for maintenance.
Before qualification testing of the software system begins, the means must be checked and certified — the means for obtaining reference data, the means for simulating tests from external objects, the means for recording and processing the test results. The qualification testing of the software system is completed by submitting to the customer for approval a set of documents containing the results of the comprehensive testing of the software version:
the corrected texts of the programs and data in the programming language and in object code, as well as the complete requirements specifications for the software components and the software system as a whole after the full completion of testing;
the Test Program for the software system covering all the requirements of the technical specification;
the set of test procedures and result-processing procedures for all sections of the test program;
the tests, scenarios and test-data generators used for testing the software and information components and the software system version as a whole;
the results and protocols of the qualification testing, and the functional and design characteristics of the software system in the real external environment;
the report confirming the specified quality, the complete characteristics of the quality of operation achieved, as well as the degree of test coverage of the software system's requirements specification;
the plan, procedures and automation tools for training the customer and users in applying the tested version of the software system;
the set of operational documentation, the description of the software system and the user manual in accordance with the conditions of the contract;
the technical specifications for the software system version, the database and the operational documentation for replication and series production;
the installation manual, the generation of the user version of the software system and the loading of the database in accordance with the conditions and characteristics of the external environment;
the report on the technical and economic indicators of the completed software system version project, the fulfillment of the plans and the resources used;
the certificate of test completion and readiness for delivery and/or submission for certification testing of the software system version.
The above organization of testing for large software systems is oriented toward the presence of a specific customer for the software complex and a limited number of users controlled by the customer. The testing of commercial application software packages, created on the initiative of a company or a team of developers for sale to a broad range of users in the absence of a specific customer, is organized somewhat differently. For such commercial software complexes it is customary to conduct qualification testing for compliance with criteria formalized by the project manager in two successive stages — Alpha-and Beta-testing. These consist of normal and forced (stress) trial operation of the finished software product by end users in accordance with the operational documentation, and differ in the number of participating users and their level of qualification.
Alpha-testing involves end users who work mainly within the same company but were not directly involved in developing the software complex. Beta-testing involves volunteer users (potential buyers) who are given a version of the software system free of charge for trial operation. Here the participation of competent, interested and well-disposed users, capable of detecting defects and improving the quality of the tested programs through their recommendations, is of particular importance. Their activity is motivated by free and early access to and mastery of the new software product, and by their own assessment of its quality. These users undertake to report to the developers information on all detected defects and errors, and to make changes to the programs and data or replace versions solely on the instructions of the developers. Only after successful operation and Beta-testing by a limited contingent of users can the project manager or the developer company's management decide to release the software system for sale to a broad range of users. The generalized results of Beta-testing can be used as a basis for certification testing.
During Alpha- and Beta-testing it is customary to distinguish between progressive and regression testing. Progressive — is understood as the testing of new program components to detect defects and errors in the program source texts and specifications. Regression testing is intended to control the quality and correctness of the programs and data after corrections have been made. The need for and scope of regression testing is determined by the fact that a significant proportion of the changes made after Alpha- and Beta-testing themselves, in turn, contain defects and errors. The number of tests and the duration of both testing stages are determined by expert judgment of the developers or the project manager, depending on the complexity of the software complex and the intensity of the flow of changes.
Ensuring the high quality of large software complexes requires appropriate problem-oriented integrated test automation systems, capable of adequately replacing the testing of programs with real objects of the external environment. At the same time, the high cost and risk of testing with real objects almost always justify substantial expenditure on such integrated systems when testing of critical software systems is required, with high demands on the quality of program operation, a long life cycle and multiple evolving versions. Instrumental facilities for automating the processes of testing and verifying programs must provide:
test definition — the developer's implementation of the testing process: input of test sets; generation of test data; input of expected, reference results;
execution of a section of the tested program between checkpoints, for which the testing tool can intercept operator input (keyboard, mouse, etc.) and for which the input data can be edited and included in subsequent test sets;
control of tests and of the program section for which the testing tool can automatically execute test sets;
analysis and processing of test results — the ability of the testing tool to automatically analyze test results: comparison of expected and actual results; comparison of files; statistical processing of results;
analysis of test coverage of the source code to detect: statements that were not executed; procedures that were not called; variables that were not referenced;
analysis of program performance while it is executing: central processor load; memory load; accesses to specified data elements and/or code segments; timing characteristics of the operation of the tested program;
simulation of the external environment — support for the testing process by means of a model simulating data from components of the information system external to the software system.
When creating generators of tests for the external environment, two fundamentally different approaches are used, which can conditionally be called the integral and the differential. In the integral or empirical-statistical approach, the basis is a formal description of the input and output information of the simulated object, as well as the functional relationship between the data at its input and output. Here the structure of the object and the processes that occur during the real operation of its components are of no significance and are not modeled. The initial data and characteristics for building such test generators are obtained from field experiments or from the study of more detailed — differential models.
Differential or simulation models of test generators are based on descriptions of the internal processes of operation of the components of the modeled object, its structure and the interaction of its constituents. The results of the operation of such models are determined by the adequacy of the knowledge about the components and their characteristics, as well as about their interrelationships. This requires sufficiently detailed information about all the processes of operation of the components of the external-environment objects, which, in turn, may require even deeper modeling of their constituents.
Unlike a field experiment, simulating the external environment and tests on a computer offers far greater possibilities for monitoring both the initial data and all the intermediate and output results of the operation of the tested object. In real systems, a number of components sometimes turn out to be inaccessible for monitoring their state, either because it is impossible to place sensors for the monitored signals in the real subsystems being tested, or because this would involve changing the characteristics of the object being analyzed itself. Another advantage of simulating the external environment on a computer is the repeatability of the operation results and the possibility of investigating a large number of testing variants and scenarios. In contrast, field experiments can often not be stopped at some intermediate phase or repeated with exactly the same initial data. Software simulation of the external environment on a computer makes it possible:
carry out prolonged, continuous generation of simulated data to determine the quality characteristics of the operation of the software system over a wide range of changing conditions and parameters, which is often impossible when using real objects;
extend the ranges of characteristics of the simulated objects beyond those actually existing or available from data sources, and also generate information flows reflecting the prospective characteristics of the information systems being created and of the external-environment objects;
create test data corresponding to critical or hazardous situations in the operation of external-environment objects, which are impossible or risky to reproduce in field experiments;
— ensure a high degree of repeatability of the simulated data under given conditions of their generation, and the ability to terminate or suspend the simulation at any phase of external-environment modeling.
Among the most complex and expensive external-environment simulators used for testing software complexes are models of: spacecraft flight; air-traffic-control dispatch centers; objects of air-defense systems; complex administrative systems. Such simulation test rigs (STR) are problem-oriented, and the volume of the programs simulating the external environment in them may even significantly exceed the volume of the corresponding software system under test. Sufficiently powerful universal simulation computers (Fig. 14.3) are allocated for their implementation. In addition, separate specialized host computers, can be used to automate program development, which together form the toolset providing for the entire life cycle of complex real-time software systems on target implementing computers.
The simulation of specific tests with real characteristics, adequate to the external-environment objects, forms the main part of typical simulation test rigs. In accordance with the full range and real characteristics of the objects, their integral or differential models are created. The choice of model types depends on the depth of knowledge about the algorithms governing the operation of the objects, the characteristics of their components, and the generalized parameters of the operation of the object as a whole. In addition, significant factors for the choice of model types are the duration of the calculation of the simulated data and the need to ensure the ability to carry out a full simulation of the external environment on the simulation computer in real time, taking into account the time spent processing the results.
When implementing the real time scale is difficult given the available computer performance, models of objects that have no feedback influence from the software system under test should be identified. For such objects, it is possible to carry out the main part of the simulation in advance, outside real time, recording for each test message the real-time value at which it should be issued for processing by the software system. Models of objects whose results depend on the operation of the software system under test can sometimes be simplified by isolating the algorithm implementing the real-time feedback and supplementing the main body of tests, prepared outside real time, with these corrections.
Fig. 14.3
The difficulty of adequately modeling certain external-environment objects, especially when a human operator-user actively participates in their operation, does not allow the entire simulation of test data for large-scale software systems to be concentrated and fully automated on simulation computers. Therefore, when implementing integrated problem-oriented STRs for testing the quality of operation of complex software systems, it is necessary to use analogs of real external-environment objects to form part of the data. A reasonable combination of part of the real external-environment objects and computer simulators ensures the creation of highly effective STRs with integrated models of sets of external-environment objects needed for real-time testing of software quality. Such rigs make it possible to supplement the automatic generation of tests by means of computer simulators and analogs of real equipment with real data from user operators who monitor and correct the operation of the information-processing system.
Fig. 14.4
In the diagram of a typical STR, a number of basic components can be identified, whose purpose and functions are shown in Fig. 14.4. For each experiment testing a real-time software system, it is necessary to prepare a plan of test scenarios and generalized initial data. On the simulation computer, the plan and generalized initial data are converted into specific parameter values that set the operation of each simulator or real external-environment object. This data is entered and converted on the simulation computer outside the real time scale, preparing for the start of the operating session of the rig and of the software system under test in real time. After that, the generation of test data begins.
Analogs of external-environment objects are used primarily to generate tests representing correlated logical variables that are difficult to describe and model on a computer. In addition, they make it possible to verify and validate certain software simulators of the external environment, which subsequently play the main role in testing. In a number of cases such analogs cannot reflect all the features of the external-environment objects, and computer simulators remain the sole sources of the corresponding portion of data for verifying software quality.
Data from the workstations of user operators must reflect the real characteristics of the impacts on the software system under test, taking into account the characteristics and qualification of the person who will use the tested programs in the real information-processing system. In addition to the primary initial data from the simulation computer, this part of the STR can also receive the results of processing a number of tests by the system under test. As a result, the feedback loop of manual and automated control of external-environment objects is closed through the human and his characteristics. The same closing of the automated control loop is also possible in analogs and simulators of real objects. Sometimes interaction between real user operators and the software system under test is required directly through the standard interface and display devices of the information system. In these cases, ensuring the testing requires the use of data logging for data arriving at the software system and issued by the programs to real objects. This makes it possible to monitor and process test data either on the simulation computer or on a separate computer after recording on intermediate media outside real time.
Data from field experiments with external-environment objects can be prepared in advance, outside the testing sessions of the software system, for example during debugging of the hardware part of the information-processing system. This data reflects the characteristics and dynamics of the operation of objects that are difficult or dangerous to connect for direct interaction with an insufficiently verified software system. In addition, such data can be used to validate the adequacy of simulators of certain external-environment objects. It is useful in cases where creating certain operating conditions for external-environment objects is very expensive or dangerous and can only be done in exceptional cases. However, field-experiment data cannot always be adequately described by conditions and generalized characteristics of their behavior. The presence of a number of random, uncontrolled factors complicates the picture of the initial data and can make it difficult to compare the results of field experiments with those obtained through software simulation of tests. In this case it is impossible to modify or take into account feedback from the software system under test.
In a number of cases, testing requires having reference characteristics of the data, arriving at the software system under test. When working with real objects, it is often necessary to
продолжение следует...
Часть 1 14. Integration, qualification testing and acceptance trials of software complexes
Часть 2 14.4. Evaluating the reliability and safety of complex software systems'
Comments