Lecture
Это окончание невероятной информации про интеграционное тестирование .
...
create special measurement systems that determine, register and prepare for processing all the necessary characteristics during the real operation of these objects (for example, the flight coordinates of aircraft during testing of ATC systems). Such measurements are carried out during the autonomous operation of the objects or during their real-time interaction with the software system. The measurement results are used to determine the quality characteristics of the software system when working with real objects.
Simulation of reference characteristics of external-environment objects serves to determine the quality of the software system's operation under ideal conditions — in the absence of distortions of the initial data, errors in measuring their parameters, faults and deliberate failures. Testing with such initial data makes it possible to assess the characteristics of defects and errors in the results caused by insufficient software quality. The reference data is then combined with certain calibrated distortions and defects of the initial data, which provides for the preparation of tests with dynamic and statistical characteristics as close as possible to real ones. At intermediate verification stages, reference characteristics help to separate the causes of result defects that depend on distortions of the initial data from those that depend on the quality of the information-processing programs under test.
Synchronization and consolidation of test data are intended to order tests from sources of various types according to the real time of their arrival at the software system, and to distribute them between the simulation computer and the target computer. As a result, streams of test data are formed for each real external-environment object, which are fed into the target computer in accordance with the operating logic of the information-processing system through the appropriate interface devices connecting to the simulation computer.
Repeatability of test sessions is ensured, in automatic test simulation, by fixing all initial data and by using software-based generation of pseudo-random numbers. With reliable operation of the analogues of the real objects and of the simulation computer, it is, in principle, possible to achieve almost absolute repeatability of quite lengthy experiments and test scenarios. Some non-identity of results in repeated experiments may be caused by malfunctions and partial equipment failures. It is more difficult to ensure repeatability of test scenarios in which a human operator actively participates. In this case it is necessary to record the operator's actions as a function of time, and then to replay them according to the recorded scenario. If required, the timing diagram can be observed to an accuracy of about 0.5—1 s, but the errors in the operator's actions and in the parameters entered by the operator may differ in each test scenario. As a result, repeatability of tests is achieved only statistically.
Registration and processing of test data characteristics must ensure their monitoring for conformity with the specified generalized characteristics of each external-environment object and with the initial data of the test session. Part of these characteristics is used for comparison with the results of the software system's operation when subsequently determining its quality indicators. This is how the testing process is carried out based on the final results of the software system's operation issued to external subscribers and for determining integral quality characteristics. However, for diagnosing and localizing failures, defects and errors, and also for evaluating individual quality characteristics, intermediate data on program execution on the target computer are needed. To register intermediate data, it must be possible to break the natural process of program execution at any specified statement or upon a write or read access to specific data on the target computer.
Breaking program execution for the analysis of intermediate data can be carried out by inserting special monitoring programs when preparing the software system for a specific test session. Such insertions, when few in number, hardly distort the real time scale. They are usually placed in the final part of individual modules or groups of programs and make it possible to monitor and localize the causes of failures and defects down to fairly large sections of the programs. At the control and break points of the natural program execution process, usually only statements that transfer control to a specialized group of programs for registering and promptly processing intermediate data on the target computer are placed. Further, these data are either accumulated and preliminarily processed on the target computer, or are promptly transmitted to the simulation computer for deeper processing.
Selection of test results can be based on a bottom-up strategy of program-operation monitoring, i.e., from analyzing the execution of individual program statements up to the stochastic results of the entire software system's operation in real-time dynamics. In this case, an excessive amount of data is registered, from which the minimum necessary for analysis is subsequently selected. A top-down strategy can also be used, i.e., an ordered, hierarchical extraction that first singles out generalized results of program operation, followed by refinement of the registered and analyzed results down to detailed monitoring of the execution of the noted program modules and individual statements. In this case, only the data actually needed for analysis in a specific test session are registered. In both strategies it must be possible to control the volume and type of the extracted and registered testing information depending on the goals of the tests. Data obtained and extracted in the process of software system quality testing should expediently be divided into the following groups:
data characterizing the initial test information and the output test results;
execution routes of the program components and their statements for certain fixed test data;
anomalous events, malfunctions, failures and data characterized by deviation of the test results from the reference values beyond permissible limits and restrictions;
characteristics of the use of various target-computer resources.
Registration of intermediate data usually corresponds to certain fairly complete stages of the software system's operation. Calls to the registration programs must obey a certain system of monitoring the dynamic operation of the software system, based on the initial hypothesis that some errors and defects in the programs and data may manifest themselves at any stage of testing. However, the number of calls to the registration programs and the monitoring of intermediate results, which require breaking the integrity of the functional programs' execution, should be limited, taking into account the permissible expenditure of time resources on their implementation. Since the main task of registration in real-time testing consists in detecting and localizing errors and the causes of failures down to a functional group of programs or a module, a more precise determination of the location of a defect should be shifted to static testing outside real time.
Since testing of modern large-scale information-processing systems makes it possible to obtain such a large amount of monitoring data that a sufficiently complete analysis of it presents a difficult methodological and technical task, the processing of results must be carried out hierarchically and in a differentiated manner. An excess of monitored quantities reduces the overall speed of the simulators and of the software system as a result of the time spent on monitoring and registration. This makes it difficult to analyze the quality of program operation in real time. In the transition to mass quality-testing experiments, it becomes necessary to substantially reduce the number of analyzed parameters and, where possible, to present them in generalized form. In each specific
case it is necessary to strive for a compromise between the completeness of registration of intermediate test data and the convenience of analyzing the generalized results.
Processing of the results of testing a real-time software system can be divided into two fairly autonomous parts: operational and summarizing. Operational processing of test results must be carried out according to simplified algorithms with high throughput, ensuring that the real time scale is preserved for the entire complex of programs under test. The main part of operational result processing is related to closing the feedback loop for simulating the dynamics of the controlled external-environment objects' operation. Operationally, selection of certain test results and their preliminary processing should also be carried out to significantly reduce the volume of the results kept.
In operational processing it is expedient to include the calculation of part of the integral data that make it possible to monitor the current process of information processing by the software system under test. It is desirable to single out, register and display critical parameter values or situations that threaten the reliability and safety of the software system's operation. The volume of such operationally displayed data must be reduced as much as possible while still being sufficient for analyzing critical situations that affect the quality of the software system's operation. These data must allow the specialists conducting the tests to record the conditions under which defects in program operation manifest themselves, bearing in mind that automatic registration always has gaps in the set of parameters recorded.
Generalizing processing of the accumulated test results can be carried out outside real time after completion of one or a series of tests. The main task here consists in calculating various integral characteristics of the software system's quality. In full-scale experiments with external objects, special measuring complexes are used to obtain reference data in real time. Particular difficulties may be encountered here in connection with the need to align in time the results of executing the tested programs with the data obtained from the external measuring complexes, whose information is used as the reference. This problem can be solved either by rigid synchronization of the operation of the system under test and the measuring system, or by using a common time system for both.
The registered and processed test results must be used to establish the conformity of the obtained quality characteristics with the specified requirements. When their deviation from the requirements of the customer's technical specification, specifications, or those declared in the documentation is detected, corrections to the programs must be developed to eliminate the discrepancy. For this, all stages of software system testing must be supported by a configuration-management system for program-component versions and by a database documenting the tests, test results and corrections made to the programs. Means of accumulating messages about failures, errors, change proposals, corrections made, and evaluated quality characteristics of versions are the basis for configuration management of the further development and improvement of the software complex.
An example of a complex test stand and of models of the external environment is a complex for verifying programs for air-traffic flight control and dispatching systems in air-traffic-control centers. For comprehensive debugging, testing, testing proper and certification of air-traffic control (ATC) programs, real-time simulation of all information arriving from the external environment is carried out. The sources of information for ATC centers are radar stations, flight crews aboard aircraft (AC), air-traffic controllers, and initial flight plans. As a result, it is necessary to simulate a number of heterogeneous objects, taking into account intense random effects, as well as the presence of control on the part of the ATC-center controllers and of the flight crew aboard the aircraft. Some rare error manifestations in the programs can be compensated for by the controllers monitoring the operation of the ATC center. It is very difficult to automatically simulate the reactions and actions of controllers on a computer, since these depend to a large extent on the qualification and the specific psychological characteristics of the controllers' behavior in various air-situation scenarios. Therefore, real dispatchers and their control equipment usually take part in the comprehensive debugging and testing of ATC-center software systems.
The requirements and recommendations given above for the functions and application of the external-environment simulation complex (EESC) are oriented toward the creation of large-scale software systems, their testing and their acceptance testing, mainly prior to transfer to regular operation. After acceptance by the customer or acquisition by users, in the course of operation and use of the software system, its regular testing and evaluation of current quality must be ensured. For this, the software system must include facilities that ensure:
generation of test sets or storage of tests for monitoring the operability, safety and integrity of the software system during operation and use;
operational monitoring and detection of defects in program execution and data processing when the software system is used for its intended purpose;
implementation of procedures for preliminary analysis of detected defects and prompt restoration of the computing process, programs and data (restart) after detection of anomalies in the software system's operation;
monitoring, accumulation and storage of data on defects, malfunctions and failures detected in the course of program execution and data processing.
Facilities for test generation and external-environment simulation as part of the software system are intended for the prompt preparation of initial data when checking various operating modes in the course of using the software system and when diagnosing manifested defects. The minimal set of test-generation facilities must be delivered to users for monitoring the use of working versions of the software system in real time and must be included in the delivery package of each user version. To place such monitoring and quality-control facilities, external and operational memory resources are needed, as well as additional computer performance. Deeper testing of the operation of versions and localization of errors should be carried out on the basis of the higher-level external-environment simulation complex (EESC) on the simulation computer, which is used by testing and certification specialists. Part of these simulation facilities can be applied as lower-level (user) facilities on the target computer for diagnostics and for ensuring complete repetition of the situations in which the user may detect operational defects.
An important function of test stands is their use as trainers for operator-users. Since the quality of a software system's operation can depend substantially on the characteristics of the specific person participating in information processing, it is necessary to measure these characteristics. It must also be possible to improve them to a level that ensures fulfillment of the specified requirements for the software system. Therefore, the process of training and measuring the characteristics of operators' real reaction organically forms part of software system testing, as does the use of the EESC for training and regularly preparing operator-users in the course of replicating and operating the software system. In addition, a test stand can serve as a prototype for developing trainers in serial information-processing systems.
Automated test simulation and use of the EESC can not only significantly improve the quality of the software system being developed, but also reduce the labor intensiveness of its creation. Even approximate estimates of the ratio of total costs of software simulation to the costs of preparing test data under real operation of objects show, in most cases, a high profitability of software simulators of the external environment. In the limit, the effectiveness of using external-environment simulators approaches the ratio of costs per unit of time for the operation of real objects to the costs of software simulation of test data under the same conditions.
When using software models on a computer, the reliability of test generation is determined by the following factors:
— the adequacy of the simulator to the simulated external-environment object or source of information;
— the instrumental accuracy of the facilities implementing the external-environment simulator;
— the statistical accuracy of the simulation process and the volume of test data taken into account in the statistical generalization of the test results;
— the accuracy of the discretization by the simulators of the real continuous processes in the simulated external-environment objects.
Of the greatest importance for determining the characteristics of the software system is the adequacy of the simulators, which depends on the degree to which secondary factors characterizing the operation of the real objects or sources of information are taken into account when creating their models. The accuracy of the computer models is determined above all by the algorithms on which they are based and by the completeness with which all the features of the simulated objects are accounted for in them. In addition, the quality of programming and the level of defects and errors in the simulation programs affect the adequacy of the simulation. Every element or factor of the simulated system not accounted for in the simulator must be evaluated by comparing the individual simulated data with the results of analytical studies or with data obtained on real systems, and its possible effect on the full required accuracy of the model and of the generated tests must be determined, taking into account the other components affecting the reliability of the simulation.
The listed factors, affecting the reliability of test generation interdependent in the EESC, and increasing the reliability of the simulation through one of the factors, given limited resources, as a rule leads to a decrease in reliability due to the influence of the others. Therefore an important task in creating simulation models is to achieve the greatest total reliability of the simulation and of the determination of the values of the software system's quality-operation characteristics, with a balanced influence of each of the factors. The attainable reliability of external-environment simulation, and consequently of the determination of the operating quality of the software system under test, naturally depend on the memory resources, performance and other characteristics of the computer on which the simulator is implemented. The parameters of the simulation computer affect, to the greatest degree, the statistical and instrumental accuracies attained in the course of operating the model. The adequacy of the models and the discretization accuracy depend on the complexity of the simulation algorithms and, consequently, on the cost of developing the simulators. Therefore, when creating complex test generators, it is necessary to achieve, as far as possible, an equal influence of the noted factors on the total reliability of the software system's quality evaluation during qualification testing.
Experience in developing large software systems has shown that the quality of the methods for solving problems at individual control stages can be objectively assessed only as part of the entire control chain and the simulation of the necessary information: the external situation, the characteristics of the input information taking errors into account, and the interacting and supporting systems. This is possible through the creation of simulation-modeling stands and by ensuring interaction over standard communication channels with prototype system components and their software complexes. This approach proved to be the most expedient, contributing to increased efficiency of research-and-development work for real-time software systems,
Evaluation of a software system's reliability includes measuring quantitative subcharacteristics and their attributes: completeness, defect tolerance, recoverability, and availability-readiness (see Table 11.2). It is assumed here that the contract, the technical specification, or the requirements specification fixes and has approved by the customer certain values of these attributes and their priorities. Measurements are carried out during the operation of the finished software product for comparison with the specified requirements and for evaluating the degree of conformity with these requirements specifications.
Reliability values are correlated with the correctness subcharacteristic, but it is possible to achieve high reliability of program operation with relatively low correctness by reducing the recovery time in the event of failures. In addition, the reliability of a software system can be evaluated indirectly during development from the predicted density of detection of hidden defects and errors, and also from the density of errors in the output results detected and eliminated during testing of the software complex's working operation (see Section 13.1). The degree of test coverage of the structure of the functional components and of the software system as a whole during debugging can serve as a guideline for predicting their potential reliability (see Section 13.4). The distribution of the real durations and the effectiveness of recovery under limited resources for program operation can be considered an additional component in evaluating reliability.
For direct, quantitative measurements of reliability attributes, instrumental facilities built into the operating system or into the corresponding components of the software system are needed. In the dynamics of the software system's real operation, these facilities must automatically select
and record abnormal situations, defects and distortions of the program and data computational process detected by hardware, software-and-algorithmic monitoring, or by users. Accumulating and systematizing manifestations of defects during program execution makes it possible to evaluate the main reliability indicators, helps determine the causes of faults and failures, and prepare data for improving software system reliability. Regular recording and generalization of such data helps eliminate situations that negatively affect functional suitability and other important characteristics of the software system.
Direct experimental methods for evaluating integral reliability characteristics of a software system are in a number of cases very difficult to implement under normal operating conditions for large-scale software complexes, because of the large mean-time-between-failures values (hundreds and thousands of hours) that must be achieved during development and recorded during testing. The difficulty of detecting and recording rare failures, together with the high cost of experiments during prolonged, multi-day operation of large-scale software systems, means that testing yields small samples of recorded failures and low confidence in the reliability estimates. In addition, in such experiments it is difficult to guarantee full representativeness of the input data sample, since the checks are determined by the specific conditions under which the given software system is used during testing.
During software reliability testing, what is detected first are failures — losses of operability. In most cases, however, the cause of the failure that occurred initially remains unknown. To identify the factor that caused the failure (the primary error or defect) and eliminate its cause, it is first necessary to determine which component of the information system triggered the given failure. The largest sources of failures are partial physical malfunctions or faults of computer hardware, as well as defects and errors in software. Stable hardware malfunctions are diagnosed fairly easily with the corresponding hardware tests, after which repair or replacement of specific units should follow. However, when a random failure occurs after which normal operation is automatically fully restored, in many cases it is difficult to unambiguously identify its primary source, especially for very rare failures.
To diagnose and eliminate random rare failures, a recording service must be organized that fixes, as completely as possible, the characteristics of the situations in which each failure manifested itself. Hardware faults are more or less random in nature, and a full repetition of a failure situation is unlikely. Program errors and defects are located in a specific place and regularly manifest themselves when external situations are fully repeated. On the basis of such indications and, where possible, a detailed description of the situations in which the failure occurred, hypotheses about its cause can be constructed. These hypotheses should be used, above all, for additional, intensive testing of the entire information system. If the cause of the failure is not found in the hardware, in-depth testing should be carried out on the functional component of the software system that is presumed to contain the defect that caused the failure. To improve software system reliability at a high mean time between failures, careful, systematic work is required from specialists who accumulate, record and analyze all failure situations that occur while the software complex is operating. These specialists should also record every correction made, in order to forecast the causes of possible additional sources of failure caused by defects in the corrections.
To identifying the trend of change in reliability indicators their recorded values must be linked in time to the moments of correcting the programs and data. By analyzing the correlation between reliability values and the process of program changes, it is possible to identify certain corrections that contain errors and reduce reliability. The resulting indicators make it possible to forecast the number of errors that must be corrected in order to achieve the required reliability values, depending on the duration of testing. As a result, the operating time to the next detection of an error or failure can be estimated.
During final acceptance and certification testing, in order to reliably determine software system reliability, there are organized multi-hour and multi-day runs of the operation of the software complex of programs in a real and/or simulated external environment, under conditions of wide variation in input data with an emphasis on stress situations that stimulate the manifestation of reliability threats. Such runs make it possible to measure the reliability characteristics achieved and determine the extent to which they conform to the requirements of the technical specification, as well as to establish them in the technical conditions and documentation for the software system.
If intensive testing of programs over a sufficiently long period does not lead to the detection of defects or errors, the specialists conducting the testing get a sense that further testing of the given program is pointless, and it is transferred into operation (see Section 13.1). Experimental study of the characteristics of complex software systems has made it possible to estimate the rate of defect detection at which large-scale software complexes are transferred to regular operation: 0.002—0.005 defects per day per person, i.e., testing specialists or all users taken together detect only about one error or defect every two to three months of using the software system. A defect-detection rate below 0.001 errors per day per person, i.e., less than one error per year per three to four specialists directly engaged in testing and operating the software system, can apparently serve as a benchmark of high reliability for the software system's information processing. If the programs operate continuously, these indicators correspond to a high mean time to defect or failure detection on the order of 5—10 thousand hours and an availability factor above 0.99. When this criterion is used, the calendar time of testing is usually taken into account, including the duration of direct testing both for detecting and for localizing defects, as well as the duration of program corrections and other auxiliary work needed to restore normal operation of the software system.
Forced (accelerated) tests for evaluating the reliability of software differ significantly from traditional methods of hardware testing. The main factors affecting software system reliability are the input data and its interaction with program defects and errors or computer hardware faults. Therefore, forcing of reliability tests is achieved by increasing the intensity of input data distortions and widening the range of variation of their values, as well as by specially increasing the intensity of information flows and program loading on the computer above normal.
Planning of forced tests should provide for the subsequent recalculation of the reliability attributes obtained to normal operating conditions. For this, it is necessary to evaluate the reliability of the programs under test depending on the intensity of data distortion or on the characteristics of computer overload, and also to apply methods for correctly recalculating the resulting indicators to normal operating conditions. During forced tests it is advisable to distinguish the following test modes:
complete distortion, and limiting and critical values of the key parameters of each type of external information and user actions;
limiting and critical combinations of values of the various interacting operating parameters of the software system;
extremely large and small intensities of the total flow and of each type of external information;
deliberate violation by users of certain provisions of the instructions and recommendations in the operating documentation.
Testing and monitoring the results of the operation of the same software systems, with an increase in the number of instances under test and normal input data, can be regarded as a kind of forced testing — beta testing (see Section 14.2). At the stage of trial operation by users of some preliminary batch of the software system, a natural expansion of input-data variants occurs, if they are mutually independent. This increases the test sets and thereby makes it possible to estimate mean time between failures in the hundreds and thousands of hours. They make it possible to detect and eliminate a significant number of defects in a relatively short calendar time and thereby bring reliability up to the required level. However, it should be taken into account that the total labor intensity of such tests increases proportionally.
A special kind of forced testing is purposeful testing of the effectiveness of the tools for online monitoring and recovery of programs, data and the computing process, for evaluating recoverability. In such tests the main task consists of evaluating the quality of operation of the automatic reliability-enhancement tools and measuring the recoverability characteristics. For this purpose, the planned operating conditions of the programs are simulated, conditions under which the operation of the software restart tools and of online, automatic recovery of operability is stimulated to the greatest extent.
Particular note should be taken of the difficulties of achieving and evaluating program reliability characterized by a mean time between failures of ~100 hours. At such a reliability level, the difficulty of detecting the failures that occur and diagnosing their causes rises sharply. A mean time between failures in the thousands of hours has in a number of cases been achieved only through the operation and maintenance of complex software systems over several years. When especially high reliability of software system operation is required, the total resource costs of achieving and evaluating it may increase by an order of magnitude. However, the increase in costs required to obtain such high software system reliability during development is difficult to provide for in practice. Therefore, various methods of software protection against program faults and failures (online restart methods) are actively used to achieve it. They make it possible to slow the growth in development resource costs as reliability requirements are raised.
Evaluation of resource efficiency consists of measuring the quantitative sub-characteristics and their attributes: the time efficiency and resource utilization of the computer by the software complex (see Table 11.2). It is assumed here that the contract, the technical specification and the requirements specification have recorded and approved the required attribute values and their priorities. In the ISO 9126:2 standard, this software system quality characteristic is recommended to be reflected by a dozen attributes, each of which is to be evaluated for average and worst-case scenarios of the software complex's operation. Table 11.3 retains only three attributes, the most important for functional suitability, which are most amenable to quantitative measurement. Evaluation of these attributes can be carried out while the finished software product is operating, or by calculation methods during development, in order to compare against the specified requirements and assess the degree of conformity to those requirements.
To measure the time-efficiency attributes, instrumentation tools built into the operating system or into the corresponding software system are required. In the dynamics of real program operation, these tools must record: the load on the computing system; the values of data-flow intensity from external subscribers; the execution duration of tasks; the characteristics of input/output device operation; the waiting time for results (response) to user tasks; and the filling of the exchange memory with external subscribers under various modes of using the software complex. The values of these characteristics depend not only on the properties and functions of the software system, but also on the features of the computer's architecture and operating system. Regular recording and generalization of such data make it possible to identify situations that negatively affect functional suitability, reliability and other constructive quality characteristics of the software system.
The need for memory resources and computer performance in the process of solving tasks can vary significantly depending on their properties, as well as on the flow, composition and volume of the input data. The degree of use of computer memory and performance within certain limits does not affect the quality of solving the software complex's functional tasks. When the intensity of input data arrival is excessively high, the time balance may be disrupted between the duration of solving the full set of the software system's tasks in real time and the computer's performance in solving those tasks — load testing. A disruption of the balance between the memory available in the computer and the memory needed to store all the incoming and processed information is also possible. To identify such situations and determine the characteristics of the software system under conditions of insufficient computer resources, tests are carried out at a high but permissible intensity of input data arrival.
The most complex task is evaluating the efficiency of computer performance resource use in real time. Here, the dependence of task-solving quality on the intensity of incoming information of various types must be determined. The main task of the tests consists of determining the probabilities with which the correspondence between the performance needs for solving the entire required set of tasks and the real capabilities of the computer and other information-system components will be violated. If this probability is small and an occasional decrease in quality due to resulting delays and omissions in the processing of messages or tasks can be considered acceptable, then a conclusion is drawn that the computer's performance corresponds to the functions of the given software system.
To evaluate the use of performance resources, the following must be measured:
real values of the intensities of incoming input data and requests to call functional programs, as well as the probability distributions of these intensities for various sources and types of requests;
the durations of the stand-alone solution of each functional task individually, processing input data or invoked by external requests, as well as periodically;
computer load under the normal mode of message and task arrival, as well as the probability of overload by tasks of various types and the distribution of overload durations under real conditions;
the effect of omissions in the processing of tasks or messages of each type, and of a decrease in the rate of solving certain tasks, on functional suitability and other quality characteristics of the software system.
The listed tasks can be solved experimentally in the course of testing the completed system, but in doing so there is a great risk that the computer's performance will prove insufficient for solving the given set of tasks in real time, which will affect the quality of the software system's use. Moreover, testing or trial-operation conditions do not always correspond to the modes of the system's mass application. Therefore, when evaluating, special measures must be taken to create real, and also controlled, conditions of software system and external-environment operation that are the most heavily loaded. Such critical situations can largely be prevented during software system development by calculating the execution durations of modules from the program text, and combining these characteristics in accordance with the structure of the software components and the entire software complex.
For a correct evaluation of the limiting throughput capacity
of the system with the given software, it is necessary to measure the following characteristics of the functional groups of programs:
— the extreme values of their execution durations and the routes on which these values are reached;
— the mean value of the execution duration of each functional group of programs over the entire possible set of software system routes, and its variance;
— the probability distribution and values of the execution duration of the functional groups of programs.
In general, to evaluate execution durations and determine the quality of program operation as a function of load, it is necessary to specify the probability of each combination of test data and measure the duration corresponding to it. After ordering the duration values, a probability distribution as a function of execution duration can be obtained. However, for complex groups of programs it is very difficult to determine the probability of each combination of input data. Therefore, in practice, in a number of cases one is limited to certain average or most probable test-data values, as well as one or several combinations of input data at which limiting values of task flows and program execution durations are expected that could negatively affect the quality of the software system's operation.
The effect of such situations of computer overload by performance can be mitigated by applying priority disciplines for the online dispatching of task execution for solving functional tasks. Depending on the characteristics of the task flows and the expected durations of their execution, priorities for their solution can be assigned, thereby increasing the efficiency of using the limited performance of the computing system for a given software complex. The rapid growth in the number of tasks being solved, their complexity, and the required performance of computing resources has stimulated the search for ways to meet customers' needs for resources to solve such tasks. Considerable attention has been paid to analyzing the effectiveness of dispatching disciplines with relative and absolute priorities. These disciplines were act
ively used in organizing computations in specialized, object-oriented real-time computers. They made it possible to increase the equivalent computer performance by 10—20% compared with priority-free dispatching disciplines. It has been shown that in many cases it is advisable to use no more than 10—15 priority levels when dispatching functional tasks, at a computer load of 80—95% and with a significant difference in the durations and importance coefficients (10—100) of the priority tasks. For the practical use of the characteristics and methods for calculating a rational distribution of computing-system performance, methods and standard models have been created that make it possible to analyze and optimize dispatching in specific systems. Under resource constraints arising from requirements to minimize the weight and dimensions of specialized, object-oriented computers in aviation, rocket and space systems, their economical use remains relevant. Moreover, in some cases it is useful to assign high priorities to especially important or short tasks, for example, for exchange with external subscribers.
Based on the test results, the listed tasks of evaluating the software system's resource efficiency can be solved, which makes it possible to analyze the factors that determine the required computer throughput and to develop measures to bring it into line with the needs. If, beforehand, during the design process, computer performance was not evaluated or was determined too roughly, there is a great risk that the rework will be extensive or that it may be necessary to replace the computer with a faster one. This is usually caused by the developers' «optimism», which leads to underestimated intuitive estimates of task-solving durations and possible limiting intensities of information flows. Long-term recording and accumulation of resource-efficiency values help identify situations in which certain functional-suitability defects in the software system manifest themselves.
The reliability of evaluating a computer's throughput with a specific software system depends on the correctness of the modeling of external message flows, as well as on the distributions used for program execution duration. To evaluate resource efficiency when preparing the technical specification and requirements specification for the software system, the model and characteristics of the external environment should be agreed with the customer, in which
the software complex will be used, as well as the dynamics of data reception and transmission (see Section 14.3). These conditions should be detailed to a level that allows the required values of the intensity of task solving to be unambiguously determined:
in the average, normal operating mode of the software system with the highest quality of functional suitability;
in the mode of limiting load, occurring with a certain probability, and with an acceptable decrease in functional suitability and certain constructive quality characteristics;
in the mode of short-term, emergency overload, capable of critically affecting the functional suitability, reliability and safety of the software system's use.
To determine the use of time resources of the computer by software complexes, it is useful to apply the recommendations of the ISO 14756 standard — Measurement and rating of performance of computer-based software systems. The standard is oriented toward evaluating: application software, operating systems, and computing complexes that include all hardware and software. The main recommendations are concentrated in two large sections and four normative annexes. Section 2 contains a general description of the measurement methods, while Section 3 provides detailed procedures for measuring and evaluating the performance of the software system as part of the information system. The description of the performance-measurement method begins with emulation — simulation of users and data flows from the external environment: their random characteristics and processes; terminal operation; and the setting of workload parameters for users and computing resources. Measurement procedures must contain recommendations for:
forming tests;
distributing them across time phases;
determining and recording test results;
monitoring the correctness of external-environment emulation;
statistical processing of measurements.
Evaluation of the performance value is recommended for determining the load on operator-users, the throughput of the software system in terms of the number of tasks per unit time, and the time scale of events in task and data processing. These results are proposed to be compared against the requirements of the customer and users in order to evaluate workloads and the sufficiency of the software system's performance in a specific external environment. The detailed measurement and evaluation procedures are distributed across six subsections: initial requirements; measurement processes; resulting data; verification of the correctness of the results; performance calculations; and evaluation of the reliability of the performance measurements. The standard is recommended for use by: testers; developers and purchasers of the software system; and also system integrators of complex computing systems.
Часть 1 14. Integration, qualification testing and acceptance trials of software complexes
Часть 2 14.4. Evaluating the reliability and safety of complex software systems'
Comments