4. Determination of real reliability of software operation

4.1. Experimental methods for determining the reliability of complex software

There are direct and forced experimental methods for determining the reliability of complex software tools. Direct experimental methods for determining the reliability of the PS used in normal conditions of operation. They require large values ​​of time to failure, so their cost is high. In addition, with such experimental methods, it is difficult to guarantee the complete representativeness of the sample of initial data, since Checks are determined by the specific conditions of use of this PS under test.

To diagnose and eliminate accidental and rare failures, a service should be organized for registering all failures with full recording of situations in which a failure occurred. If the cause of the failure is not hardware, then you need to conduct in-depth testing of the functional component, which may contain a defect that caused the failure.

During the final acceptance and certification tests, for a fairly reliable determination of the reliability of substation, many hours and many-day runs of the program complex in real or simulated external environment are organized in the conditions of wide variation of the initial data. Such runs allow you to measure and record the achieved quality indicators and the degree of their compliance with the requirements of the technical specifications. If for a sufficiently long period of time no defects or errors are detected, the MS is transferred to service.

Forced tests of PS reliability are carried out by increasing the intensity of distorting the source data and expanding the variation of their values, as well as by increasing the information flow and loading the PS above normal. For forced testing, the following test modes are used:

with complete distortion of the key parameters of each type of external information;

with the limiting and critical values ​​of the parameters of each type of information;

with limiting and critical combinations of the values ​​of various interacting parameters;

at extremely large and small intensities of the total flow of external information;

at extremely high intensities and the absence of each type of external information.

As a kind of forced testing, it is possible to consider testing and monitoring the results of the operation of the same PS with an increase in the number of test specimens and normal source data. This allows, depending on the number of copies of the PS, to extend test suites in the appropriate number of times and makes it possible to evaluate the time between failures for hundreds and thousands of hours. The reliability of complex critical PSs is significantly reflected in memory and performance overloads during program execution. In order to identify failures due to these reasons, during the accelerated tests, testing is carried out at a high intensity of receipt of the initial data.

4.2. Organization and stages of testing for testing the reliability of complex software

4.2.1. Aims and stages of testing the reliability of program complexes

The main stages of testing and testing the program complex and its components are presented in Figure 4.1.

For each stage, the figure presents the basic initial data and results of testing and testing. The first two stages relate to testing and debugging the correctness of components in statics, i.e. with fixed real-time values, and described above (p.2.5).

Testing and testing of the software as a whole consists of the following steps:

real-time testing according to the modeling bench or test generators that simulate individual objects of the external environment;

real-time testing with simulators of individual objects of the external environment and real impacts from operators — users;

testing in fully adequate real or simulated external environment and with real impacts from user operators.

Figure 4.1 - The stages of testing complex software packages

At all stages, except for direct verification of the functioning of the programs, two more works are performed:

work on the methodological support of testing and the creation of environments in the automatic generation of tests;

processing of test results and evaluation of achieved program quality indicators

Each of the stages of testing ensures the creation of a specific intermediate product and must have a document confirming the achieved quality characteristics of the program.

4.2.2. Organization of final tests of the program complex

The final tests of the program complexes are carried out in two ways, depending on the availability of customers.

If the complex of programs was developed for a specific customer, then joint acceptance tests are carried out, in which the customer and the developer take part. A commission is created that checks the fulfillment of the requirements of the technical specification and the compliance of the software with the documentation submitted. In the final tests, in addition to functional tests, it is necessary to conduct testing in the modes of limiting the use of resources to verify the reliability of the PS.

Testing of commercial software packages created at the initiative of the developer in the absence of a specific customer is carried out in two stages, which are called A lfa - and beta testing . Tests are carried out to meet the criteria defined by the project manager. They consist in normal and accelerated trial operation by the end users of the software product, in accordance with the accompanying documentation.

For alpha testing, end users are involved who work for the same company but are not directly involved in the development of this project. For beta testing, voluntary users are involved who receive a free version of the software for trial operation. These users are obliged to inform developers of any defects and errors found and to replace versions according to the instructions of the developers.

After a successful beta test, the decision is made to transfer the software product for sale to a wide range of users. A summary of beta test results can be used as part of a certification test.

With alpha and beta testing, it is customary to separate progressive and regressive testing. Progressive means testing new software components to identify defects and errors in source programs and specifications.

Regression testing is intended to control the quality of changes in post-adjustment adjustment programs. Its necessity is determined by the fact that a significant proportion of changes after alpha and beta testing, in turn, contain errors. The volume of tests and the duration of both stages of testing are determined by the project manager depending on the complexity of the program complex and the intensity of the flow of changes.

4.3. Test generation to determine the reliability of complex software

Manual test preparation is expedient when a few values ​​have deep logical connections that are difficult to formalize and present analytically. At the same time, manual preparation of tests is easier than developing a test generator. But such cases are rare, and in most cases it is necessary to automate the generation of tests.

Generators of software tests should provide the ability to create tests in all the variety of behavior dynamics and characteristics of objects of the external environment of this software. In this case, usually only a small part of the tests is deterministic, and the main part reflects the stochastic and dynamic behavior of external objects.

When creating test generators, two fundamentally different approaches are used: integral and differential.

With an integral approach, the basis is a formal description of the input and output information of the object being simulated, as well as the functional connection between the data at its input and output. At the same time, the structure of the object and the processes that are realized during the actual functioning of its components are not modeled and do not matter.

In the differential approach, detailed information is required on all the processes of functioning of the components of the object of simulation. Differential or simulation models of test generators are based on this information.

In contrast to the natural experiment, the simulation of the external environment and tests on a computer has more possibilities to control both the initial data and all intermediate and output results of the functioning of the object under study. Software simulation of the external environment allows you to:

conduct continuous continuous generation of simulated data to determine the reliability of the PS, in a wide range of conditions and parameters, which is often impossible for real objects;

expand the range of characteristics of the simulated objects beyond the real or available data sources and generate information flows reflecting the perspective characteristics of the created systems and objects of the external environment;

create test data corresponding to critical or dangerous situations of the functioning of environmental objects, which are impossible or risky to accomplish in field experiments;

to ensure high repeatability of the simulated data under given conditions of their generation and the possibility of stopping or suspending the simulation at any phases of the simulation environment.

Some even unreliable situations of the values ​​of the initial data are critical and especially important for the functioning of the entire system for which the software is developed. Selecting and simulating such situations allows you to test PS in critical situations that are impossible or dangerous to create on real objects.

However, it is not always when a software simulation correctly and fully reflects all the characteristics of the external environment and it may be necessary to conduct separate control checks of the MS in interaction with real objects.

4.4. Registration and processing of software reliability test results

Registration of test results should record for analysis not only the output of the programs, but also the process of implementing their execution. When localizing defects and errors, it is often necessary to be able to control the process and the results of the execution of individual operators in the program in the software. To do this, it should be possible to register any intermediate data and their connection with the initial tests. At the same time, it is necessary to allocate data that are most useful for assessing the quality and reliability of PS.

The data obtained and released in the process of testing the reliability of PS can be divided into the following groups:

data characterizing the initial test information and the test results,

routes of execution of software components and their operators with some fixed test data;

abnormal events, failures, failures and data, characterized by the deviation of test results from standards for acceptable limits and limitations;

characteristics of the use of various computer resources.

To obtain this data when executing programs requires memory resources and computer performance. Depending on the volume and complexity of the software and resources of the computer on which it operates, memory and performance are allocated for the implementation of the means of recording, processing and summarizing the test results. The relative share of resource costs for these funds is usually 5-10% of the total resources for the implementation of the PS.

Processing of test results should be hierarchical and differentiated.

The registered and processed results of reliability tests shall be used to establish the compliance of the obtained characteristics with the specified requirements. For the accumulation of messages about failures, errors, suggestions for changes, made adjustments and characteristics of versions, a test database is created. This database should collect information that consists of the following main parts:

data on failures, defects and errors, conditions of their manifestation and characteristics of detecting tests, as well as suggestions for changing the programs to be analyzed to highlight those for which program adjustments will be developed - a journal of proposed changes ,

developed changes of programs for making adjustments in the next version of PS - a journal of approved adjustments ,

characteristics of basic versions and a set of changes made in each of them - a log of characteristics of basic versions of PS,

the characteristics and parameters of the users to whom the corresponding versions of the software were transferred for use and the features of the operating environment they have - the log of parameters of the user versions of the software

The accumulated data on changes and the history of adjustments are to be stored throughout the entire life cycle of the PS. The destruction of information about completed or proposed changes can lead to high costs for their restoration. Therefore, the database of changes should be duplicated and maintained by methods and means of maintenance.

4.5. Testing methodology when testing the reliability of complex software

The purpose of the methodology is to organize and regulate the testing and testing of complex complexes of application programs and their components in order to achieve compliance of the characteristics created by software with technical requirements and specifications agreed between the customer and the developer. The methodology is focused on testing large-scale PS, operating in real time and consisting of many software components. The stages of testing and testing the reliability of the software package are presented in Figure 4.1. The method of testing components at fixed time values ​​is described above in Section 2.5. This method presents four final stages of testing in real time, aimed at achieving high quality and reliability of a complex of programs consisting of several functional components.

The method involves the following stages of testing complex software:

1. Testing and debugging software components in real time. At this stage, you need to perform 5 types of work:

development of tools to simulate the external environment in real time;

development of tools for processing test results, testing and assessing the quality and reliability of a functional component in real time;

preparation of tests for testing functional components in real time;

completion of testing the reliability of the functional components in real time;

processing of test results and assessment of the quality and reliability of the functional component in real time.

2. Testing and testing of a complex of programs according to the imitators of the external environment. At this stage, there are 4 types of work:

2 .1. integration of all functional components in the full version of PS;

2.2. preparation of tests for PS in real time according to simulators;

2.3. completion of testing and testing of PS in real time according to simulators;

2.4. processing of test results and assessment of the achieved quality and reliability of the PS in real time according to simulators.

3. Testing and testing the reliability of the software package under the effects of user operators. The stage involves the implementation of 5 types of work:

3.1. development of a simulator and semi-natural environment simulator for testing PS with the participation of operators - users;

3.2. preparation of tests that simulate the external environment and scenarios of impacts from operators - users on the functioning of the PS in real time;

3.3. completion of testing of the PS version in real time according to the data of the external environment simulator in the presence of influences from operators-users

3.4. testing software packages focused on identifying certain types of defects and in extreme situations;

3.5. processing of test results in real time and assessment of the quality and reliability of the PS operation in interaction with operators-users and in extreme situations.

4. Testing of a complex of programs in a real external environment. At this stage, there are 7 types of work:

4.1. development of methods for safe coupling of a complex of programs with a real external environment;

4.2. integration of the tested software with a real external environment;

4.3. preparation and certification of measuring parameters of data from environmental objects entering the program complex;

4.4. completion of tests of a complex of programs in real time and in a real external environment;

4.5. processing of test results and assessment of the achieved quality and reliability of a complex of programs in real time and in a real external environment;

4.6. adjustment of technological and operational documentation for the version of the software according to the results of all stages of testing for presentation to the customer;

4.7. execution of the certificate of completion of the developers' tests and presentation of the version of the software complex to the customer for acceptance testing or certification.