Lecture
2.1.1. The principles of building components to ensure the reliability of the software
The reliability of complex software is primarily determined by the quality of their components - modules and functional groups of programs. Their high quality is achieved by systematic testing and debugging. The relative simplicity and visibility of most components allows the use of sufficiently formalized methods and techniques for automated planning and effective implementation of testing, which this section is devoted to.
For modern software is characterized by the implementation of the principles of modular construction of applications and complexes. In the structure of modern PS, subsystems are identified that implement certain groups of functional tasks, components that are formed by decomposing subsystems. Modules are, if possible, unified between different subsystems. Unified interfaces and modules interaction protocols allow solving the tasks of PS development by replacing individual modules without changing other parts of the system.
In accordance with the ideology of open systems, software components must meet two important requirements: portability and the possibility of joint work together with other remote components.
The task is reduced to the maximum possible reuse of developed and debugged software components, when changing hardware platforms, operating systems and interaction processes. A well-thought-out and streamlined PS structure directly affects the achieved reliability of operation and the safety of their use, as well as the complexity of their development.
When developing the structure of software components and software systems in general, it is necessary to formulate the criteria for its formation. Depending on the characteristics of the problem area, the criteria for selecting a structure may be as follows:
reliability of operation and safety of use;
efficient use of memory and computer performance;
the complexity or duration of development;
modifiability of PS and ensuring the possibility of changing the composition and functions of components while preserving the principles of structural design and quality of basic versions of PS.
In accordance with the goals and objectives of the design, it is necessary to select the dominant criteria at the stage of system analysis and formation of the PS structure.
When choosing methods and technologies for developing software components, their dimensionality is of fundamental importance. Methods and technologies for testing programs also depend on their size. Consider testing methods for individual software modules or their small groups that solve a fairly simple functional task (component). These test objects have a number of basic features that distinguish them from complex software tools and affect the methods and tools used for testing and debugging.
The small dimension of the program modules ensures their visibility and the possibility of a detailed analysis of the functions, structure and process of solving problems. Detailed control can be carried out with an accuracy to each operator in the program. The control has access to all the logic for solving the problem and all possible routes for program execution.
Another feature of testing and debugging software modules is the need and the possibility of ensuring their high quality and suitability for use in different software systems. Therefore, completing the testing and debugging of software modules their certification with the application of the characteristics of the completeness of testing and the range of test data.
The third significant feature of testing software modules is the proximity of secondary errors to their causes — primary errors. This facilitates the diagnosis and localization of primary errors, which relate mainly to program and algorithmic types.
There are several methods for testing and debugging software modules that are focused on detecting errors of certain types.
Methods of testing control flows are designed to detect errors in the structure of software modules and information processing routes.
Methods of testing data streams reveal errors in the computational part of programs and information conversion processes. This orientation allows you to streamline the sequence of application methods. Each method provides the detection of not only secondary errors, but also the localization of certain types of primary errors.
Methods of debugging software modules differ significantly from the methods of complex debugging of complex software, in which priority is given to verifying the reliability of the programs and various combinations of data from the external environment.
There are two main strategies for testing software modules (components):
structural testing (white box testing, program-driven testing, control flow testing.),
functional testing (black box testing, data flow testing).
Structural testing is the initial stage, because with an incorrect structure, the most gross distortions of the results and even the absence of some results are possible. Testing of control flows consists in checking the correctness of data transmission sequences and the formation of program execution routes with different source data.
With such a strategy, the basis is the structure of the program module, built in the text of the program as a graph. The graph highlights and organizes the program execution routes. Test kits are prepared, each of which must be realized along a specific route. The deviation of the test from the selected route is considered as an error. After eliminating the errors of the executed and selected routes for each of them, the data processing process is checked and errors in the results of their conversion are detected. Testing is completed with full coverage of the program graph, or with full use of the resources allocated for testing (Fig. 2.1, a).
a b
Figure 2.1. - a) structural testing scheme,
b) functional testing scheme
Structural design has several advantages when testing logic programs with a small amount of computation.
Functional testing. When using this strategy, the program is considered as a black box, i.e. its internal structure is unknown. Such testing is intended to clarify the circumstances in which the behavior of the program does not meet its specifications. The execution of each function of the program is checked over the entire domain of definition (Fig. 2.1, b).
This strategy has advantages with a relatively simple program structure and the predominance of computing operators in it.
It is almost always advisable to use both strategies. Programs with a complex logical structure and a small computational part are better tested first by the first strategy, and for routes with computational operators, data flow analysis is used. Programs that have a simple structure and contain volumetric calculations, first use the second strategy, and testing on the first is completed.
Both strategies can use deterministic and stochastic testing methods.
In deterministic testing, specific sets of input data are given, which correspond to specific values of reference results. The test results at the output or at the intermediate points are compared with the reference points - this allows detecting and localizing the error.
When using stochastic testing methods, source data is used in the form of sets of random variables described by their distributions and basic distribution parameters. Such distributions of the source data should correspond to the reference distributions of the results, which together form the test. The results obtained during the testing process are summarized and presented by the distributions of their values, which are compared with the reference ones. Individual specific test results are not monitored. Evaluation of the quality of the functioning of the programs and the results of testing is made according to the degree of compliance of the distributions obtained and their parameters with the reference distributions.
When testing software components can distinguish the following steps:
testing of formalized requirements specifications for software and information modules, software groups and software systems;
testing of software modules coded and prepared for testing at the source level of programs and at the level of object inputs;
testing groups of software modules, i.e., components that solve complete functional tasks.
Let us consider in more detail these stages and the tasks solved on them.
1. The main purpose of testing specifications is to verify the completeness and mutual correspondence of the functions prescribed to the program or information components of different levels. In addition, the correspondence of the descriptions of information at the inputs and outputs of the interacting software modules and program groups, as well as the correspondence with the descriptions of information modules in the database, is checked. Such testing is advisable to carry out a descending method, starting with the specification of the complex or groups of programs.
Testing the consistency of interfaces in the software component specifications is used to detect errors in variable descriptions and control transfers during module interactions. According to the block diagram of the program complex, a list of successively called modules is established and the correctness of call descriptions in the program specifications is checked. For each input variable, its name and descriptions in the specifications of this module are compared to its descriptions in the specifications of those modules where the same variable is used.
2. The main task of testing software modules is to verify the correctness of processing by the modules of incoming information and the resulting output data in accordance with the functions presented in the specifications. The correctness of the structure of the modules and the applied structural components: cycles, blocks, switches, etc. should be checked. Verification subject to information processing routes in each module and the correctness of their implementation, depending on the source data. The completeness of the test is determined by the degree of coverage of possible routes of the program. The second stage performs several tasks.
2.1. Testing of calculations and data transformations by program modules serves to detect errors in the computational part of the program. Benchmarks are the results of preliminary calculations using the formulas laid down in the module. Testing of software modules should be carried out on ordered data sets, taking into account the degree of their influence on the output results. From this position, there are 2 types of data processing:
able to completely change the scope of results
modifying results within a certain limited domain.
The first type of processing corresponds to the initial data at critical points and at the boundaries of the regions of change of variables. With such critical values, the route of the program execution may change, as a result of which the greatest change in the results is possible. Therefore, data processing testing should first of all be aimed at checking the execution of programs with the values of variables affecting the choice of route and the logic of the functioning of programs.
The second type of processing corresponds to data in a bounded or unlimited domain of definition, which can be divided into some set of mating regions. Changing the data inside such an area does not affect the program execution route. Therefore, to test the functioning of the program from the entire set of values, it is sufficient to use when testing only a few values inside and near the borders of the area. In the process of testing, the accuracy of the calculations performed, the correctness of the scaling and dimension of the quantities being processed, the correctness of the formation of logical quantities, etc. are checked.
2.2. Testing the completeness of the functions performed by the module is designed to detect errors in the developed program regarding the functions specified by the program specification. Testing is subject to the implementation of all functions defined by the specification, and the characteristics of the intermodular interface. When checking the correctness of the function, the results of the structure preceding the testing and calculation in the module are used. Special tests are created to check unsupervised functions.
3. Testing of software components is intended to verify the correctness of the solution of rather large autonomous functional tasks. At this stage, the correctness of the control and information links between the groups of modules is checked, as well as the correctness of the functions performed during the processing of information in the group of programs. This significantly increases the complexity of the tested objects, as well as the volume and complexity of the tests. At this stage, several tasks are performed.
3.1. Testing the structure of a group of programs is used to detect errors of conformity of the actual structural structure of the group of its specification. Verifies the correctness of calls to software modules and control returns when they interact in the test group. The technology of testing the structure of groups of programs is largely similar to the testing of modules.
3.2. Testing the intermodule interface in a group of programs is designed to detect errors of informational links between the modules in the group. Each variable and array of variables should be checked for the identity of the descriptions in the texts of the programs of the interacting modules, as well as for compliance with the original program specifications.
3.3. Testing the implementation of restrictions on the use of memory and the permissible duration of the execution of a group of programs is designed to detect errors in the use of real computer resources by the developed functional group of programs. The most difficult is to test the use of computer performance resources in real time.
3.4. Testing the completeness of solving a functional task by a group of programs serves to finalize the testing of a group of programs and detect errors that remain after previous types of testing. This type of testing is carried out in accordance with the specification of requirements for a group of programs. Of particular importance are tests for testing programs at specific points of the values of the original parameters and under critical combinations of conditions in the data.
3.5. Testing the completeness of the correctness of the documentation is intended to detect errors of compliance with the real program of its accompanying design and operational documentation. This testing is final and is intended to record the quality of official documentation on software components. In addition, the compliance of the documentation with the standards for the design and execution of programs is checked.
The purpose of testing the structure of software modules is to check the correctness of the selected routes of program execution and to detect mainly logical errors in the formation of routes. When testing the structure of programs, it is necessary to have information about the complete set of real execution routes in each program. This allows you to control the degree of checking routes in an orderly manner and to prevent individual routes from being accidentally missed. It is convenient to carry out the selection of tested routes using graph models of programs.
When planning the testing of the program structure, two tasks arise first of all:
- formation of criteria for allocating routes for testing;
- the choice of strategies to streamline the selected routes.
As criteria for the allocation of routes most often used the following criteria:
the graph coverage of the program minimizes the number of routes covering each arc of the graph at least once,
the allocation of linearly independent routes based on the concept of cyclomatic number, differing by at least one arc,
allocation of routes for all possible combinations of arcs included in the routes.
In real programs, part of the routes may not be feasible due to inconsistencies in the condition. Cycles in programs lead to a significant increase in the number of routes. You can plan testing according to one of the criteria or use successively stricter route allocation criteria.
Route ordering strategies should take into account the complexity of the routes and tests to test them. First you need to test the main group of routes within the resources allocated for testing. If there are restrictions, some of the routes may not be verified.
The streamlining of routes when planning testing is based on the use of the following program characteristics:
the number of lines of text in the selected route (1 strategy),
the number of conditional jumps determining the formation of each route (2 strategy),
probabilities of the execution of routes in the real operation of the program (3 strategy).
In the first strategy, the routes that are the longest in the number of lines and text and in execution time are tested first. This strategy is most appropriate when testing programs that have computational nature of data processing under a small number of logical conditions.
In the second strategy, priority is given to routes that are most complex in terms of the number of branch conditions being analyzed. Used when testing logic programs with a small amount of computation.
When arranging routes according to the third strategy, the main difficulty lies in estimating the branching probabilities in conditional jumps and switches, as well as in estimating the number of cycle executions. These values should be indicated by the program developers. This strategy allows you to more fully assess the level of verification programs.
The complexity of the program is determined by the number of interacting components, the number of links between the components and the complexity of their interaction. During the operation of a program, the diversity of its behavior and the diversity of connections between its input and resulting data is largely determined by the set of routes for program execution. The complexity of the program module depends not so much on the size (number of lines) of the program, but on the number of individual routes for its execution. Experimentally confirmed the adequacy of the use of the structural complexity of programs to assess the complexity of testing, as well as the likelihood of undetected errors and the cost of developing software modules in general. The complexity of testing software modules can be estimated by the number of routes Mk needed to test them, or more fully by the total number of conditions Ek,which must be specified in tests for passing all routes of the k-th program:
,
where Ei is the number of predicate conditions defining the i-th route.
Consider an example of assessing the complexity of testing programs for various criteria for allocating routes. In fig.2.2 presents an example of the initial graph of the module of the program, containing 14 vertices, 20 arcs and 3 cycles. Such a program contains about 30-50 operators in a high-level language and can be considered as fairly typical.
Figure 2.2 - Example of software module graph
For a complete verification of the module by the first criterion, four routes are enough (E is the number of predicate vertices on the route):
1-2-14 (E1 = 1),
1-3-4-6-8-6-8-14 (E2 = 5),
1-3-5-7-9-10-11-12-14 (E3 = 5),
1-3-4-5-7-9-10-9-10-11-7-9-10-11-12-13-14 (E4 = 9).
По этому критерию гарантируется проверка всех передач управления между операторами программы и каждого оператора не менее одного раза. Самый длинный по количеству вершин маршрут не охватывает только 3 вершины из 14 и только 6 дуг из 20. После проверки еще двух маршрутов вне контроля остается одна вершина и две дуги. Однако при этом критерии не учитывается комбинаторика сочетания условий на разных участках маршрутов, например при сочетаниях ветвлений в вершинах 3 и 12. Сложность программы при выделении маршрутов по этому критерию характеризуется числом маршрутов равным четырем и сложностью тестирования равной 20. Эта величина характеризует суммарное количество условий, которое необходимо задать в тестах для полной проверки всех маршрутов, выделенных по первому критерию.
The second criterion for choosing routes to assess the complexity of testing the structure uses the definition of the cyclomatic number of the original graph of the program being checked.
Cyclomatic number (cyclomatic complexity) is a software metric that quantifies the logical complexity of a program. It determines the number of independent paths in the basic set of the program and the upper estimate of the number of tests, which guarantees the one-time execution of all program statements.
Cyclomatic number is calculated in one of three ways:
The cyclomatic number is equal to the number of regions of the flow graph;
According to the formula V = E - N + 2,
where V is cyclomatic complexity,
E - the number of arcs of the graph,
N is the number of nodes in the graph.
3. According to the formula V = p + 1,
where p is the number of predicate nodes of the graph.
By calculating the cyclomatic complexity, you can determine the number of independent paths that should be sought in the graph.
For our example, V = 8.
This criterion provides the initial program graph with a single check of each linearly independent acyclic route and each linearly independent cycle, which together form the basic routes. Each linearly independent route or cycle differs from all others by at least one vertex or arc. For our example, the set of routes tested by this criterion is formed from three linearly independent cycles:
6-8-6;
9-10-9;
7-9-10-11-7
and five linearly independent acyclic routes:
1-2-14;
1-3-4-6-8-14;
1-3-5-7-9-10-11-12-14;
1-3-4-5-7-9-10-11-12-13-14;
1-3-4-5-7-9-10-11-12-14.
In this case, the total complexity of the tests, taking into account all the conditions for passing routes once, becomes equal to 25.
The third criterion for checking and determining the complexity of testing the program structure includes the requirements of a single check for not only linearly independent, but all linearly dependent cycles and acyclic routes. It consists in the analysis at least once of each of the real acyclic routes of the original graph of the program and of each cycle reachable from all these routes. For our example, according to this criterion, it is necessary to execute 6 acyclic and 5 routes, from which elementary cycles can be reached. To implement the selected routes in 11 tests, it is necessary to set 66 conditions in total. At the same time, the feature of the last four routes with cycles, as well as the corresponding acyclic routes, is a complete enumeration of combinations of branches in 3 and 12 vertices.In real programs, some routes may not be feasible due to incompatibility of conditions, which are sequentially analyzed at different vertices.
При оценках числа маршрутов в ациклических программных модулях целесообразно пользуется критерием цикломатического числа. Для оценки суммарной сложности тестов, необходимых для полной проверки ациклических программных модулей можно использовать величину, равную произведению числа маршрутов на число вершин в самом длинном маршруте, деленному на 2. Максимальная сложность тестов по второму критерию для реальных ациклических программ близка числу ветвлений в квадрате.
Увеличение числа вершин графа в 4 раза приводит к возрастанию структурной сложности более чем в 10 раз, потому для упрощения тестирования программ необходимо ограничить размеры прямых моделей в пределах 10-12 вершин.
Наличие циклов в программе способно резко увеличивать сложность их тестирования. Полное, исчерпывающее тестирование должно охватывать проверку каждого маршрута в цикле при всех возможных итерациях цикла и при всех сочетаниях циклов с маршрутами ациклической части программы.
Рассмотрим пример графа программы с циклом (рис.2.3).
Предположим для простоты, что число маршрутов в нижней ациклической части программы М3 =1. Тогда полное множество маршрутов М состоит из полной совокупности всех маршрутов М1 в верхней ациклической части программы и группы маршрутов М2, в которой к каждому маршруту из М1 присоединено 1, 2… итерации (витка) цикла, причем на каждой итерации выполняется по крайней мере один из внутренних маршрутов тела цикла.
Например, для графа, имеющего один цикл, требующего исполнения пяти итераций (витков) с тремя внутренними маршрутами, а также содержащего М1 = 10 ациклических маршрутов, проходящих через цикл, суммарное число маршрутов для исчерпывающего тестирования М = (Зх5)х10 = 150. При возрастании любого из сомножителей (числа независимых ациклических маршрутов, проходящих через цикл, числа внутренних маршрутов тела цикла или числа его итераций) пропорционально растет и сложность тестирования. Поэтому исчерпывающее тестирование реальных сложных программ с циклами практически невозможно.
На сложность тестирования цикла оказывает влияние его структура и два параметра: число маршрутов в теле цикла и число итераций цикла. При реальном исполнении простейшего цикла между его итерациями может существовать зависимость, по крайней мере, трех видов:
на разных итерациях цикла исполняются независимо все возможные маршруты тела цикла;
на всех итерациях цикла исполняется один и тот же маршрут тела цикла или некоторая определенная их последовательность;
на разных итерациях цикла в силу наличия семантических связей исполняется подмножество реализуемых маршрутов тела цикла, зависящее от данных или от номера итерации.
Рисунок 2.3 – Пример графа программы с циклом
Зависимость первого вида встречается наиболее редко. In this case, there is a need for a complete enumeration of all internal routes of the loop body in combination with each number of iterations. При этом сложность тестирования цикла определяется сразу обоими параметрами: числом маршрутов тела цикла и числом итераций и приближается к сложности исчерпывающего тестирования.
With the dependence of the second type, the number of routes of the cycle body practically does not affect the complexity of the cycle. The number of iterations required to test the calculations in the loop body (for example, taking into account the required accuracy) becomes decisive.
The most common is the third addiction. In this case, when assessing the complexity of testing, the number of routes of the loop body is decisive. The simplest estimates of the complexity of cyclic structures should be carried out assuming that the order of implementation of the routes of the cycle body does not depend on the iteration number. In this case, when assessing the complexity of cyclic structures, the approach from the position of the minimum necessary number of checks for iterations of cycles is constructive.
Для оценки сложности структурного тестирования логических программ с простейшими циклами необходимо выбрать критерии проверки. По первому критерию ациклическая часть программы покрывается минимальным числом маршрутов, в которые входят маршруты, проходящие через цикл, размыкающие его и образующие минимальное покрытие тела цикла. Кроме того, к покрытию добавляются маршрут, содержащий замыкающую дугу цикла.
По второму критерию ациклическая часть программы проверяется количеством тестов, равным цикломатической сложности ациклической части программы. При этом к каждому такому маршруту присоединяются все примыкающие к нему циклы. Проверка каждого цикла осуществляется одним маршрутом, содержащим столько итераций, какова цикломатическая сложность тела цикла, а тело цикла покрывается линейно независимыми маршрутами.
При таких оценках определяющими факторами являются полнота проверки тела цикла, условий его замыкания и размыкания. Разнообразие реальных циклических структур в программах сильно усложняет их анализ и получение общих характеристик сложности и достигаемой корректности тестирования. При применении вложенных циклов со сложной структурой и с большим числом ветвлений в теле цикла сложность тестирования резко возрастает и велика вероятность сохранения в программе не обнаруженных ошибок. Однако абстрактные циклы с простейшей структурой приводят к относительно небольшому увеличению суммарной сложности тестов. Поэтому в процессе проектирования программ необходимо в максимальной степени упрощать циклические компоненты в структуре, которые во многих случаях определяют достигаемую корректность программных модулей.
Функционирование любой программы можно рассматривать как обработку потока данных, передаваемых от входа в программу к ее выходу. The input data are sequentially used to determine a series of intermediate results, up to and including the required set of output data. Задача анализа потока данных состоит в установлении корректности их обработки и выявлении ошибок в тестируемой программе. Эта задача может решаться статически — без исполнения программы (анализом по ее тексту) и динамически — путем реального исполнения программы на компьютере при различных исходных данных.
Programming in high-level languages allows you to fully and efficiently analyze data streams in statics, i.e. at the level of the source code of programs. The data involved in computing in high-level programming languages is explicitly defined by name, type, access method and usage. This allows us to consider the program as a multigraph given by the structure of control transfers and the data conversion graph involved in the calculations (Fig. 2.4).
The vertices of the left row correspond to the program operators, and the right row correspond to the variables and constants processed by the program. The control flow between operators is represented in the figure by solid lines, and the data flow is shown in dashed lines. The intersection of control and data flows is carried out in branching operators: condition checks and cycles. Joint analysis of control and data flows allows you to verify the correctness of variable definition areas on program execution routes.
The consequences of errors in the program can appear as small changes in some variables in the calculation process and as a complete distortion or absence of the required values at the output. It is advisable to test the software module on ordered data sets, taking into account the degree of their influence on the output results. From this point of view, it is advisable to distinguish two types of data processing for further analysis:
- completely changing the domain and values of the results;
- altering the results within a certain limited, correct domain of definition.
Figure 2.4 - Example of a program graph with a flow of control
and data stream
The first type of processing corresponds to the initial data at critical points and at the boundaries of the regions of change of variables. При таких критических значениях может изменяться маршрут исполнения программы, из-за чего возможно наибольшее изменение результатов. Поэтому обычно тестирование обработки данных, прежде всего, направлено на проверку исполнения программ при значениях переменных, влияющих на выбор маршрута и логику функционирования программы (стратегия областей). Boundary conditions are situations that occur in close proximity to the boundaries of the areas of change of the variables being processed. Число таких критических значений каждой переменной может быть на несколько порядков меньше, чем число значений по всей внутренней части области изменений этой величины. Most of the critical values can significantly affect the results and are subject to the most thorough testing. В этой части тестирование обработки данных по содержанию близко к тестированию структуры программы. Маршруты формируются в процессе анализа и обработки данных на последовательных операторах условий в тексте программы. Таким образом, всё множество маршрутов является реализуемым и определяется составом реальных тестовых данных. Набор сочетаний исходных данных в тестах непосредственно влияет на степень охвата тестированием участков программы. Сравнив проверенные маршруты с маршрутами, выделенными по графу программы при различных критериях можно оценивать достигнутую полноту тестирования модуля и степень его корректности.
The second type of processing corresponds to data in a bounded or unlimited domain of definition, which can be divided into some set of mating subdomains. Changing the data inside such an area does not affect the program execution route. Поэтому для проверки функционирования программы из всего множества значений достаточно использовать при тестировании только несколько значений внутри и вблизи границ области. Количество величин, используемых для тестирования при обработке этого вида, может быть на несколько порядков меньше полного числа значений каждой переменной в области. В процессе тестирования проверяется точность выполняемых вычислений, правильность масштабирования и размерности обрабатываемых величин. At the same time, testing should cover the entire range of changes for each variable processed and each resulting value.
При анализе обработки данных в пределах внутренних областей их определения методы тестирования нужно применять упорядоченно в следующей последовательности:
Тестирование корректности записи и считывания переменных при вычислениях и полноты состава выходных данных на всех маршрутах исполнения программы.
Testing the accuracy of the results of calculations and the correctness of processing each variable.
Testing for full compliance with the composition, values and accuracy of the output specification requirements.
In the given sequence, individual testing methods allow, first of all, to identify primary errors that are able to distort the results to the greatest extent. With limited resources and such a sequence of testing in the program errors may occur that have less effect on the correctness of the output data.
2.3.2. Testing with data values that determine the program execution routes (area strategy, 1st processing type)
Data processing routes may depend on any type of value. When choosing the direction of branching, variables and constants are involved, expressed as real, integer, boolean, symbolic, vector, and other values. The domains of definition of such quantities depend on their types and content and represent both individual points and disconnected regions, as well as an unlimited continuous sequence of values. One of the tasks of testing is to check the comparability of the compared types of values and the identity of the conditions for their encoding. Critical values affecting the choice of routes, in many cases are not fixed, but are formed during the processing and comparison of several variables. Predicates that determine the choice of routes for program execution are formed as a result of calculations on linear program sections. These sites are small and on average contain about 10 teams. Calculations in most cases are the simplest linear transformations of input data. In addition, predicates are usually also simple, in most cases with one input variable, less often with two.
Each limited source data area corresponds to a specific route in the program. The boundary of the region consists of a set of sections of the boundary, each of which is determined by a single simple predicate that forms an arc of the route in the graph of the program. The total number of predicates in the route is the upper limit of the number of boundary sections of the input variables of this route. Thus, the program with respect to the data flow performs the function of dividing the source data space into regions, each of which corresponds to one executable route. Errors in the program may be due to the modification of the border area of a particular route.
The above method of orderly regular testing based on the definition of data change areas is very effective. The complexity of tests grows linearly with increasing dimension of the source data space (the number of variables) and with an increase in the number of predicates on routes. For many typical modules, the complexity of tests turns out to be permissible for a complete verification of the module. The limitations of the area verification method can manifest themselves in complex organization of cycles, when the number of routes and conditions being analyzed sharply increases.
2.3.3. Testing the correctness of the definition and use of data on the routes of program execution (2nd type of processing)
Each value on the program execution path is read from memory and, after being used for calculations, is written to computer memory for storage and subsequent processing. The alternation of read and write operations of variables can be disrupted as a result of errors in the program. To identify such errors, testing of the correctness of the recording and reading of real values and data or analysis of these operations in the program text is carried out.
Errors in memory allocation usually occur when there is no value to be read in the corresponding register. The reasons for this may be the following:
the operation of recording the required value is absent;
write operation exists, but write is performed in the wrong case;
the correct recording of the required value in this register is destroyed by an unforeseen recording of another value until the use of the primary record.
These errors may not be detected on all routes of program execution. The task of testing is reduced to the detection of erroneous combinations of read-write operations for each value and addresses of registers with which these operations are carried out. For this purpose, information on the addresses of registers and the names of the written and readable values is required.
Testing the process of using variables is based on data flow analysis. All definitions of variables that can reach each vertex of the control program graph are revealed. For each point of the program, it is necessary to establish which variables act at this point, i.e. what data generated before coming to this point is used after exiting it. Data flow analysis allows you to detect errors caused by the violation of the correct sequence of write operations and data read. This analysis is carried out along the program execution routes.
This type of testing is performed predominantly with integer and real values in the inner part of their definition areas. In addition, the accuracy of calculations can be monitored with boundary values that were previously used to test routes by definition areas.
Test values for verifying calculations for simple numeric variables are constructed in an orderly fashion, taking into account the following rules:
input test data should take values close to the largest and smallest, as well as 1-2 intermediate values;
Testing should be carried out at all special values of input variables, namely: at the points of sharp increase or rupture of derivatives, at zero, single and extremely small numerical values;
input test values should provide program verification for output results that have particular points of abrupt change or rupture of derivatives;
if the values of a variable depend on the values of another variable, then it is necessary to test with special values of combinations of variables (equality, small and extremely large difference, zero and unit values).
Thus, for each simple numeric variable, except for three points near and at the boundaries of the domain of definition, it is usually necessary to test the program at 3-4 intermediate and at 2-5 special points of the input data values. If the variables are presented in the form of arrays, then the amount of testing increases significantly. In this case, it is necessary to take into account the structure and size of the array, the presence in the structure of special points or substructures.
Currently, a lot of tools and integrated systems have been created that provide an effective and convenient testing environment for developers of software components. At the same time, not only test methods used, but also service tools that create a comfortable environment for developers are of great importance. Automated testing systems are adapted to the peculiarities of the software being created and the hardware design environment.
Modern systems for testing software components should provide the following features:
convenient user dialogue with automation tools;
use of the design database for the accumulation and storage of information about the developed programs, their versions, test and reference data;
automatic detection of typical errors in the texts of source programs;
automated test planning;
implementation of test options to achieve maximum program correctness in the context of limited testing resources;
evaluation of the achieved program correctness according to the selected testing criteria and determination of the main indicators of the quality of the created program components;
automated registration and documentation of all changes performed in the programs.
Test automation tools are divided into statistical and dynamic.
Statistical analyze source code of programs without their execution. They automate the analysis and verification of the formal correctness of program texts and the identification of appropriate types of errors. The group of static tools includes:
controls for structural program correctness,
controls for writing and reading variables,
search tools for persistent semantic errors,
Means of data preparation for testing planning, which ensure the selection of typical structures in the module (cycles, alternatives, execution routes) and their ordering according to certain rules,
means of calculating the duration of the use of modules.
When using dynamic tools, programs are executed in object code. Means of dynamic testing are divided into:
- fixed assets that ensure the execution of programs in accordance with test tasks and
- Aids that take into account the test performed, its results and making adjustments.
Auxiliary tools are used mainly in cases where software components will be used in critical software systems and special requirements are imposed on their reliability.
For each component tested and debugged, the main tests that were used during the testing, the results of the evaluation of the testing completeness and the correctness of the software component achieved should be stored. These data, together with an assessment of the complexity of the components, structural and informational characteristics are combined into a passport of the program certification.
This methodology establishes the basic methods, technologies and documents that provide testing and debugging of software modules and components consisting of several modules. The purpose of testing is to achieve the required correctness of the software module.
The initial data for testing that needs to be recorded and documented before the start of work are:
Documentation on the developed program module or component. These include:
terms of reference and / or specialization of requirements for the development of the program,
program description in the form of a printed document
user's manual,
source code of the program.
Rules for building and describing programs at different levels and the languages used for their presentation, rules for structural building and component interfaces among themselves, as well as with the external environment.
Specific methods for testing programs: static and dynamic, deterministic and stochastic, and others, depending on specific tasks.
Quality criteria for testing and debugging programs.
Reference values and distributions of source data and results, reflecting the required functions and quality indicators of the program component being created.
Tolerances for deviation of performance results and quality indicators from reference values.
Real test and debug resources.
Before starting testing specific software modules and components, you need to make a test plan. It should include the following steps:
Selection of the test method corresponding to the main goal.
Planning for testing in accordance with the selected method and taking into account resource constraints.
Preparation of tasks for testing with indication of monitored parameters, source data and reference results.
Implement the testing process and get results.
Comparison of results with standards.
Diagnostics, localization and correction of errors and defects.
Evaluation of the completeness of the test and the need to use other test methods.
Definition of program quality testing.
The testing process is carried out in several stages.
At stage 1, the identity of the source text of the program presented on the data carrier with the source text presented in the program document is tested.
At stage 2, integration of the program and information modules included in the component in statistics is performed. At the same time, all interfaces between the modules are checked and their inconsistency with the specifications descriptions is revealed.
At stage 3, an analysis of control flows is performed. In the text of the program, modules, procedures, and functions are distinguished and operators are analyzed that perform control of the computational process. For all levels of the hierarchy of the program, flow graphs are built, which are used to highlight the routes of the program.
At the 4th stage, the analysis of data flows is performed. Testing the correctness of data processing without executing the program. The purpose of this stage is to establish the correspondence between the areas of definition of data sets and the routes of their processing in the program.
At the 5th stage, inconsistencies between interfaces between software modules and information modules included in the component are eliminated.
At the 6th stage, the test results are processed and the quality and correctness of the component are assessed.
At stage 7, a test of the completeness of test sets is performed. The testing process is considered complete if all sets of input test values have been processed and there have been no program failures or results distortion. The fact that a component meets the specified requirements is considered established if the following results are obtained:
the program has the correct structure,
for any function there is a non-empty set of routes for its implementation,
each route completes in a finite number of steps
no route and data mismatch
there are no unrealized and dead-end routes
software documentation conforms to the requirements of the standards and implemented functions.
At the 8th stage, integration of the functional component with other program groups is performed. The interaction of the tested component with other components and the database model of the whole PS is checked.
At stage 9, the functional component is connected to the computer's operating system. After the main problems of interaction of the component with other groups of programs are fixed, it can be connected to the operating system as a load module.
At the 10th stage, the test results are processed, the quality and correctness of the functional component are assessed. At this stage, the results of the analysis of the technical specification or specification, the descriptions of the application of the program (user manual) are compared with the presented test suites and the conclusion is made about the ability of this test suite to fully verify the declared functionality.
When testing programs, 2 types of documentation are created and used:
documentation reflecting the state of test objects,
documentation describing the processes and test results.
Each document must have:
formulated purpose and scope,
professional categories for which it is intended and which it is designed,
stages of work on which it is necessary to apply,
functioning, informative part in accordance with its purpose.
The system for documenting processes and test results is presented in the ANSI / IEEE 829 standard, which should be used as a basis for real-world development. The documentation system covers:
- test planning,
- test specification
- reports on test results.
In terms of testing, it is recommended to formalize:
- purpose of the plan and the purpose of testing,
- objects to be tested,
- the composition of reference materials and definitions of basic terms,
- organization of work, the main schedule of their implementation,
- resources used and the distribution of responsibility of specialists in the project,
- used technological tools and techniques,
- the nomenclature of the issued reporting documents.
The test specification should reflect:
- testing scheme, which specifies the approach to testing and specifies specific criteria for a software product, allowing to evaluate: fit - not fit;
- descriptions of tests and test cases,
- specification of test procedures and scenarios.
Reports on test results should include documents:
- test log, which records the operations performed during the test, and the changes that are made in the programs,
- report defects that require further investigation,
- final report, which summarizes the results of work on testing software components.
Comments
To leave a comment
Software reliability
Terms: Software reliability