Test data, test sets and equivalence classes

When discussing the task of test design, two important artifacts were described: test scenarios and test suites. Without test data, these two artifacts cannot be implemented. They are descriptions of conditions, scenarios and paths, but they do not provide specific values for implementation. Test data itself is not an artifact, but it noticeably affects the success or failure of a test. Testing cannot be carried out without test data. Test data is required for the following:

• input data to create a condition
• output data to evaluate a requirement
• auxiliary data (as a precondition for the test)

Therefore, determining values is an important part of the work when creating test suites, see Work Product: Test Suite and Guideline: Test Suite.

When determining the actual test data, four parameters should be considered:

• volume - the amount of test data
• completeness - the degree of variation of the test data
• coverage - the applicability of the data to the testing objective
• architecture - the physical structure of the test data

It is also necessary to check extreme boundary values. For example, if a string input is expected, this means an empty string and a very long one; if a number is expected, this means a large negative number, a large positive number, zero, and so on.

These parameters are described in detail in the following sections:

Volume

Volume is the amount of data used for testing. Data volume is an important parameter; if there is too little data, it may fail to reflect all possible situations, and if there is too much, it will be hard to work with. It is best if testing starts with a small volume of data suitable for critically important (and, as a rule, positive) test suites. As testing confidence increases, the data volume should expand until it covers all significant deployment cases in the environment (within reasonable limits).

Completeness

Data completeness means the degree of variation of the test data. It can be increased by creating more records. This is often enough, but in fact we need to see the variations in the data that could be encountered in a real situation. Without this, testing can miss certain errors, and not everyone is limited to withdrawing strictly $40.00 from an ATM. Therefore, the test should account for data variation in a real environment, for example, withdrawing $20.00 or $110.00. In addition, the data must account for possible variations of actual data, for example:

• Names may include titles, numbers, punctuation and suffixes:
  • Prof. Petrov, Mrs. A. Markizova, J. Gay-Lussac
• Addresses may be written on several lines:
  • 9011 Broadway Street
    Suite 111
  • 2311 Broadway
    Floor 18
    Mailstop 1b
• Country codes and phone numbers:
  • Lexington, MA, USA + 0111 626 0000
  • Kista, Sweden +45 1 22 33 44 00
  • Paris, France +33 2 33 44 55 66

Test data values may involve choosing physical or statistical data based on real data. Both methods have their own value and are recommended for use.

To create test data based on physical data, determine the valid range of values for each element and make sure that for each such element the test data contains at least one record with valid values.

Example:

This table contains the minimum number of records that could represent valid data values. For each of the three account number ranges there is one record, all PIN codes fall within the valid range, there are several variants of account balance including a negative one, and the records cover all three account types. This table contains the minimum amount of data, and it can be improved by adding values at the boundary of each range and within the range -

The advantage of the physical representation is that the test data is limited in volume, easy to manage, and contains valid values. The disadvantage is that it does not reflect the statistical trends of real data. Real data is usually statistically heterogeneous, and its trends can affect performance. This factor will not be taken into account when working with the physical representation.

The statistical approach to creating test data reflects the characteristics of the data that the real system works with. For example, one could analyze a production system database and reveal the following:

• Total number of records: 294031
• Total number of type S accounts: 141135 (48%)
• Total number of type C accounts: 144075 (49%)
• Total number of type X accounts: 8821 (3%)
• Account numbers and PIN codes are evenly distributed

given this statistics, the test data could include 294 records (rather than 4, as before):

This table reflects only account types. In the statistical approach to creating test data, significant data elements should be included. In this example, that could mean accounting for actual account balances.

The disadvantage of the statistical approach is that the data may not reflect the full range of valid values.

Usually a combination of both methods is applied, and the test data includes values that reflect both the performance aspect and the completeness aspect.

The concept of test data completeness applies both to test data used as input data for the test, and to that used as auxiliary data to already existing data.

Coverage

Coverage is the applicability of the test data to the purpose of the test. It is related to the volume and completeness of the data. A large amount of data does not mean that it necessarily contains the data needed. As with data completeness, we must make sure that the data meets the purpose of the test, that is, whether executing the test will answer the questions posed.

For example, in the following table the first four records contain valid values for each data element. However, for account types C and X there are no records with a negative balance. Therefore, although this test data does include a record with a negative balance (acceptable completeness), it would not be sufficient for testing negative balances for all account types (insufficient coverage). This omission is overcome by adding records containing a negative balance for the two other account types.

The concept of test data coverage applies both to test data used as input data for the test, and to that used as auxiliary data to already existing data.

Architecture

The concept of the physical structure of test data applies only to test data used as auxiliary data to already existing data, for example, in a database or a rules table.

Testing is not a process that happens only once. Testing is repeated both within and between iterations. In order for testing to be accurate, reliable and efficient, the test data should be returned to its original state before executing the test. This is especially important for automated tests.

Accuracy, reliability and efficiency of testing are achieved when the test data does not depend on external factors, and its state is known before, during and after test execution. To achieve this goal, two problems need to be solved:

• volatility / isolation - eliminating external influences on the data
• initial state - precise knowledge of the initial state of the data and reproducibility of that state

All of these issues affect working with the test database, designing the test model, and interacting with other objects.

Volatility / Isolation

Test data volatility can arise for the following reasons:

• external influences unrelated to the test change the data
• test participants do not know what data is used by other participants

For testing to be reliable and complete, the test data must be fully isolated from such influences. The following methods are used for this:

• separate test environment - test participants work in a fully isolated environment, physically separated from the rest. They have their own test object and data. This is achieved, for example, by having each test participant work on their own computer.
• separate base instances of test data. Test participants work with their own instance of data, isolated from other influences. The physical environment, and possibly the test object itself, may be shared, but each participant works with their own instance of data, which in this case turns out to be unaffected by outside influence.
• partitioning of test data within the database. All participants work with a shared database and know what data is used by other participants (and do not use it). For example, one participant works with records 0 - 99, another - with records 100 - 199, or one works with customers whose last names begin with letters A-M, and another - with N-Z.

Initial State

An important issue in the architecture of test data is the state of the data at the start of the test. This is especially important when automating testing. Both the test object itself and the test data must be in the required, controlled state. This makes it possible to achieve repeatability of test results and confidence in the reliability of the tests.

Four strategies are usually used to address this issue:

• data update
• data re-initialization
• data reset
• reliance on current data

All of them are described in more detail below.

The actual method will depend on various factors, including the physical characteristics of the database, the technical competence of the test participants, the availability of external roles (unrelated to the test), and the test object itself.

Data Update

The preferred way of returning data to its original state is data update. This creates a copy of the baseline data in its original state. Upon completion of the test run (or before it starts), the copy of the test data is placed into the test environment. This ensures the identity of the test data before the start of the test.

With this method, data can be saved for different initial states. For example, test data may include archives for the state at the end of the day, end of the week, end of the month, etc. In this case, the test participant can quickly restore the state of any test, for example, a use-case test corresponding to the end of the month.

Data Re-initialization

If the data cannot be updated, the next way is to restore the original state of the data using some program. Data can be re-initialized for certain use cases with the help of tools that restore the initial value of the test data.

In doing so, special care must be taken to ensure that no errors creep into the data, its relationships and key values.

One advantage of this method may be testing with invalid values from the database. Under normal conditions the database does not contain invalid values, because they are not allowed to be entered there (for example, this is prevented by a validation rule in the client). However, the data may be changed in another way (for example, during an update from another system). When testing, it is necessary to make sure that invalid data will be recognized and handled correctly, regardless of how it arose.

Data Reset

A simple method of restoring the original state of the data can be "reversing the changes" made to the data during testing. This method relies on the ability of the test object to undo changes, that is, to add back deleted data and restore the values of modified data.

This method also carries corresponding risks, namely:

• all actions without exception must be reversed
• it relies on the use case of the test object. It is necessary to make sure that it works correctly and can be used to reset the data.
• it is not always possible to restore the values of keys, pointers and indexes in the database

If this method is the only one available in your environment, then do not use keys, indexes and pointers in the database as verification tools. So, to determine whether a patient was registered in the database, use their name, not the system-generated patient ID.

Reliance on Current Data

This approach is the least acceptable for restoring the initial state of the test data. In fact, it does not solve this issue. Instead, the state of the data upon completion of one test becomes the initial state of the test data for another test. Usually this requires making changes to the input data for the test and/or to the use cases and test data used for analyzing the results.

In some cases such an approach is necessary, for example, at the end of the month. If there is no archive of data for the end of the month, then the test data and test scenarios for each day of the month must be "rolled forward" to bring the data into a state suitable for the end-of-month test.

The following risks are associated with this method:

• it is not always possible to restore the values of keys, pointers and indexes in the database
• the data is constantly changing
• the reliability of the results requires additional verification

Equivalence Classes and Boundary Values

Equivalence class — a part of the domain of input or output data for which the behavior of a component or system is considered the same.

An equivalence class – is a set of tests from which the same result is expected. In the simplest case, a test represents a set of input data entered into the program under test. In the case of equivalent tests, this data has common properties.

A group of tests constitutes an equivalence class if the following conditions are met:

One should strive to identify as many equivalence classes as possible. This will save testing time and make testing more efficient, sparing the tester from repeating equivalent tests. Having split all intended tests into classes, one can then select in each of them one or several of the most effective tests; there is no need to run the remaining tests.

Example The system asks the user to enter an Arabic digit in a field. Equivalence class = [0,1,2,3,4,5,6,7,8,9] To check correctness, it is enough to take one element, for example, 4.

Boundary values Problems very often arise if values at the boundaries of equivalence classes are entered. A boundary value — an input value that lies at the edge of an equivalent domain or at the smallest distance from either side of the edge, for example, the minimum or maximum value of the domain.

example Boundary values For the age of majority, the boundary values — are 17 and 18. At 17 a person is not yet of age, while at 18 — they already are.

Usually, for a system (with the exception of the simplest applications), it is impossible to test all logically possible input combinations. Therefore, one of the most important tasks for developers is to select, for testing, those combinations where the majority of errors are most likely to be found.

Testing based on equivalence class analysis (synonyms: equivalence partitioning, domain analysis) is a black-box testing analysis. The goal of the analysis is to reduce the total number of test cycles to a minimum while detecting the maximum possible number of errors. This method makes it possible to divide a set of input and output data into a finite number of equivalence classes and select a representative test value for each class. The test results for a class's representative value are considered "equivalent" to other values of the same class. If no errors are found when testing the representative value, it is assumed that all other "equivalent" values will also be free of errors.

invalid values

Invalid - means not valid, incorrect, wrong, not meeting requirements. For example: Invalid e-mail means that the address has an incorrect format, consists of incorrect characters in an incorrect sequence

The advantage of equivalence class analysis is that it allows one to adhere to a selective strategy and reduce the combinatorial explosion of potentially necessary tests. This method makes it possible to derive logical criteria for selecting individual tests from among all possible tests. Below are areas where a large number of tests is required, and where the use of equivalence classes could ease the work:

1. Combinations of independent variables
2. Dependent variables based on hierarchical relationships
3. Dependent variables based on temporal relationships
4. Cluster relationships based on market models
5. Simulated composite relationships

Guidelines for finding equivalence classes:

Strategies

There are various strategies and methods for conducting testing with partitioning into equivalence classes. Some examples are given below:

Equivalence Class Partitioning

The theory of equivalence class partitioning, proposed by Glenford Myers [MYE79], is aimed at reducing the total number of test scenarios required by partitioning input conditions into a finite number of equivalence classes. Two types of classes are created: the valid input data of the program is regarded as a valid equivalence class, and all other input data is placed into an invalid equivalence class.

Below are guidelines for determining equivalence classes:

1. If an input condition specifies a range of values (for example, a program "accepts values from 10 to 100"), then one valid equivalence class (from 10 to 100) and two invalid ones (less than 10 and greater than 100) are defined.
2. If an input condition specifies a set of values (for example, "fabric can be of various colors: red, white, black, green, brown"), then one valid class (with valid values) and one invalid class (with all invalid values) are defined. Each value of the valid class must be handled separately.
3. If an input condition contains a must-be expression (for example, "the input string must contain uppercase characters"), then one valid class (uppercase characters) and one invalid class (all other input variants except uppercase characters) are defined.
4. All actions performed "long before" the execution of a task are considered an equivalence class. All actions performed shortly before the program terminates are considered another class. All actions performed immediately before the program launches another operation are also considered a separate class.
5. If a program is configured to work with memory ranging from 64 MB to 256 MB, then this range is considered an equivalence class. Any other amount, greater than 256 MB or less than 64 MB, is considered an invalid class.
6. Partitioning of output events into classes is related to the program's input data. Although some input equivalence classes may correspond to output events of the same type, it is recommended to consider input classes separately.

Boundary Value Analysis

For each equivalence class, it is assumed that boundary conditions allow errors to be found with a greater probability than other conditions. Boundary conditions are considered to be the nearest values lying on either side of the extreme values that define the boundaries of the class.

When testing boundary conditions, the following values are selected from the tested range: the minimum value (min), the value one greater than the minimum (min+), the value one less than the maximum (max-), and the maximum value (max). In this case, developers select several test scenarios for each equivalence class. With a relatively small number of tests, the probability of detecting errors is high. Developers do not have to test a huge number of values for which the test results differ insignificantly from one another.

Guidelines for selecting boundary values:

1. If the valid value of a floating-point variable ranges from -1.0 to 1.0, test the values -1.0, 1.0, -1.001 and 1.001.
2. If the range of valid input values is an integer from 10 to 100, test 9, 10, 100, 101.
3. If the program is configured to work with uppercase characters, test the boundary values for A and Z. Test @ and [, because in the ASCII code the character @ precedes A, and the character [ immediately follows Z.
4. If the program's input or output data represents an ordered set, test the first and last elements of the set.
5. If the sum of the input data is expressed as a number (n), test the program where the sum equals n-1, n or n+1.
6. If the program supports a list, test the values from that list. All other values are considered invalid.
7. When reading from or writing to a file, test the first and last characters of the file.
8. The smallest monetary unit is one cent or its equivalent. If the program supports some range from a to b, test a -0.01 and b +0.01.
9. If several ranges of values are specified for a variable, each range is considered a separate class. If the subsets of values from these ranges do not overlap, test the extreme values, as well as the boundary values above the upper boundary and below the lower boundary.

Special Values

After applying the two boundary value analysis strategies described above, it is recommended to examine the program for "special values", with the help of which many errors can be detected. Some examples are given below:

1. In the case of integers, it is essential to test zero, if it belongs to the valid equivalence class.
2. When testing time, it is necessary to test the values 59 and 0 for each element (hour, minute and second), regardless of the restriction set for the input variable. In addition to the boundary values of the input variable, the values -1, 0, 59 and 60 must always be tested.
3. When testing a date (year, month and day), test scenarios should be added, for example, for the number of days in a specific month, the number of days in February in a leap year, or the number of days in a non-leap year.

The list of equivalence classes is best organized in the form of a table. There are usually many equivalence classes, so a convenient and well-thought-out way of organizing the collected information is needed. Example of a list of equivalence classes:

• The possible outcomes of selections from lists and menus should be analyzed. Any item in a list of options offered by the program can represent a separate equivalence class. Each item of a menu or list of options is handled by the program in a special way, so all of them are subject to verification.
• Groups of variables that participate jointly in certain calculations whose result is limited to a specific range of values should be identified. For example, the class of valid values includes the magnitudes of the three angles of a triangle, which together sum to 180°. Invalid values can be divided into two equivalence classes: those with a total value less than 180°, and those with a total value greater than 180°.
• Options for the operating environment should be thought through. It happens that a program works well only with certain types of monitors, printers, modems, disk devices, or any other equipment connected to the system. Therefore it is important to determine the classes of equivalent system configurations.

It is sufficient to run one or two tests for each equivalence class. The best of them are those that check values lying on the boundaries of the class. Incorrect comparison operators (for example, > instead of ≥) cause errors only at boundary values of the arguments. At the same time, a program that fails on intermediate values of a range will almost certainly also fail on its boundary values.

Each boundary of an equivalence class needs to be tested from both sides. A program that passes these tests is very likely to pass all the other tests belonging to that class as well.

The "Category Partitioning" Method

Ostrand and Balcer developed a partitioning method that allows testers to analyze the system specification, create test scenarios, and manage them. While most strategies focus on working with source code, the Ostrand and Balcer method also involves using specification and design data.

The main advantage of this method is that it allows errors to be detected even before the code is created, because the source of input is the specification, and testing is based on its analysis. Shortcomings in specifications can be detected at an early stage, often before they are implemented in code.

The steps for applying the "category partitioning" method are listed below:

1. Analyze the specification: break down the set of system functions into functional units that can be tested independently of one another, using both the specification and the implementation.
   Next:
   
   1. Determine the parameters and environment conditions on which the execution of the function depends. Parameters are the input data of the functional unit. Environment conditions are the states of the system on which the execution of the functional unit depends.
   2. Determine the characteristics of the parameters and environment conditions.
   3. Break the characteristics down into categories on which the behavior of the system depends.
      
   At this stage, ambiguous, contradictory, and missing elements of the description of the system's behavior will be discovered.
   
2. Divide the categories into choices. Choices are situations that may arise unexpectedly. Each choice contains the same type of data as the category.
   
3. Determine the constraints and relationships between choices. Choices from different categories are interrelated, and this needs to be taken into account when composing the test scenario. Constraints prevent contradictions from arising between choices with different parameters or from different environments.
   
4. Relying on the information about categories, choices, and constraints, develop test scenarios. If an error occurs when using a particular choice, do not add it to other choices when composing the scenario. If a choice can be fully tested with a single test, this means it is a special case of the choice or a special value.

Examples of finding equivalence classes and partitioning boundaries for test data

See also

• [[b5545]]
• [[b5191]]
• [[b7769]]
• [[b5187]]

See also

• 7. Development of the schedule of it-project
• Web testing: features, kinds and sequence
• Methods to ensure the high quality of the developed software
• Types and kinds of software testing. Testing levels and testing methods