Lecture
Regression testing is a type of testing aimed at verifying changes made to an application or its environment (a defect fix, code merge, migration to a different operating system, database, web server, or application server), to confirm the fact that functionality that existed previously still works as before (see also sanity testing, or consistency/correctness checking). Regression tests can be either functional or non-functional tests.
As a rule, test cases written at the early stages of development and testing are used for regression testing. This guarantees that changes in the new version of the application have not damaged existing functionality. It is recommended to automate regression tests, in order to speed up the subsequent testing process and detect defects at the early stages of software development.
The term "regression testing" itself, depending on the context in which it is used, can have different meanings. Sam Kaner, for example, described 3 main types of regression testing:
Regression testing (from Lat. regressio — backward movement) — a collective name for all types of software testing aimed at detecting errors in already-tested sections of source code. Such errors — where, after changes are made to a program, something that was supposed to keep working stops working — are called regression errors (regression bugs).
Regression testing (according to some[which?] sources) includes new bug-fix — verifying the fix of a newly found defect, old bug-fix — verifying that a previously fixed and verified defect does not reproduce in the system again, and also side-effect — verifying that the operability of previously working functionality has not been broken, if its code could have been affected while fixing certain defects in other functionality. The commonly used methods of regression testing include re-running previous tests, as well as checking whether regression errors have entered the next version as a result of code merging.
From software development experience it is known that the recurrence of the same errors is a fairly frequent occurrence. Sometimes this happens because of weak version-control technique or because of human error when working with the version control system. But just as often the solution to a problem turns out to be «short-lived»: after the next change to the program, the fix stops working. And finally, when rewriting some part of the code, the same errors that were present in the previous implementation often resurface.
Therefore it is considered good practice, when fixing a bug, to create a test for it and run it regularly with subsequent changes to the program. Although regression testing can also be performed manually, it is most often done using specialized programs that allow all regression tests to be run automatically. Some projects even use tools for automatically running regression tests at a given time interval. This is usually done after every successful build (in small projects), or every night or every week.
Regression testing is an integral part of extreme programming. In this methodology, design documentation is replaced by extensible, repeatable and automated testing of the entire software package at every stage of the software development process.
Regression testing can be used not only to verify the correctness of a program; it is also often used to assess the quality of the result obtained. For example, when developing a compiler, running regression tests examines the size of the resulting code, its execution speed, and the compilation time of each of the test examples.
In their article, S. Yoo and M. Harman provide the following classification of regression testing:
Test suite minimization seeks to reduce the size of the test suite by removing test cases from the set of tests based on a given criterion. There are three approaches, the first of which applies automated security testing to detect vulnerabilities by studying application faults that can reveal known malicious programs, such as viruses or worms. This approach considers only the tests that failed in the previous version, for re-running in the new version of the system after the fault has been fixed.
Another approach is intended for detecting and eliminating vulnerabilities in minor releases of web applications. It sets up a rigid link to the pages of the previous version using iterators, which are selected for studying the web pages that contain vulnerabilities.
And, finally, the third approach proposes testing with self-adaptation of the system for already-known failures. The authors avoid reproducing already-known errors by considering for execution only those tests that revealed known failures in previous versions.
The test prioritization task concerns the correct ordering of tests, which maximizes the desired properties, such as early detection of faults. Moreover, current approaches to priority assignment consider only vulnerabilities.
One of the methods proposes fault-based priority tests that directly use knowledge of their ability to detect faults.
Another proposes a modifiable record-and-replay system that allows a recorded, executed version of the application to be rewritten into a new, modified one. Their execution is prioritized based on determining the optimal modifiable rewriting using a cost function and measuring the difference between the original execution and the modified one on replay.
The selection method makes it possible to choose a subset or all of the test cases in order to verify the changed parts of the software. The following approaches test both security mechanisms and vulnerabilities.
Regression testing is performed when changes are made to existing functional capabilities of the software, or when there is a bug fix in the software. Regression testing can be implemented through several approaches. Successful passage of all tests by the modified program provides confidence that the changes made to the software have not affected existing functional capabilities, which must remain unchanged in any case.
In agile project management processes, where the software development life cycle is very short, resources are lacking, and changes are made to the software very frequently, regression testing can introduce a great deal of unnecessary overhead.
As a rule, regression testing is carried out using automation tools, but the current generation of regression testing tools is not designed to handle database applications. For this reason, when running a regression test on applications that use databases, unplanned expenses may arise, since this will require a great deal of manual labor.
The fundamental problem in program maintenance is that fixing one bug has a fairly high probability (20—50 %) of introducing a new one. So the whole process proceeds on the principle of «two steps forward, one step back».
Why can't errors be fixed more carefully? First, even a hidden defect manifests itself as a failure in just one place. In reality, however, it often has ramifications throughout the whole system, usually not obvious ones. Any attempt to fix it with minimal effort will fix the local and obvious part, but unless the structure is very clear, or the documentation very good, the remote consequences of that fix will go unnoticed. Second, errors are usually not fixed by the program's author, but often by a junior programmer or an intern.
Because of the introduction of new errors, program maintenance requires significantly more systematic debugging per statement than any other kind of programming. In theory, after every fix the entire set of test cases against which the system was previously verified should be re-run, to make sure that it has not been damaged in some incomprehensible way. In practice, such regression testing really ought to approach this theoretical ideal, and it is very costly.
— F. Brooks The Mythical Man-Month, or How Software Systems Are Created
Comments