Smoke, Sanity, Regression and Re-test: the differences

Lecture

As a test engineer, you have probably heard of such types of testing as «smoke», «sanity testing», «re-test», and regression testing. It is quite possible that you use many of these types on a daily basis.

In this article I would like to bring clarity and explain the difference between these types of testing, and try to figure out where to draw the boundaries (even if conventional) between where one type of testing ends and another begins.

For newcomers to testing (and even experienced testers), distinguishing between these concepts can be difficult. Indeed, how do you tell where sanity testing ends and smoke testing begins? How narrowly should we limit the check of a part of the system's functionality or its components in order to call it «smoke» testing? Is entering a login/password into a website's user login form a smoke test, or is the very fact of its appearing on the site's page already a passed test?

Strictly speaking, you will still be able to carry out testing even if you cannot say exactly what the difference is. You may not even think about distinguishing which particular type of testing you are currently doing. But nevertheless, in order to grow professionally, you need to know what you are doing, why, and how correctly you are doing it.

Basics

Below are brief definitions of the types of testing that we are comparing today:

Smoke tests: performed every time we receive a new build (version) of the project (system) for testing, while considering it relatively unstable. We need to make sure that the critical functions of the AUT (Application Under Test) work as expected. The idea of this type of testing is to identify serious problems as early as possible, and to reject this build (send it back for rework) at an early stage of testing, so as not to get bogged down in long and complex tests, thereby not wasting time on software that is already known to be defective.
Sanity testing: used every time we receive a relatively stable build of the software, to determine its operability in detail. In other words, this is where validation takes place that important parts of the system's functionality work according to requirements at a low level.

Both of these types of testing are aimed at avoiding a waste of time and effort, at determining the software's shortcomings and their severity more quickly, and at determining whether it deserves to move on to a phase of more thorough and detailed testing or not.

Re-test: carried out in the case where a feature/functionality already had defects, and these defects were recently fixed
Regression tests: this is actually what takes up the lion's share of time and what test automation exists for. Regression testing of the AUT is carried out when it is necessary to make sure that new (added) application functions / fixed defects have not affected the current, already existing functionality that worked (and was tested) previously.

For a better understanding, below is a comparison table of these concepts and their areas of application:

Smoke	Sanity	Regression	Re-test
Performed with the goal of checking that the critically important functional parts of the AUT work as they should	Aimed at establishing the fact that certain parts of the AUT still work as they should after minor changes or bug fixes	Confirm that recent changes in the code or the application as a whole have not had a negative impact on already existing functionality/set of functions	Re-checks and confirms the fact that previously failed test cases pass after the defects have been fixed
Goal — to check the «stability» of the system as a whole, in order to give the green light for more thorough testing	The goal is to check the general state of the system in detail, in order to proceed to more thorough testing	Goal — to make sure that recent changes in the code have not had side effects on established, working functionality	Re-test checks that the defect is fixed
Re-checking defects is not the goal of Smoke	Re-checking defects is not the goal of Sanity	Re-checking defects is not the goal of Regression	The fact that the defect is fixed is confirmed by Re-Test
Smoke testing is performed before regression testing	Sanity testing is performed before regression testing and after smoke tests	Carried out based on the project's requirements and the availability of resources (covered by autotests); «regression» can be carried out in parallel with re-tests	— Re-test is performed before sanity testing — Also, re-test has a higher priority than regression checks, so it should be performed before them
Can be performed automatically or manually	More often performed manually	The best reason to automate this type of testing, since manual testing can be extremely costly in terms of resources or time	Not amenable to automation
Is a subset of regression testing	A subset of acceptance testing	Performed with any modification or changes in an already existing project	Re-test is carried out on the fixed build using the same data, in the same environment, but with a different set of input data
Test cases are part of the regression test cases, but cover extremely critical functionality	Sanity testing can be performed without test cases, but knowledge of the system under test is required	Regression testing test cases can be derived from functional requirements or specifications, user manuals, and are carried out regardless of what the developers have fixed	The same test case that identified the defect is used

Let me give an example of distinguishing these concepts on my current project.

Example: we have a web service with a user interface and a RESTful API. As testers, we know:

That it has 10 entry points, for simplicity, in our case located on one IP
We know that all of them accept a GET request as input, returning some data in json format.

Then a number of statements can be made about which types of tests need to be used at which point in time:

By making one simple GET request to one of these entry points and receiving a response in json format, we already confirm that smoke testing has passed.
If one of these entry points also returns data from the database, while the first one does not, then an additional request needs to be made to make sure that the application
correctly handles requests to the database. And with that, the «smoke» test is finished.

That is, we made a request — a response came from the service, and it did not «smoke», that is, it did not return a 4xx or 5xx error, or something unintelligible instead of json. At this point it can be said that the «smoke» test has passed. To check that the UI works the same way, it is enough to simply open the page once in a browser.
Sanity testing in this case will consist of making a request to all 10 API entry points, comparing the received json with the expected one, as well as checking for the presence of the required data in it.
Regression tests will consist of smoke + sanity + UI performed together all at once. Goal: to check that adding the 11th entry point has not broken, for example, password recovery.
Re-test in this example is a targeted check that, for example, an entry point in the API that broke works as intended in the next build.

At the same time, if this API also accepts post requests, then obviously these particular requests need to be included in a separate sanity test set. By analogy with the UI, we will check all the pages of the application.

Let's Sum Up

I hope that after reading this article, you will gain clarity on which type of testing you are using at which stage, and what the difference is between these types of testing. As mentioned at the beginning, the boundary between these concepts is quite conventional and is left to your discretion within the scope of the project.

Often «consistency testing» or «testing for sanity» is called by the term «sanity testing». I think this comes from the phonetic properties of the English word sanity, which sounds similar to something «sanitary». Both variants can be found on the internet. With respect to this article, please consider «sanity» testing as «consistency testing».