Sanity Testing

Sanity testing is narrowly focused testing sufficient to prove that a specific function works according to the requirements stated in the specification. It is a subset of regression testing. It is used to determine the operability of a specific part of the application after changes have been made to it or to the surrounding environment. It is usually performed manually.

The difference between sanity testing and smoke testing (Sanity vs Smoke testing)

Some sources mistakenly believe that sanity testing and smoke testing are one and the same. We, however, believe that these types of testing have "vectors of movement" pointing in different directions. Unlike smoke testing, sanity testing is directed deep into the function being tested, while smoke testing is directed wide, to cover as much functionality as possible with tests in the shortest possible time.

Smoke and sanity testing are the most confusing topics in software testing. There is a huge amount of literature on this topic, but most of it is confusing.

The key differences between Smoke and Sanity Testing can be understood with the help of the following diagram:

To evaluate the diagram above, let's first understand -

What is a software build?

If you are developing a simple computer program that consists of only one source code file, you simply need to compile and link that one file to create an executable file. This is a very simple process.
Usually, this is not the case. A typical software project consists of hundreds or even thousands of source code files. Creating an executable program from these source files is a complex and time-consuming task.
To create an executable program, you need to use «build» software, and this process is called « Software Build ».

In Smoke Testing, the test cases select the most important functionality or component of the system. The goal is not to perform exhaustive testing, but to make sure that the critical functions of the system work normally.
For example, a typical smoke test: Make sure the application starts successfully, Make sure the graphical interface responds ... and so on.

What is sanity testing?

Sanity testing is a kind of software testing performed after receiving a software build with minor changes in code or functionality, to make sure that bugs have been fixed and that no additional problems have arisen because of these changes. The goal is to determine that the proposed functionality works roughly as expected. If a sanity test fails, the build is rejected in order to save the time and money associated with more rigorous testing.

The goal is «not» a thorough check of new functionality, but determining that the developer applied some rationality (sanity) when creating the software. For example, if your scientific calculator gives the result 2 + 2 = 5! Then there is no point in testing advanced functions such as sin 30 + cos 50.

Differences between smoke testing and sanity testing

Note.

• Both sanity tests and smoke tests are ways to avoid wasting time and effort by quickly determining whether the application is too broken to warrant any thorough testing.
• Sanity testing is also called tester acceptance testing.
• Smoke testing performed for a specific build is also known as a build verification test .
• One of the best industry practices is to perform a daily build and smoke test of software projects.
• Smoke and sanity tests can be performed manually or with the help of automation tools . When automated tools are used, the tests are often triggered by the same process that generates the build itself.
• Depending on your testing needs, you may need to run both sanity and smoke tests on a software build. In such cases, you would first run the Smoke tests and then proceed with Sanity Testing. In industry, test cases for sanity testing are usually combined with test cases for smoke tests to speed up test execution. As a result, the terms are often confused and used as synonyms.

Sanity check

In short, there is no such thing as sanitary testing.

The word Sanity itself is more simply translated as:

• soundness of mind,
• sanity,
• a normal psyche.

There are no orderlies in it at all. What you call Sanity is, in essence, Smoke testing.

I still don't know when these terms appeared, but it has long been clear that these are two different names for one and the same phenomenon. Testers simply call it Smoke, and programmers call it Sanity. That's just what they got used to, that's how it took root in their literature.

Judging by my experience reading old books for programmers, sanity came first.

Kaner, in his book Testing Computer Software (1988, — mine is the second edition, from 1999 — mentions neither smoke nor sanity. The closest thing to them is in chapter 3 ‘Test types & Software development‘ on page 51, and SUDDENLY it’s called «Acceptance testing» there:

Each time you receive a new version of the program, check whether it’s stable enough to be tested. If it crashes at the slightest provocation, don’t waste your time on it. This first bit of testing is called acceptance or qualification testing.

On page 54, the same topic with clarifications, in the section ‘Final acceptance testing and certification‘:

If your company developed the program on contract, the customer will ran an acceptance test when you deliver it. In small projects, this test may be informal. For most projects, however, test details are agreed to in advance, in writing. Make sure the program passes the test before trying to deliver it to the customer. An acceptance test usually lasts less than a day. It is not a thorough system test. Beizer (1984) describes the preparation and execution of formal customer acceptance tests. Perry (1986) is, in effect, a customer’s guide to creating acceptance tests. Consider using Perry (1986) to structure your negotiations with the customer when you jointly design the acceptance test.

Quote from books.google.com

For those who can't manage to read more than one sentence in a paragraph, let me paraphrase:

agree the set of acceptance tests with the customer in advance, in writing. Make sure yourself that the release won’t fail them. Beizer (1984) and Perry (1986) wrote about this earlier.

This feels strange, coz nowadays «Acceptance testing» is explained as «the customer tests it however they want». And yes, they’ll probably base it on our smoke tests. I assume that nowadays this term is simply interpreted incorrectly, but that’s no longer important.

But in 2001, Kaner already mentions Smoke testing in «Lessons Learned in Software Testing: A Context-Driven Approach» on pages xxvi (that’s the preface, where key term definitions are stated right away), 41, 121, 162, 163, 169, 141 and 144. And not once is sanity testing mentioned.

Rex Black, in «Key Testing Processes» (2004) mentions both terms together. In the Russian-language «Lori» edition from 2006, they’re on page 532:

Acceptance testing, fitness-for-service testing (Smoke Test, Sanity Test). Testing that checks how stable the build submitted for testing is, so that regular testing can begin. This is usually a subset of the whole test suite, typically automated, touching every part of the system, at least superficially. Good acceptance tests usually run the system long enough for serious reliability and operability problems to surface. The term “smoke test” (a test for smoke) comes from electrical engineering: when an engineer switches on a circuit, the first test checks whether any components are smoking.

Wikipedia

SUDDENLY reports that sanity — isn’t even from the world of testing at all: the term exists on its own and applies to every aspect of human activity, from setting a dinner table to dropping a nuclear bomb on the heads of peaceful Japanese folk.

In computer science, a sanity test is a very brief run-through of the functionality of a computer program, system, calculation, or other analysis, to assure that part of the system or methodology works roughly as expected. This is often prior to a more exhaustive round of testing.

A huge, gigantic difference from smoke test, huh?!

ISTQB (download the file Glossary all terms 3.1.pdf).

Sanity test is declared a synonym of smoke test (page 61).

You don’t doubt the ISTQB canon, do you?!

The ISTQB explainer

It is a kind of software testing which is done by the testers to ensure that the functionality is working as expected.

A huge difference from smoke test, huh?!

Someone did almost the same review of books in search of the same terms as I did: almeln.github.io, and arrived at the same conclusions.

On translations: «Yes, but Google translates this word exactly that way!»

Google translates the word Sanity

Google translates the way most people think is right. You can suggest other translations of words there, you can even organize a flash mob and get some word translated in a completely idiotic way, and Google will be totally fine with it.

And lastly. I assumed that this terminology clusterfuck was unique to our Russian-speaking sector. For example, look what they write on protesting.ru:

Some sources mistakenly believe that sanitary and smoke testing are one and the same. We, however, believe that these types of testing have «vectors of movement», directions pointing different ways.

But it thrives on the English-speaking side too. I quickly googled up an article on the subject from h2kinfosys.com — both terms are presented as something separate. I have no idea what kind of outfit this is from the state of Georgia (USA), but about themselves they say «H2k Infosys provides world class IT Training, Real-time Live Project to gain hands on experience, and IT Staffing services».

See also

[[b6091]]

[[b6090]]

See also

• Build Verification Test
• Smoke Testing