Sanity Testing vs Smoke Testing

I came across the following post on LinkedIn about Sanity and Smoke testing, which I thought I’d share here:

Key questions:
What is the difference between Sanity and Smoke testing?
Should we do Smoke testing or Sanity testing on a newly introduced feature in a stable Build?
Is it best to do Sanity or Smoke testing when the build is unstable?

1 Like

Another question that may be risen:

  • Why can’t you know if a version is workable (smoke) and at least barely works (sanity) at building time? Are we meeting the Scribe’s Oath #3: “I will produce, with each release, a quick, sure, and repeatable proof that every element of the code works as it should”?

Smoke-Sanity-Regression…there is too much terminology out there causing much confusion.
You should start with the bare minimum amount of automated tests, even only one, and continue adding more tests till you´ve reached your goal.
This is the key: set your goal. Do you need to check all the features? Do you need to check all your main features? Do you want to check the features the user exercises most?..

NOTE: I used the word “check”…against the word “test”. Pretty interesting difference

2 Likes

I find it interesting that you mention automated tests.

Here, when discussing the use of sanity and smoke testing, I’m using the definitions defined in the following blog: http://www.softwaretestingstuff.com/2009/12/difference-between-smoke-sanity-testing.html

Smoke testing covers all areas of the software application without getting too deep. As you say, we decide which features to test. The goal is only to check they are working, not exhaustively test these features.
Sanity testing covers a more targeted area of the software, based on changes that have been made to the software.

With this in mind, smoke testing can be done with manual testing or test automation, however sanity testing would be harder to automate. Also, since sanity testing is more targeted to cover recent changes, these tests are unlikely to be required unless another change is made to the exact same area.

Smoke tests need to be repeatable, but sanity tests do not which means there is unlikely to be much value from test automation.

1 Like

Sanity and smoke tests are used sometimes interchangeably (wrongly). Even I sometimes fall in that mistake.

They are not specifically related with automated “tests” (i.e. checks), although many times they’re supported as automated “tests”. Why? Because it’s the most expedite way of validating if you broke anything.

Normally I would complement these automated checks with some exploratory testing on top of it, so you can find something that goes beyond the strict and well delimited scope of automated tests.

However, if your basic automated tests fail then probably you need to have a look at them and check what is wrong (it can also be the automated check itself because your system now works in a different way).

I also agree with @juanalvarezarquillos in the sense that you need to “set your goal”; I would rephrase it as assessing your risks and mitigate them properly.

If you have a high-level risk, you should start by looking at that first.

Smoke tests are used to find out if your build is stable enough to be tested exhaustively in a proper environment. Does your software start and keeps running? Is the web server and all remaining services up as expected? If it does reach the minimums, then it can’t be further tested, and development needs to look at immediately.

On the other hand, sanity tests are performed after basic smoke tests. The purpose of sanity tests is to check that changes that were introduced in the build don’t affect the core features of the system.

You can do sanity tests using whatever approach you want (manual scripts, automated scripts, exploratory testing). I’d recommend having a set of automated checks that go over the main features of the AUT/SUT, to give you a quick feedback.

Are smoke and/or sanity tests regression tests? Well, sanity tests for sure and, IMO, smoke tests also as they validate the system based on its architecture, which is consequence of non-functional requirements implemented in previous versions.

Concerning your two questions:

Should we do Smoke testing or Sanity testing on a newly introduced feature in a stable Build?

Is it best to do Sanity or Smoke testing when the build is unstable?

you should do both. Smoke tests should be the ones you or the machine will perform for you. Having a good set of smoke tests can evaluate the stability of it. As more you may find in the advance about the stability of the SUT, better, as it will avoid unnecessary works afterwards.

If smoke tests passed, then you can perform sanity tests and all the regression tests (start with the automated "ones” - i.e. automated scripts/checks). Probably you have to decide which regression tests to perform…and here comes risk assessment: if you have to choose, start with by addressing the higher risks in the context of your build/release :slight_smile:

1 Like

I’ve definitely used these interchangeably, though I do like the nuance presented here.

I’ll probably try to use them more properly moving forward, but I am not going to ding anyone for using them “wrong”.

2 Likes