Mutation Testing: Getting Started And Tools To Use

🗄️ Archive
2

Although it’s often called Mutation Testing it’s not actually Testing, it’s more an analysis – therefore it’s sometimes also referred to as “Mutation Analysis” – because it analyzes your Unit Test Suite, if it’s capable of finding bugs by injecting modifications based on a ruleset (mutation operators) into your product code and run your unit test suite against it to see if it’s capable of finding this modification (= killing the mutant).
It’s completely automated, so you don’t have to do anything for this, but you need to interpret the results and eventually derive actions from them.

There are two statistics,

The first sets the goal for the Unit Test foundation of every test automation pyramid. And the second validates, that mutation testing is a valid way of verifying how close we are to that goal (if its capable of killing a mutant, it’s also capable of killing a bug coupled to that mutant).

Q1

If you’re on Java, pitest.org is certainly the way to go, mature tooling, well documented, active community. Applying it to your project is pretty straight forward:

And you’ll get a nice report, which mutations your unit test suite was not able to kill. Further you get a “Mutation Score”, which is a percentage of how many mutants got killed. This is a far better indicator of your test coverage than line or branch coverage. If you achieve a high 90%, you’re pretty close to the 77% goal mentioned above.

In case your project is big (i.e. > 20kloc) you should consider limiting the scope (i.e. by applying a package filter) because Mutation Testing requires much processing power which increases exponentially with the size of the code base.

If you’re on JavaScript or TypeScript, https://github.com/stryker-mutator/stryker-mutator.github.io is the tool, though I don’t have much practical experience with it (I don’t do much JS dev)

Q2

Yes, once I started with mutation testing, I use it in all my (Java) projects for some reasons.

  • It’s conducting an automated review of my unit test. No human could provide me the same information in the same amount of time.
  • Because it’s automated, it’s very easy to apply (at least for Java, don’t know about the other languages), apart from a strong CPU it doesn’t need much effort from my side.
  • It totally changed the way I create my Unit Test Suites, how I write those tests. I do it in a much more systematic way than I did before.
  • It effectively helps me close the gaps in my Unit Test Suites, I use Line/Branch Coverage only as weak indicators, once I’m at ~70-80% Line or Branch Coverage, I start mutation testing, sometimes even a bit earlier, especially if I have some unit tests for more complicated code.
  • In some cases it actually found a gap in my unit test suite, that didn’t find a bug which was actually there. Although I had Line/Branch Coverage on that part.
  • When I’m on a tester role, it gives much better confidence on the quality of the unit test suite the programmers wrote, so I can concentrate on more interesting matters to test
  • I don’t do code reviews for unit tests anymore. If I do, I only look on style aspects, but not the functionality (brains are not made to run the code, we computers for that)

Q3

see Q1, stryker and pitest are the tools do conduct mutation testing in JavaScript respectively Java.

My (incomplete) List of Mutation Testing tools

Java

  • Pitest http://pitest.org/ <-- use this one, best you can get for free, also for Scala & Kotlin

For Java, I would avoid the following one, because there are more from academic research and most of them are no longer maintained

  • muJava
  • Jester
  • Judy
  • Javalanche
  • Jumble
  • Major

JavaScript

Ruby

.Net/Mono/C#

PHP

Haskel

(and sorry for the link obfuscations, as newbie I’m only allowed to add 2 links)

8 Likes