# Mutation Testing: Getting Started And Tools To Use

**URL:** <https://club.ministryoftesting.com/t/mutation-testing-getting-started-and-tools-to-use/12042>\
**Category:** Archive\
**Tags:** automation\
**Created:** [7 December 2017 19:06 UTC](https://club.ministryoftesting.com/t/mutation-testing-getting-started-and-tools-to-use/12042 "2017-12-07T19:06:02Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![gmuecke](https://sea2.discourse-cdn.com/flex020/user_avatar/club.ministryoftesting.com/gmuecke/32/2401_2.png) [@gmuecke](https://club.ministryoftesting.com/u/gmuecke)\
**Post date:** [10 December 2017 20:32 UTC](https://club.ministryoftesting.com/t/mutation-testing-getting-started-and-tools-to-use/12042/2 "2017-12-10T20:32:05Z")

</div>

Although it’s often called Mutation _Testing_ it’s not actually Testing, it’s more an analysis – therefore it’s sometimes also referred to as “Mutation _Analysis_” – because it analyzes your Unit Test Suite, if it’s capable of finding bugs by injecting modifications based on a ruleset (mutation operators) into your product code and run your unit test suite against it to see if it’s capable of finding this modification (= killing the mutant).  
It’s completely automated, so you don’t have to _do_ anything for this, but you need to interpret the results and eventually derive actions from them.

There are two statistics,

- ~77% of all production bugs could be covered by a unit test (see [https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf) )
- ~73% of real faults are coupled to mutants generated by commonly used mutation operators (Link [http://people.cs.umass.edu/~rjust/publ/mutants\_real\_faults\_fse\_2014.pdf](http://people.cs.umass.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf))

The first sets the goal for the Unit Test foundation of every test automation pyramid. And the second validates, that mutation testing is a valid way of verifying how close we are to that goal (if its capable of killing a mutant, it’s also capable of killing a bug coupled to that mutant).

## Q1

If you’re on Java, [pitest.org](http://pitest.org) is certainly the way to go, mature tooling, well documented, active community. Applying it to your project is pretty straight forward:

- add the pitest plugin to your project, see quickstart documentation [http://pitest.org/quickstart/maven/](http://pitest.org/quickstart/maven/)
- run it: `mvn org.pitest:pitest-maven:mutationCoverage`

And you’ll get a nice report, which mutations your unit test suite was not able to kill. Further you get a “Mutation Score”, which is a percentage of how many mutants got killed. This is a far better indicator of your test coverage than line or branch coverage. If you achieve a high 90%, you’re pretty close to the 77% goal mentioned above.

In case your project is big (i.e. \> 20kloc) you should consider limiting the scope (i.e. by applying a package filter) because Mutation Testing requires much processing power which increases exponentially with the size of the code base.

If you’re on JavaScript or TypeScript, [https://github.com/stryker-mutator/stryker-mutator.github.io](https://github.com/stryker-mutator/stryker-mutator.github.io) is the tool, though I don’t have much practical experience with it (I don’t do much JS dev)

## Q2

Yes, once I started with mutation testing, I use it in _all_ my (Java) projects for some reasons.

- It’s conducting an automated review of my unit test. No human could provide me the same information in the same amount of time.
- Because it’s automated, it’s very easy to apply (at least for Java, don’t know about the other languages), apart from a strong CPU it doesn’t need much effort from my side.
- It totally changed the way I create my Unit Test Suites, how I write those tests. I do it in a much more systematic way than I did before.
- It effectively helps me close the gaps in my Unit Test Suites, I use Line/Branch Coverage only as weak indicators, once I’m at ~70-80% Line or Branch Coverage, I start mutation testing, sometimes even a bit earlier, especially if I have some unit tests for more complicated code.
- In some cases it actually found a gap in my unit test suite, that didn’t find a bug which was actually there. Although I had Line/Branch Coverage on that part.
- When I’m on a tester role, it gives much better confidence on the quality of the unit test suite the programmers wrote, so I can concentrate on more interesting matters to test
- I don’t do code reviews for unit tests anymore. If I do, I only look on style aspects, but not the functionality (brains are not made to run the code, we computers for that)

## Q3

see Q1, stryker and pitest are the tools do conduct mutation testing in JavaScript respectively Java.

My (incomplete) List of Mutation Testing tools

Java

- Pitest [http://pitest.org/](http://pitest.org/) \<-- use this one, best you can get for free, also for Scala & Kotlin

For Java, I would avoid the following one, because there are more from academic research and most of them are no longer maintained

- muJava
- Jester
- Judy
- Javalanche
- Jumble
- Major

JavaScript

- Stryker [https://github.com/stryker-mutator/stryker](https://github.com/stryker-mutator/stryker)

Ruby

- Mutant [https://github.com/mbj/mutant](https://github.com/mbj/mutant) (still active)
- Heckle [https://github.com/seattlerb/heckle](https://github.com/seattlerb/heckle) (probably abandoned)

.Net/Mono/C#

- NinjaTurtles [http://www.mutation-testing.net/](http://www.mutation-testing.net/) (looks effectively abandoned)
- VisualMutator [https://visualmutator.github.io/web/](https://visualmutator.github.io/web/)
- Nester [http://nester.sourceforge.net/](http://nester.sourceforge.net/)

PHP

- Humbug [https://github.com/humbug/humbug](https://github.com/humbug/humbug)

Haskel

- MuCheck [https://hackage.haskell.org/package/MuCheck](https://hackage.haskell.org/package/MuCheck)

(and sorry for the link obfuscations, as newbie I’m only allowed to add 2 links)

---

_[View the full topic](https://club.ministryoftesting.com/t/mutation-testing-getting-started-and-tools-to-use/12042)._