Artikel uit SDN magazine 145

Many of us nowadays use unit tests to automatically check the quality of our code. The great thing about unit testing is that we have found a way to automate manual testing. Automated tests have a lot of advantages over manual testing. They are way faster to run, repeatable and all the developers can run those tests. But have you ever considered that unit tests are just code.

Which raises the question, “if unit testing is for ensuring the quality of my code and my unit tests is code, how do I know the quality of my testing code?”

That is what code coverage is for right? Why would you need mutation testing when you already have a 90% or higher code coverage. Code coverage tells you if you have tested all your code paths right? Well, that is not entirely true. Code coverage only tells you if your code has been hit by a test. Not if your test actually tests your code. That is where mutation testing will come into play, mutation testing will tell you if you have tests that… test your code.

How does it work

It probably all sounds a bit abstract, but stay with me. I will try my best to explain how it works, what tool to use for .NET and how easy it is to do mutation testing.

Mutation testing will start off by running all your unit tests to verify all your tests succeed. After it has done this the mutation test tooling will try to break your tests. It will do so by inserting bugs, or mutants as they are called, in your code. When these mutants are introduced at least one of your tests should fail. If no tests fail it is an indication your tests might not be sufficient.

Let’s put that into practice, consider the following code with unit test:

namespace Demo

{

public class DemoCode

{

public int AddNumbers(int v1, int v2)

{

return v1 + v2;

}

}

 

[TestClass]

public class DemoTest

{

[TestMethod]

public void MyTestMethod()

{

// Arrange
var sut = new DemoCode();

// Act
int result = sut.AddNumbers(1, 1);

//Assert
Assert.AreEqual(result, result);

}

}

}

The class DemoCode has one method which will add numbers and return the sum. Our test method however will compare the result to the result and therefore will always return true. When we run our test it will tell us our code works as intended.

Now we run our mutation test, this will alter the code of our AddNumbers method as follows.

public class DemoCode

{

public int AddNumbers(int v1, int v2)

{

return v1 – v2;

}

}

Notice that the ‘+’ operator has changed to the ‘-’ operator. The problem we now face is that our test will still pass while we are now subtracting instead of adding the numbers. This means our test is not sufficient to the purpose of the method and bugs can be easily introduced.

Mutations can be done on multiple different operators and value types. It can flip Booleans, change the number of an integer, empty a string or even declare a variable NULL. It also works on collections and even regex statements.

Who ensures the quality of the mutation testing code

You are probably wondering who will ensure the quality of the mutation testing framework since that too is still code. Well, unit tests and mutation tests do. The wonderful thing about mutation testing is that with it we are finally able to close the testing loop. The Stryker team and community write unit tests, integration tests and are running Stryker to test Stryker.

Use Stryker for .NET

Now the moment you have been waiting for, or just skipped to because you just wanted to know which tool to use. For .NET I would recommend to use Stryker as your mutation testing tool. Stryker is an open source tool which is easy to install, use locally and implement in your pipeline. You can also use Stryker for Scala and Javascript.

All nice and well, but you want to know how to actually use Stryker. It is quite easy, first you will have to install Stryker, you can do so with the ’install stryker-dotnet –tool-path’. After you have installed Stryker you can easily run it by changing the working directory to the path where your test project lives and running Stryker with the command dotnet-stryker from the installation location.

Stryker can create a multitude of reports, it can create a html report, a json report and you can even create a dashboard to see the difference with another branch. This will enable you to see if the pull request you’ve created will lower your so-called mutation score, a score given on how many mutations have survived, or maybe improve your score.

To make it even easier to use the Stryker in a pull request is the integration with SonarQube. You can let Stryker create an issue for every surviving mutation. Together with the fact that SonarQube has Pull request decoration, you don’t even have to look at the report manually or go to SonarQube to see if there has been any changes. You can simply wait for the build to finish and see if there are any comments on your pull request.

Who is it for

Mutation testing is for everyone who has unit tests and wants to build high quality software. It is one of those tools that give you more control and insight on what the quality of your code is and it can prevent bugs from being introduced in your software.

In summary the added value of using Stryker, or any mutation testing tool, is the ability to determine the quality of your tests which in turn will tell something about the quality of your application. This is in contrast to code coverage which only tells you if your code has been hit by a test instead of telling you if your codeMutatio has been tested. Mutation testing can be a great asset in preventing bugs from entering your code and therefore should be in the CI-pipeline of every project.

For more information on how to use Stryker and other functionality it has to offer, please go to their website at https://stryker-mutator.io/