Questions tagged [mutation-testing]

A method of software testing that involves modifying the source code or compiled intermediate code (Java: byte code, .NET: IL). Test suites that do not fail under each of these "mutations" is considered to be inadequate.

Mutation testing is a method of testing the adequacy of test suites. By applying mutations to the code under test, the effectiveness of a test suite can be judged. If the test suite fails under this mutation, it is effective against it.

Different mutation types are used: Ma, Offutt and Kwon describe intra-method, inter-method, intra-class and inter-class mutations. The simplest to implement is intra-method mutation, for which some of the mutations that may be used are:

  • Statement deletion.
  • Replace each boolean subexpression with true and false.
  • Replace each arithmetic operation with another one, e.g. + with *, - and /.
  • Replace each boolean relation with another one, e.g. > with >=, == and <=.
  • Replace each variable with another variable declared in the same scope (variable types should be the same). (source: Wikipedia)
191 questions
49
votes
11 answers

Is mutation testing useful in practice?

Do you have any examples of real life applications of mutation testing? Does it work better than simple test coverage tools? Or is it useless? What are the advantages/disadvantages of mutation testing in the real world?
Mnementh
  • 50,487
  • 48
  • 148
  • 202
47
votes
6 answers

What mutation-testing frameworks exist for C/C++?

Mutation testing has been out there for a while now, and it seems there are at least one or two commercial mutation testing frameworks for C/C++. Have you used them? What are your experiences? Are there any open source alternatives?
Jason Orendorff
  • 42,793
  • 6
  • 62
  • 96
32
votes
11 answers

What mutation-testing frameworks exist?

In another question I asked if mutation-testing is useful in practice. As I didn't get any answers that satisfy me, I want to check the current tools myself. So I need an overview about current existing mutation-test-frameworks. Which of them are…
Mnementh
  • 50,487
  • 48
  • 148
  • 202
31
votes
1 answer

JUnit tests pass but PIT says the suite isn't green

While trying to run a PIT mutation test I get the following error: mutationCoverage failed: All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite. The tests run just fine when I do a normal…
Bobby
  • 451
  • 1
  • 4
  • 9
16
votes
2 answers

Mutation testing tool for Python 2.7

While writing another unittest in Python and after learning about multiple testing methods, I got curious how many mutants would be killed if I performed mutation testing on my code. The problem is that I use Python 2.7 and the only tool I found is…
15
votes
2 answers

Are there any good mutation testing tools for ruby 1.9 and RSpec2?

I used to use Heckle, but it is incompatible with ruby 1.9 because of issues with ParseTree. I've looked for alternatives, but the only thing that looked promising was Chaser, and that did not have any clear documentation that I could use to see if…
Matt Van Horn
  • 1,654
  • 1
  • 11
  • 21
13
votes
5 answers

My PITEST won't run. Coverage generation minion exited abnormally. I need help to configure my pom.xml properly

When running mvn org.pitest:pitest-maven:mutationCoverage, I get the error as follows ( Environment: Windows 10, Maven 3.6.1, Java 11, junit-jupiter 5.4.1, pitest 1.4.7) [ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.7:mutationCoverage…
Øyvind Roth
  • 217
  • 1
  • 2
  • 11
13
votes
1 answer

Mutation not killed when it should be with a method with an auto-injected field

I have the following: public class UnsetProperty extends Command { @Resource private SetProperty setProperty; public String parse(String[] args) { if (args.length != 4) { throw new RuntimeException("Incorrect number…
Madara's Ghost
  • 172,118
  • 50
  • 264
  • 308
12
votes
2 answers

Java Mutation Testing Integration with Maven

Our project will integrate mutation testing as part of its build cycle. Right now, we are on the evaluation phase. So far, I have only found that Jester has a maven-plugin (Grester). I have not been able to find anything for the other mutation…
chahuistle
  • 2,627
  • 22
  • 30
12
votes
1 answer

llvm based code mutation for genetic programming?

for a study on genetic programming, I would like to implement an evolutionary system on basis of llvm and apply code-mutations (possibly on IR level). I found llvm-mutate which is quite useful executing point mutations. As far as I have understood,…
alex
  • 315
  • 1
  • 11
10
votes
2 answers

FSharp.Compiler.Service compile to dll errors

I am currently working on a project to implementing mutation-testing for F#. To access the FCS process I am using the FSharp.Compiler.Service package. I am having some trouble with the compilation process and the compiled DLL. I'm using…
dukedagmor
  • 171
  • 5
10
votes
3 answers

Excluding tests from execution by PIT

I have to exclude my integration tests from their execution by PIT. There is an option excludedTestClasses since version 1.3.0. I tried to pass over these tests by the following configration of the Maven plugin of PIT.
Oliver
  • 3,815
  • 8
  • 35
  • 63
9
votes
2 answers

What is the difference between Property Based Testing and Mutation testing?

My context for this question is in Python. Hypothesis Testing Library (i.e. Property Based Testing): https://hypothesis.readthedocs.io/en/latest/ Mutation Testing Library: https://github.com/sixty-north/cosmic-ray
8
votes
1 answer

pitest excludedMethods maven

I am trying to exclude PIT from mutating I/O methods, such "close" and "flush". Here is my Maven configuration: org.pitest pitest-maven 1.1.3
q3769
  • 192
  • 1
  • 12
8
votes
1 answer

Mutation Testing Tool for Android

I'm very interested in Mutation Testing, so I've tried some tools like mujava and this tool worked well with Java code. I've also read that applying mutation testing on Android isn't trivial. So, my question is, does anyone know a tool or a way to…
bianconero
  • 215
  • 1
  • 12
1
2 3
12 13