Mull might need more mutation operators (get_sign example demonstration)

[stanislaw]

This is a test program:

int get_sign(int x) {
  if (x > 0) {
    return 1;
  }

  if (x < 0) {
    return -1;
  }

  return 0;
}

int main() {
  // exit code 0 -> test passes
  // exit code 1 -> test failed

  if (get_sign(1) != 1) {
  	return 1;
  }
  // if (get_sign(-1) != -1) {
  // 	return 1;
  // }
  if (get_sign(0) != 0) {
  	return 1;
  }

  return 0;
}

This is the Makefile:

default: compile
	/sandbox/build/build.mull.dir/tools/driver-cxx/mull-cxx \
		-test-framework=CustomTest \
		-compdb-path=fixed.comp.json \
		-mutators=all \
		-ide-reporter-show-killed \
		hello-world

compile:
	clang -fembed-bitcode -g main.cpp -MJ comp.json -o hello-world
	echo "[" > fixed.comp.json
	cat comp.json >> fixed.comp.json
	echo "]" >> fixed.comp.json

run: compile
	./hello-world

This is the execution result:

Running mutants (threads: 6): 6/6. Finished in 13ms.

Killed mutants (6/6):

/sandbox/mull-hello-world-example/step-4-second-mutation/main.cpp:2:9: warning: Killed: Replaced > with >=
  if (x > 0) {
        ^
/sandbox/mull-hello-world-example/step-4-second-mutation/main.cpp:2:9: warning: Killed: Replaced > with <=
  if (x > 0) {
        ^
/sandbox/mull-hello-world-example/step-4-second-mutation/main.cpp:6:9: warning: Killed: Replaced < with >=
  if (x < 0) {
        ^
/sandbox/mull-hello-world-example/step-4-second-mutation/main.cpp:6:9: warning: Killed: Replaced < with <=
  if (x < 0) {
        ^
/sandbox/mull-hello-world-example/step-4-second-mutation/main.cpp:2:9: warning: Killed: Replacing scalar with 0 or 42
  if (x > 0) {
        ^
/sandbox/mull-hello-world-example/step-4-second-mutation/main.cpp:6:9: warning: Killed: Replacing scalar with 0 or 42
  if (x < 0) {
        ^

Survived mutants (0/6):

Mutation score: 100%

Total execution time: 178ms

It can be seen that there is no survived mutation that would hint at the missing test case (the one that is commented out).

Would be nice to find out which mutation would reveal this missing test case.

[stanislaw]

One mutation that will work is "remove if block". And this leads us to a yet another bit of work of actually implementing that mutation operator.

[jnohlgard]

A normal test coverage measurement (without mutations) would show that the -1 case is never executed. Mutation testing is primarily useful to discover situations where the normal test coverage shows that a line is covered but the actual results are not properly verified.