Mull is a LLVM-based tool for Mutation Testing with a strong focus on C and C++ languages.
Mutation Testing, a fault-based software testing technique, serves as a way to evaluate and improve quality of software tests.
A tool for mutation testing creates many slightly modified versions of original program and then runs a test suite against each version, which is called a mutant. A mutant is said to be killed if the test suite detects a change to the program introduced by this mutant, or survived otherwise. Each mutation of original program is created based on one of the predefined rules for program modification called mutation operators. Each mutant is represented by a mutation point: a combination of mutation operator and location of a mutation in the program’s source code. To assess the quality of a test suite mutation testing uses a metric called mutation score, or mutation coverage.
You can install Mull using prebulit packages or build it yourself from sources as described here: Hacking on Mull
Please, read the intro first, then look at examples.
Mull works on the level of LLVM Bitcode relying on debug information to show
results, therefore you should build your project with -fembed-bitcode and
debug information enabled.
Not every mutation found at Bitcode level can be represented at the source level, so depending on your project Mull may show lots of Junk Mutations. Mull can filter them out by looking at the source code, but for that you need to provide compilation database, or compilation flags, or both.
Please, note: Clang adds implicit header search paths, which must be added
explicitly via -compilation-flags. You can get them using the following
commands, for C and C++ respectively:
> clang -x c -c /dev/null -v
... skipped
#include <...> search starts here:
/usr/local/include
/opt/llvm/5.0.0/lib/clang/5.0.0/include
/usr/include
/System/Library/Frameworks (framework directory)
/Library/Frameworks (framework directory)
End of search list.
> clang++ -x c++ -c /dev/null -v
#include <...> search starts here:
/opt/llvm/5.0.0/include/c++/v1
/usr/local/include
/opt/llvm/5.0.0/lib/clang/5.0.0/include
/usr/include
/System/Library/Frameworks (framework directory)
/Library/Frameworks (framework directory)
End of search list.
If your project depends on dynamic libraries other than libc or lib[std]c++,
then you may need to provide library search paths manually: Mull does not assume
any library search paths because it is very platform dependent.
On Linux, the library search paths are controlled via this file /etc/ld.so.conf,
on macOS it should just work.
Currently, Mull supports Google Test framework and so called Custom Test framework. For Google Test, Mull assesses each test individually, giving a better overview. For Custom Test, Mull treats the whole program as one test.
Mull supports several mutators, you can specify which ones to use. See the full list here:
mull-cxx --help
Build OpenSSL:
git clone https://github.com/openssl/openssl.git
cd openssl
./config -no-asm -no-shared -g -O0 -fembed-bitcode
make all
Run without junk detection:
mull-cxx -test-framework=CustomTest test/bio_enc_test
Run with junk detection:
mull-cxx -test-framework=CustomTest \
-compilation-flags="\
-I . -I crypto/modes -I crypto/include -I include -I apps \
-isystem /usr/local/include \
-isystem /opt/llvm/5.0.0/lib/clang/5.0.0/include \
-isystem /usr/include" \
test/bio_enc_test
Note, the isystem header search paths are the implicit paths used by Clang,
that we should add explicitly. These are platform/version dependent.
Upon success you should see similar output:
Extracting bitcode from executable (threads: 1): 1/1. Finished in 1215ms.
Loading bitcode files (threads: 8): 505/505. Finished in 205ms.
Compiling instrumented code (threads: 8): 505/505. Finished in 17ms.
Loading dynamic libraries (threads: 1): 1/1. Finished in 0ms.
Searching tests (threads: 1): 1/1. Finished in 2ms.
Preparing original test run (threads: 1): 1/1. Finished in 81ms.
Running original tests (threads: 1): 1/1. Finished in 690ms.
Searching mutants across functions (threads: 8): 315/315. Finished in 35ms.
Filtering out junk mutations (threads: 8): 1654/1654. Finished in 4453ms.
Preparing mutations (threads: 1): 1/1. Finished in 528ms.
Applying mutations (threads: 1): 809/809. Finished in 10ms.
Compiling original code (threads: 8): 505/505. Finished in 11ms.
Running mutants (threads: 8): 809/809. Finished in 111615ms.
Survived mutants (569/809):
/opt/mull/openssl/openssl/test/testutil/driver.c:193:29: warning: Math Add: replaced + with -
j = rand() % (1 + i);
^
/opt/mull/openssl/openssl/test/testutil/driver.c:210:17: warning: Math Add: replaced + with -
++num_failed;
^
/opt/mull/openssl/openssl/test/testutil/driver.c:212:49: warning: Math Add: replaced + with -
test_verdict(verdict, "%d - %s", ii + 1, test_title);
.....
..... skipped
.....
/opt/mull/openssl/openssl/crypto/mem.c:249:9: warning: Remove Void Call: removed CRYPTO_free
CRYPTO_free(str, file, line);
^
/opt/mull/openssl/openssl/crypto/o_str.c:86:10: warning: Math Add: replaced + with -
l++;
^
/opt/mull/openssl/openssl/crypto/o_str.c:90:14: warning: Math Add: replaced + with -
return l + strlen(src);
^
/opt/mull/openssl/openssl/crypto/o_str.c:84:17: warning: Negate Condition: replaced > with <=
for (; size > 1 && *src; size--) {
^
Mutation score: 29%
Total execution time: 118987ms
Build fmtlib:
git clone https://github.com/fmtlib/fmt.git
cd fmt
mkdir build.dir
cd build.dir
cmake \
-DCMAKE_CXX_FLAGS="-fembed-bitcode -g -O0" \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
make core-test
Run without junk detection and only one mutator:
mull-cxx -test-framework=GoogleTest -mutators=math_add_mutator ./bin/core-test
Run with junk detection:
mull-cxx -test-framework=GoogleTest \
-mutators=math_add_mutator \
-compdb-path compile_commands.json \
-compilation-flags="\
-isystem /opt/llvm/5.0.0/include/c++/v1 \
-isystem /opt/llvm/5.0.0/lib/clang/5.0.0/include \
-isystem /usr/include" \
./bin/core-test
You should get the following output:
Extracting bitcode from executable (threads: 1): 1/1. Finished in 218ms.
Loading bitcode files (threads: 4): 4/4. Finished in 373ms.
Compiling instrumented code (threads: 4): 4/4. Finished in 12ms.
Loading dynamic libraries (threads: 1): 1/1. Finished in 0ms.
Searching tests (threads: 1): 1/1. Finished in 1ms.
Preparing original test run (threads: 1): 1/1. Finished in 68ms.
Running original tests (threads: 8): 29/29. Finished in 71ms.
Searching mutants across functions (threads: 8): 2630/2630. Finished in 64ms.
Filtering out junk mutations (threads: 8): 157/157. Finished in 3313ms.
Preparing mutations (threads: 1): 1/1. Finished in 1076ms.
Applying mutations (threads: 1): 26/26. Finished in 12ms.
Compiling original code (threads: 4): 4/4. Finished in 12ms.
Running mutants (threads: 8): 26/26. Finished in 805ms.
Survived mutants (18/26):
/tmp/1553947865/fmt/test/gmock-gtest-all.cc:9758:53: warning: Math Add: replaced + with -
const int actual_to_skip = stack_frames_to_skip + 1;
^
/tmp/1553947865/fmt/include/fmt/core.h:309:19: warning: Math Add: replaced + with -
reserve(size_ + 1);
^
/tmp/1553947865/fmt/test/core-test.cc:416:45: warning: Math Add: replaced + with -
for (std::size_t j = 0; j < num_inputs; ++j) {
^
/tmp/1553947865/fmt/test/core-test.cc:415:43: warning: Math Add: replaced + with -
for (std::size_t i = 0; i < num_inputs; ++i) {
^
/tmp/1553947865/fmt/test/core-test.cc:416:45: warning: Math Add: replaced + with -
for (std::size_t j = 0; j < num_inputs; ++j) {
^
/tmp/1553947865/fmt/test/core-test.cc:415:43: warning: Math Add: replaced + with -
for (std::size_t i = 0; i < num_inputs; ++i) {
^
/tmp/1553947865/fmt/test/core-test.cc:416:45: warning: Math Add: replaced + with -
for (std::size_t j = 0; j < num_inputs; ++j) {
^
/tmp/1553947865/fmt/test/core-test.cc:415:43: warning: Math Add: replaced + with -
for (std::size_t i = 0; i < num_inputs; ++i) {
^
/tmp/1553947865/fmt/test/core-test.cc:416:45: warning: Math Add: replaced + with -
for (std::size_t j = 0; j < num_inputs; ++j) {
^
/tmp/1553947865/fmt/test/core-test.cc:415:43: warning: Math Add: replaced + with -
for (std::size_t i = 0; i < num_inputs; ++i) {
^
/tmp/1553947865/fmt/test/core-test.cc:416:45: warning: Math Add: replaced + with -
for (std::size_t j = 0; j < num_inputs; ++j) {
^
/tmp/1553947865/fmt/test/core-test.cc:415:43: warning: Math Add: replaced + with -
for (std::size_t i = 0; i < num_inputs; ++i) {
^
/tmp/1553947865/fmt/test/core-test.cc:416:45: warning: Math Add: replaced + with -
for (std::size_t j = 0; j < num_inputs; ++j) {
^
/tmp/1553947865/fmt/test/core-test.cc:415:43: warning: Math Add: replaced + with -
for (std::size_t i = 0; i < num_inputs; ++i) {
^
/tmp/1553947865/fmt/include/fmt/format.h:1086:67: warning: Math Add: replaced + with -
if (next_arg_id_ >= 0) return internal::to_unsigned(next_arg_id_++);
^
/tmp/1553947865/fmt/include/fmt/format.h:933:54: warning: Math Add: replaced + with -
*--buffer = static_cast<Char>(data::DIGITS[index + 1]);
^
/tmp/1553947865/fmt/include/fmt/format.h:939:39: warning: Math Add: replaced + with -
*--buffer = static_cast<Char>('0' + value);
^
/tmp/1553947865/fmt/include/fmt/format.h:1086:67: warning: Math Add: replaced + with -
if (next_arg_id_ >= 0) return internal::to_unsigned(next_arg_id_++);
^
Mutation score: 30%
Total execution time: 6072ms
Here is the starting point: CONTRIBUTING.md
Mull it over: mutation testing based on LLVM (preprint)
@INPROCEEDINGS{8411727,
author={A. Denisov and S. Pankevich},
booktitle={2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)},
title={Mull It Over: Mutation Testing Based on LLVM},
year={2018},
volume={},
number={},
pages={25-31},
keywords={just-in-time;program compilers;program testing;program verification;mutations;Mull;LLVM IR;mutated programs;compiled programming languages;LLVM framework;LLVM JIT;tested program;mutation testing tool;Testing;Tools;Computer languages;Instruments;Runtime;Computer crashes;Open source software;mutation testing;llvm},
doi={10.1109/ICSTW.2018.00024},
ISSN={},
month={April},}
Copyright (c) 2016-2019 Alex Denisov [email protected] and Stanislav Pankevich [email protected]. See LICENSE for details.