On the way to v1.0

[AlexDenisov]

This issue is supposed to be a place to discuss what needs to be implemented for version 1.0.
These should include show-stoppers but can also include nice-to-haves. We'll need to decide which ones belong to which group.

Everyone is encouraged to share their ideas and suggestions. I'll start.

---

• Multithreading support: mull itself is multithreaded, but if the program under test is multithreaded, then Mull may produce non-deterministic results. Currently, Mull gathers the code coverage based on call trees. In a single-threaded program (most of the time) the call tree is deterministic (i.e., the order of the calls is the same from run to run). A possible solution is to use another, 'linear' coverage approach: to only collect whether the function was executed or not. It has some drawbacks but will give the most deterministic behavior. Not needed anymore
• TLS support: this is being in the works currently emutls: Add helper shared library for TLS test subjects #745. Not needed anymore
• Diff reporter: Mull has several reporters (mutation elements, SQLite, CLI). it would be very nice to also have a reporter that produces git-like patches so that they can be applied to the source code or used as part of CI integration. Related is it possible to save the source code level mutant record? I want to analysis them #10198 #587.
• CI integration: mull can be run as part of the CI process, but the integration is not trivial and does not have smooth CI integration. Directly related to Integration of mull with GitHub Annotations #738.
• Diff-based mutation filter: at the moment, Mull does not respect version control systems. For better CI integration it is essential to filter out mutants that are unrelated to a pull or merge request.
• Non covered mutants: as described here Feature request: mull line coverage #636.
• More test frameworks: it would be nice to have more test frameworks supported. Related Add Boost.Test support #174 Add Catch support #175 Add CppUnit support #191 Add Doctest support #492 TestFrameworks: Add Boost.Test support for discovering test cases #746. More suggestions are very welcome. Cannot support it anymore
• AST-based mutations: after a lot of considerations we decided that AST-based mutations (based on Clang plugins) are the must-have.

---

The above list is not exhaustive and may be extended.

[AlexDenisov]

• Smooth junk detection: currently, Mull requires a user to provide 'implicit' C/C++ header search paths manually. It would be nice if we can automate this process in a transparent way.

[jnohlgard]

Great to collect all the important features in one ticket. Would it be useful to also create a v1.0 Label or Milestone to mark PRs and issues which are part of this effort?

[AlexDenisov]

That's a good idea. I'll create a milestone and a label. I guess for the milestone to work I'd also need to create an issue per each feature/task.

[chgans]

About the diff filter, are we talking something like clang-tidy-diff?
https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html

[AlexDenisov]

About the diff filter, are we talking something like clang-tidy-diff?
https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html

That is correct.

[ligurio]

Vote for "Add ability to catch mutants using tests written on other languages #778"

[ligurio]

More test frameworks: it would be nice to have more test frameworks supported. Related #174 #175 #191 #492 #746. More suggestions are very welcome.

It is definitely good for those who use these frameworks. But I believe it would be a nightmare for Mull maintainers to support and test all of these frameworks. I would suggest to generalize integration with test frameworks (no ideas how) or simplify integration that will allow users easily integrate without changing Mull source code.

Also there is a report published by Jetbrains where developers answered on question "Which unit-testing frameworks do you regularly use, if any?". Below is a distribution of different unit testing frameworks for C and C++:

Source: C language:

Source: C++ language

[AlexDenisov]

Dropping JIT is certainly on the list #798.

[AlexDenisov]

Yet another follow-up: since the JIT is gone there is no special need for multithreading and TLS support. Also, we cannot have/don't need explicit test frameworks support anymore.

I edited the original comment to reflect the current state.