Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for generating coverage #296

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Add support for generating coverage #296

wants to merge 2 commits into from

Conversation

maxammann
Copy link
Contributor

@maxammann maxammann commented Oct 20, 2023

This adds experimental support for generating coverage using LLVM and cargo-llvm-cov. Note that replacing cargo-llvm-cov would be simple. It might simplify stuff even as cargo-llvm-cov basically just invokes llvm-profdata and llvm-cov.

How to use

Suppose you have a setup like in the examples directory of the repository. Then:

  1. Install cargo-llvm-cov: cargo install cargo-llvm-cov
  2. Install LLVM tools (they might get installed by cargo-llvm-cov automatically, we need to test that): rustup toolchain install nightly --component llvm-tools-preview
  3. Run cargo test
  4. Optionally, fuzz the project.
  5. Execute cargo-test-fuzz test-fuzz target --no-instrumentation --replay corpus --coverage

Coverage is generated as HTML in target/llvm-cov/html.

image

TODO:

  • Should --coverage imply --no-instrumentation?
  • Should --coverage be compatible with fuzzing? (probably not, even though it might just work)
  • Make sure that llvm-tools-preview and cargo-llvm-cov get installed or promt for installation.
  • On macOS I get the following warning during generation: warning: 4 functions have mismatched data
  • During previous experiments I got such coverage data:
    image
    I suspect this is due to the fact that test-fuzz injects code into the function-to-test here:
    #write_concretizations_and_args

    Either move that code somewhere else, or put it into a function, ad a no_coverage instruction and call that from the target function.

Details

The follwing commands are executes to generate coverage. We should validate them:

     Running `/Users/max/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-profdata merge -sparse -f /Users/max/projects/examples/target/examples-profraw-list -o /Users/max/projects/examples/target/examples.profdata`
     Running `/Users/max/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-cov show -format=html -instr-profile=/Users/max/projects/examples/target/examples.profdata -object target/debug/deps/alloc-34551e9cc1ff6b3f -object target/debug/deps/arc-bf8fdeaeed680770 -object target/debug/deps/assert-ba6618cd737ff666 -object target/debug/deps/associated_type-7ed59268473251eb -object target/debug/deps/auto_concretize_0-736514e466198bc3 -object target/debug/deps/auto_generate-7287f7108f5b00a9 -object target/debug/deps/conversion-c5a93588844a6594 -object target/debug/deps/debug-ff5617e8189a0951 -object target/debug/deps/default-79bdc830b898ec0f -object target/debug/deps/from-2bc06bf9dd96fea9 -object target/debug/deps/generic-23602c4ce1352038 -object target/debug/deps/hello_world-594474f770d22b32 -object target/debug/deps/hello_world-cff0c92d5749ded3 -object target/debug/deps/lifetime-f34199c00924285b -object target/debug/deps/manual_leak-7a7f8bfdea2a3f10 -object target/debug/deps/mut_ref-295c3cd2737a4d53 -object target/debug/deps/parse_duration-d93f2ce3d20bd782 -object target/debug/deps/qwerty-72231f74eb29ea0a -object target/debug/deps/rename-503984417115eb3f -object target/debug/deps/return_type-3b4f1b217dc76985 -object target/debug/deps/serde-26f50433033e3e5d -object target/debug/deps/test_fuzz_impl-2a426bdf99ecc53a -object target/debug/deps/unserde-4a2d01352b369939 -object target/debug/hello-world -ignore-filename-regex 'test-fuzz|/rustc/([0-9a-f]+|[0-9]+\.[0-9]+\.[0-9]+)/|^/Users/max/projects/examples(/.*)?/(tests|examples|benches)/|^/Users/max/projects/examples/target($|/)|^/Users/max/\.cargo/(registry|git)/|^/Users/max/\.rustup/toolchains($|/)' -show-instantiations=false -show-line-counts-or-regions -show-expansions -show-branches=count -Xdemangler=/Users/max/.cargo/bin/cargo-llvm-cov -Xdemangler=llvm-cov -Xdemangler=demangle -output-dir=/Users/max/projects/examples/target/llvm-cov/html`

fixes #293

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Max Ammann seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add tooling for generating coverage
2 participants