How much testing should happen in the repository?

[MilesMcBain]

I raise this as something that the community could benefit from hearing CRAN's perspective on, but also a high level question for future top-level repositories.

CRAN's current policy settings and activities seem to discourage exhaustive testing as part of CRAN's checks. This is because even a single test failure for trivial functionality on one uncommon platform can trigger a two-week "fix or be archived" ultimatum from CRAN. See {lubridate} for a recent example.

When these ultimatums land at inconvenient times, they can create stress for maintainers. There are many anecdotal reports, even from people in this working group, of skipping selected tests on CRAN to avoid the inconvenience.

CRAN's policy does not have much to say on the matter, except that:

Long-running tests and vignette code can be made optional for checking, but do ensure that the checks that are left do exercise all the features of the package

So the question is how much testing is appropriate for a core repository?

• Should package authors be encouraged to reach for 100% test coverage in repository checks?
  • If so, can that be incentivised?
  • If not, can we create better guidance as to the types of things that should be tested?
• Are there resource-driven constraints on the amount of testing the average package can do? E.g. a number of minutes of running time?
  • If so, how can resource use be optimised?
• Are there particular classes of test that are better off not run on the core repository?
  • In CRAN's case, can we let people know about these, and save them some hassle?

[MilesMcBain]

Putting my developer hat on:

I think it's most important to have tests running on the core repo that exercise your dependencies, because having them change underneath you is what will break core features of your package. The newer or faster-moving a dependency is, the more thorough you should be about testing code involving it on the core repository.

[hadley]

Answering from my perspective as tidyverse developer (i.e. this is the advice I'd give to my team, although they might not 100% agree with it):

• I personally don't think it's important to aim for 100% test coverage. I think it's good idea to aim for high coverage, but IMO 100% is usually overkill and sometimes requires you to resort to tricks to hide parts of your code that you can't easily test automatically. I think it would be useful to show test coverage in more places.
• Generally, I think it's good for CRAN to run some tests to ensure that your package is working correctly, but they are not a CI platform, and it's more important that you're testing thoroughly elsewhere. Our default position is to run all tests on CRAN, but we have no problem skipping anything even mildly troublesome on CRAN. (And I think our decision to automatically skip snapshot tests on CRAN has proven to be correct).
• That said, we generally avoid running tests that rely on external services (i.e. some other website) on CRAN. There's too high a risk that the site will be temporarily down, creating a R CMD check failure that needs to be remedied. We also have tests that only run on CI because there's no way share to share secrets with CRAN.
• For the tidyverse in particular, we also want to ensure our packages work on older versions of R, which is not a service that CRAN provides.
• When testing a dependency, it's very easy to accidentally write tests that depend on incidental features, causing your package to "break" when something unimportant changes in a dependencies (e.g. the text of an error message changes, which we see fairly frequently).

[wlandau]

Taking a step back here, I think the underlying competing pressures would be greatly mitigated by the possible collaboration with R-hub. Thinking about an ideal world where every single CRAN machine is a container image that everyone else can access (including the Dockerfile). I think that would take out a lot of the guesswork in terms of figuring out what to test.