Simple tests in docstrings

[jneug]

I gave tidy a shot for the documentation of an utility package I made. I wanted to add some simple tests for the functions right into the docstrings, like you can do with Pythons' doctest.

I simply made a #test() function, that is passed to the scope of #parse-module and it works fine:

#let module-scope = ( : )  // general scope for tests
#let test( scope: (:), ..tests ) = {
  for test in tests.pos() {
    let msg = if test.text.starts-with("not ") {
      test.text.slice(4) + " should be false, was true"
    } else {
      test.text + " should be true, was false"
    }
    assert(
      eval(test.text, scope:module-scope + scope),
      message: msg
    )
  }
}

In a docstring it can be used like this:

/// Clamps a value between `min` and `max`.
///
/// // Tests
/// #test(
///   `clamp(0, 100, 50) == 50`,
///   `clamp(33%, 99%, 100%) == 99%`,
///   `clamp(-5in, 8in, -6in) == -5in`,
///   `clamp(-5in, 8in, 4in) == 4in`,
/// )
#let clamp( min, max, value ) = {
  // ...
}

This is not a fully featured test suit, but for some simple tests, it could be a nice feature.

Maybe it could be an addition to tidy to support simple tests with a custom syntax like:

/// Clamps a value between `min` and `max`.
///
/// // Tests
/// ~ clamp(0, 100, 50) == 50
/// ~ clamp(33%, 99%, 100%) == 99%
/// ~ clamp(-5in, 8in, -6in) == -5in
/// ~ clamp(-5in, 8in, 4in) == 4in
#let clamp( min, max, value ) = {
  // ...
}

This could improve error messages on failed tests, because it could add real line numbers for the test to the messages.

Thanks for the great package. :-)

[Mc-Zen]

Hi @jneug ,

Cool!

I feel that - broadly speaking - tests should in general be separated from the documentation and for most packages that produce mostly visual output some properly instrumented testing will be necessary.

On the other hand, I think this would be a nice addition - in fact I really much like your idea! It would be useful for smaller tests and for asserting that example code in the docstring works (and keeps working).

Additionally, the advantage of displaying helpful error messages in case of a failed assertation is of course nice.

The test function is somewhat similar to this example function.

We'll need to see how far to go with the testing support but this seems like a good start. The suggestion for the custom syntax seems good, I cannot think of many cases where the tilde would be ambigous, and if it is mandatory to have it at the start of the line as you wrote it, then I see no conflicts with other Typst markup. One open question would be whether to allow and how to handle multiline test cases with the custom syntax.

Thanks for the proposal!

[Mc-Zen]

Hi @jneug ,

On main I implemented some basic docstring test support now! It basically has the syntax and form that you suggested, except that I decided to go for

>>> 2 + 3 == 5

instead of the tilde because I found that with small font sizes, the ~ is a bit harder to distinguish from the dash used to declare parameters.

Additionally, the scope passed to parse-module() is automatically forwarded to the test scope but it is still possible to add more definitions to the test scope using test(scope: ..., tests...).

Furthermore, a few test assertations are available (currently we just have eq(a,b), ne(a,b) and approx(a,b,eps) but more can be added). This will also generate even more elaborate error messages showing the failed code as well as the values to which both sides have been evaluated.

>>> eq(2+3, 5)
>>> approx(calc.sin(calc.pi), 0)

For the shorthand syntax (only), the module name and line numbers are displayed.

I'd appreciate it if you want to try it out a little bit.

One note about a breaking change for style developers: evaluating the descriptions is now deferred to the showing step and there is a function called eval-docstring available in tidy.utilities to do that. Originally I felt that this should be part of the parsing step but the advantages of the new way outweigh that by far. For instance, it is now possible to customize the (new) example function and it opens more possibilities for the future (same issue as with show-reference).

[jneug]

Very cool. This is perfect for my use case.

Will try it out as soon as I got some time to spare.

[jneug]

I finally got the time to use the new functionality. Works great in general.

I found one small issue in testing.typ on line 76: the assert error message is missing the message key.

[jneug]

But the error reporting is great. Good job.

[Mc-Zen]

Just fixed that!

[Mc-Zen]

Thanks for trying out the new features and spotting the bugs ;)

I'll need to add some documentation for docstring testing and then version 0.2.0 is on its way

[Mc-Zen]

Happy New Year @jneug !

It took me a while (doing other stuff) to finish the release but it's there now (PR at typst/packages pending) and docstring tests are supported. Thank you for the proposal :)