Automatic Testing of documentation

[mmesiti]

(Already mentioned in issue #3)
Maintaining up-to-date documentation can be hard. While there are a lot of tools that can be used to write documents interspersed with code and make them reproducible by locking the version of the software that they are produced with, making sure that the documentation of a project is up to date with the code is more difficult.
This problem is even more crucial for system documentation, where by design the system changes over time.

There are tools like doctest, that take the code examples contained in docstrings and check that the results are the expected ones, but are there tools that allow to do this on pieces of documentation that do not live inside source code?

I would like to discuss the tools and techniques, and possibly discuss/design a tool that can help in this regard, that can be used to test the in-line code examples that are usually contained in documentation.

Depending on the interest, it could be a longer hackathon-like session, where people would come with their own piece of documentation and create a test suite for it while exchanging ideas on how to do this reliably, or just a discussion/brainstorming session about this topic.

Honest part of this post:

1. I do not think that any standard tool for this exists, but I may be wrong and then this BOS might have little sense
2. Having a 90-minute session with only discussions might be a little too long, but it could be too little to do anything practical, on the other hand it could be not worth more than 90 minutes.

(Submitting this 2 minutes before the deadline - if that wasn't already yesterday - for fun and insomnia)

[HeidiSeibold]

Thanks @mmesiti for this last minute submission 👏

If you were to implement something, how much time would you need for e.g. an MVP? What do you think?

[chillenzer]

Great idea! I discussed this with @chryswoods on RSEcon22 in relation to his amazing talk on Tutorial-Driven Development (great recommendation btw). We agreed that there is need for such a tool and that the current tooling around this is sparse at best. Rmarkdown was the closest we could come up with. Would be very interested in contributing.

[HeidiSeibold]

I added a label to this break-out. Can you check if you feel it is appropriate and change it if not? Let me know if you have any questions.

[mmesiti]

Thanks for the comments! The tags all do make sense. The one about 'coding' might require a real answer to this question...

Thanks @mmesiti for this last minute submission clap

If you were to implement something, how much time would you need for e.g. an MVP? What do you think?

This is something that I still do not know. It depends on the details of the technologies used in the documentation, I think.

The use case that I am "closest to" is taking existing pieces of documentation, written using mkdocs. I could present an analysis of that case, and a possible solution... but this might not be what others are interested in.

In this very discussion (and elsewhere) Rmarkdown was mentioned. It also look like Pluto Notebooks (#20) could be as close as Rmarkdown to a solution to this problem. Yet another fancy alternative could be org babel for Emacs hackers. I do not know any of these well enough to assess how close they are to a solution, and how hackable they are into a solution, and how they could be used for existing documentation that is not yet using that format (i.e., how easy the migration is). If someone would like to contribute with a mini-walkthrough of these, this could make this BOS more worthwhile.

2*90-minute slots would be enough to run:

• A presentation part (15-30 minutes):
  • describe the problems in slightly more detail - What are the pieces of documentation that we would like to work on?
    • What do they describe? (e.g. a system, a piece of software... )
    • What is the technology behind them? (e.g. mkdocs, sphinx, jekyll... )
    • How close are the examples in those documentation to something that can be directly executed? Do all the snippets need to be executed in order, or are they independent from each other?
  • describe some existing related technology/tool (e.g. Rmarkdown, pluto.jl, org babel, jupyter notebooks). Here I'd try to be flexible: for example if everybody has just been to a BOS on pluto.jl, maybe we can avoid telling again what it is.
• A discussion/brainstorming part (30-45 minutes): how can we apply existing tools to our problems, what do we need to implement ourselves? What features should this have? Can we implement something general or do we need to treat different flavours of the problem separately?
• A "hackathon" part (rest of the time - duration of the summary part): where we get our hands dirty and try to see if our thoughts were correct
• A summary part (5-10 minutes): where we tell each other our results.

With a single slot, the "hackathon" part could be reduced, and the summary part would also be shorter since there should be less work to report on.

[HeidiSeibold]

Thanks for the clarifications 👍

So what I need from you to be able to make a decision on this breakout are the following infos:

Who could be people interested in collaborating on this?

(feel free to tag them with their GitHub username if they have one)

Sounds like you (@mmesiti), @chillenzer and @chryswoods are obvious choices so far. Anybody else?

How much time do you need for this?

2*90=180 minutes

Abstract

(Can be short)

[chillenzer]

Sure, I'm in!

[HeidiSeibold]

Hi @mmesiti can you please respond by Tuesday morning, thanks 😃

[mmesiti]

Who could be people interested in collaborating on this?

Organization: @mmesiti , @chillenzer (If I am not mistaken)
Partecipation: Whoever has faced the problem of stale documentation

How much time do you need for this?

180 minutes

Abstract

Maintaining up-to-date documentation can be hard. When the behaviour of a software differs from the documented behaviour, users might lose trust in the documentation. This problem is equally important, if not more, for system documentation.

In this session we would like to collect experiences and cases, to discuss the tooling and practices available
to mitigate this problem.

We also aim at working concretely on documentation that has maintenance issues, to create systems that can spot
differences between the real and the documented behaviour, share our experiences and discuss similarities and divergencies in our approaches, to try and design, or even create, reusable tools.

[pancetta]

Hi! I just added the "Accepted" label to this BOS. Welcome on board!
https://un-derse23.sciencesconf.org/program

[pancetta]

Hi all, the unconference is only 3 weeks away now! On day 1 there will be a breakout blitz where all session organizers should advertise their sessions. 1 minute, 1 slide, let people know what you intend to do. Please prepare this slide in advance and add it right here (PDF please), by September 20.

[mmesiti]

Here is the slide:
automatic-doc-test-intro.pdf

[pancetta]

Here is the main hub for taking notes: https://pad.gwdg.de/FkFJTslFQhq-UF3Es6q4rw#

[pancetta]

Have fun with the session(s)! Please add the pad you're using also here for people to see what you did.

If possible, please prepare a 1 minute wrap up of your session for the farewell session on Thursday afternoon! What did you do in the session, how would you like to continue, how can people contribute after the unconference etc. We'll go through the blitz slides again one by one as in the blitz session.