tests: Add pytest and nushell based tests

[cgwalters]

tests: Prune testcloud cache on each run

Because it's not a cache, it's a copy (there's no cache invalidation,
GC, etc.)

Signed-off-by: Colin Walters [email protected]

---

dockerignore: Exclude more things

This avoids doing a container rebuild when touching the tmt
bits for example.

Signed-off-by: Colin Walters [email protected]

---

tests: Add pytest and nushell based tests

I've been trying to keep this project in "one" programming
language by writing even tests in Rust...but specifically
for our integration tests it's pretty painful not just to
compile them but have to deal with baking them into the base image.

The tmt framework is very GHA like in that it scrapes the
git source tree and copies it into the target environment, which
works really well with scripts.

Now, if you know me you know I am not a fan of dynamic programming
languages like bash and Python. I'm one of those folks that actually
tries to use Rust for things that feel like "scripts" i.e. they're
mostly about forking external processes (see the xtask/
crate which uses "xshell").

Some of our testing code is in Rust too. However...there's a giant
tension here because:

• Iteration speed is very important for tests and scripts
• The artifact being an architecture-dependent binary pushes us
  to inject it into container images; having the binary part
  of the bootc image under test conceptually forces us to reprovision
  for each test change, which is super expensive

Most other people when faced with the testing challenge would
just write shell scripts (or Python); that's definitely what tmt
expects people to do.

The podman project has a mix of a "bats" suite which is all
bash based, and a Go-based framework.

The thing is: bash is easy to mess up and has very little ability
to do static analysis. Go (and Python) are very verbose for forking external
processes.

I've been using https://www.nushell.sh/ for my interactive shell
for quite a while; I know just enough to get by day to day
(but honestly sometimes I still type "bash" and run a few things there
that I know how to express in bash but not nu)

Anyways though, nushell has a lot of desirable properties for
tests (which are basically scripts):

• Architecture independent
• Running an external process requires zero ceremony; it's the
  default!
• But it is easy to e.g. scrape JSON from an external binary
  into a rich data structure
• A decently rich standard library

The downside is, it's a new language. And in the end, I'm
not going to say it's the only way to write tests...maybe we
do end up with some more bash. It wouldn't be the end of the world.
But...after playing with this, I definitely like the result.

OK, and after some debate we decided to add Python too, so this
demos a pytest test.

Signed-off-by: Colin Walters [email protected]

---

Closes: #543

[mvo5]

nushell looks very cool, thanks for the pointer!

[mvo5]

I read in the commit message that dynamic languages are not the preferred choice but I do wonder if maybe pytest might deserve a second look? Combined with tooling like pylint/mypy/ruff it many(or only some?) of the dynamic language pain points go away and while subprocess management is a little bit more complicated it's also more explicit, i.e. it's always clear what an arg for execv() is (i.e. less quoting issues). The upside(s) would be that it's more familiar to many people and that many of the things a testsuite needs (asserts, diffing if assert fails, error reporting etc) are already available.

This particular example in pytest wouldn't look too bad (IMHO):

import json
import subprocess


def test_bootc_status_json_smoke() -> None:
    """ verify bootc status --json looks sane """
    st = json.loads(subprocess.check_output(["bootc", "status", "--json"]))
    assert  st["apiVersion"] == "org.containers.bootc/v1alpha1"

Of course this might change once a more complex or pipeline heavy example come into play. But of course those are only my 2¢ and I can see that nushell has many upsides too!

[cgwalters]

Right, that example is trivial; here's a less trivial one that motivated this that's already in another PR:

https://github.com/containers/bootc/pull/620/files#diff-e6320bbd16d432b6d223da82f11037166a0da7a0c81b2575c4505aeaf6085cc9

More generally...what I've been trying to aim for is recreating some of the flavor/experience I'm used to with the "coreos-assembler kola external tests", which today are mostly shell script (and by which I have often been burned), but I like the framework around it.

Now admittedly...that test ended up being only about half running external binaries; less than I thought. I am totally not a nushell expert and so don't take it as any kind of reference example, but it felt somewhat nice. Especially the second half where it's really mostly just forking external binaries.

(A tangential aside, if you want a good sense of nushell, I came across https://github.com/nushell/nushell/blob/main/.github/workflows/release-pkg.nu which is very much the kind of thing that might start out as a small bash script split out of a Github actions run invocation which is 98% forking things; then later it grows from 10 to 100+ lines and you feel the need for better language features, but turning it into Python again I think would make it a good bit more unnecessarily verbose than it is nushell)

This all said...the next day after sending up this PR I definitely was feeling conflicted about introducing a new (relatively) obscure language (though I was pleased to discover it has a language server at least!). Introducing new languages into a codebase isn't something to do lightly - it implies a large ongoing maintenance and forces context switching for everyone who wants to do the basics of change the code and change the tests in this case.

Combined with tooling like pylint/mypy/ruff it many(or only some?) of the dynamic language pain points go away

I admit I stopped writing Python day to day years ago now, and the language has changed a lot. I came across
https://kobzol.github.io/rust/python/2023/05/20/writing-python-like-its-rust.html recently which was really useful.

and while subprocess management is a little bit more complicated it's also more explicit, i.e. it's always clear what an arg for execv() is (i.e. less quoting issues).

To be clear, nushell and also the xshell macro we're already using in the Rust-based tests are both concise and don't have that quoting issue. (bash of course, does unless you rigorously add quotes, or you add shellcheck to your CI, which 90% of the time points out unnecessary things)

---

Anyways bigger picture...the other strong arguments for Python here is that:

• It's already in the base images we are targeting
• It's a language people already either know, or can learn easily
• In today's world LLMs know Python well too and they can be very helpful for translating/explaining/creating
• It's already used for tests in bib

And of these things the latter is interesting...getting to a point where bootc and bib actually share more logic around building and testing seems like an important goal. So given that, I'm fine adding tests in Python too...let me try updating this PR to do that.

(But yes the downside is I personally have low experience with the Python ecosystem like pytest etc., but I can learn)

[cgwalters]

On a related topic, I did reference in the commit message:

The podman project has a mix of a "bats" suite which is all bash based

Which if you look at the surface like https://github.com/containers/podman/blob/7b4f6ec576ca6419db8d9a3d0a93c635ae05fccc/test/system/055-rm.bats is...generally OK I think. But if you look at the implementation, things like https://github.com/containers/podman/blob/7b4f6ec576ca6419db8d9a3d0a93c635ae05fccc/test/system/helpers.bash#L908 are an eldritch horror...

[cgwalters]

This language stuff is easy to get sidetracked on, but just one other interesting bit...nushell is less dynamic than either bash or python; by default each script is parsed and type-checked before being executed at all. For example, just interactively in my shell:

> ls | from json
Error: nu::parser::input_type_mismatch

  × Command does not support table input.
   ╭─[entry #1:1:6]
 1 │ ls | from json
   ·      ────┬────
   ·          ╰── command doesn't support table input
   ╰────

And typoing a variable name in a nushell script is caught by default. Of course though it's still a pretty dynamic language, a lot of things have the "any" type and there's lots of ways to fail at runtime.

Anyways...testing some python and pytest now.

[cgwalters]

OK I added infrastructure sufficient to run pytest too...not yet dropping the nushell stuff since I'm in a bit of a "sunk cost fallacy mode" 😄 and don't feel like rewriting the larger test I did in the other PR for now...but, we'll see how things go. Maybe we can drop it if it isn't holding its weight later.

[mvo5]

OK I added infrastructure sufficient to run pytest too...not yet dropping the nushell stuff since I'm in a bit of a "sunk cost fallacy mode" 😄 and don't feel like rewriting the larger test I did in the other PR for now...but, we'll see how things go. Maybe we can drop it if it isn't holding its weight later.

Thank you and indeed, please do not drop nushell just on my feedback, I would love to hear more voices here and I personally would not mind trying something new! nushell does sound really interesting! So maybe the dual approach is fine for now (but maybe now I'm also overcompensating for pushing back earlier…).

[henrywang]

I like the nushell so much as I'm a big fan of shell. nutshell was one of my RHEL for Edge test tool candidates. But I finally choose the ansible to run interactive tests because I need to run test out of VM and tests need to be run in RHEL, CS, and Fedora.

[cgwalters]

I like the nushell so much as I'm a big fan of shell. nutshell was one of my RHEL for Edge test tool candidates.

Ah, interesting to hear!

But I finally choose the ansible to run interactive tests because I need to run test out of VM and tests need to be run in RHEL, CS, and Fedora.

To be clear, Ansible I think should definitely continue to play a role I think. What we're doing with it in provisioning is OK; but for the common use case of "mostly run arbitrary commands, potentially scraping output into JSON" it's also very verbose and in a lot of those cases is really arbitrary code masquerading as declarative YAML.

I would say for sure though we should continue to have Ansible be a gating test on the Fedora-bootc derivative integration testing and releasing.

[ckyrouac]

might want to add a note that this requires a rootful connection to podman, e.g. export CONTAINER_CONNECTION="podman-machine-default-root". Also might be good to note this will appear to hang while copying the bootc source into the container since the source directory is large.

[cgwalters]

Also might be good to note this will appear to hang while copying the bootc source into the container since the source directory is large.

See teemtee/tmt#3037