ci: use Testing Farm instead of GH Actions + MacOS + Vagrant

The current solution to run a VM on MacOS shared runners using Vagrant
is becoming very unreliable and almost always breaks. Replace it with
Testing Farm [1], utilizing the "Schedule tests on Testing Farm" GH
Action [2].

Advantages:
 - more reliable
 - allows us to test also on aarch64
 - currently no usage limits for public projects (will likely change)

Disadvantages:
 - requires an API key to be stored in the project's secrets (under the
   name TESTING_FARM_API_TOKEN)
 - GitHub won't allow the target project's secrets to be used when
   running a workflow on a pull request from a fork, therefore PRs won't
   be automatically tested unless the contributor sets their own API key
   in their fork
 - only people with a Fedora account in the fedora-contributor group can
   currently obtain an API key on their own (others would need to ask
   for it via email)
 - no real-time view of test progress (may become available in the
   future via the artifacts view)

The new CI runs the testsuite on a similar testing matrix as the old
one, although it only tests on the latest Fedora version and
additionally tests on the aarch64 architecture. It also runs the NFS
tests (./tools/nfs.sh), which the old one didn't.

[1] https://testing-farm.io/
[2] https://github.com/marketplace/actions/schedule-tests-on-testing-farm

Signed-off-by: Ondrej Mosnacek <[email protected]>

.fmf/version

@@ -0,0 +1 @@
+1

.github/workflows/checks.yml

@@ -1,5 +1,6 @@
 name: checks
 on: [push, pull_request]
+
 jobs:
   style-check:
     runs-on: ubuntu-latest
@@ -10,43 +11,19 @@ jobs:
       - uses: actions/checkout@v4
       - run: sudo chown $(id -u):$(id -g) .
       - run: tools/check-syntax -f && git diff --exit-code
-  fedora-test:
-    runs-on: macos-12
+  run:
+    runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
         domain: [unconfined_t, sysadm_t]
-        env:
-          - { version: 38, kernel: default }
-          - { version: 39, kernel: default }
-          - { version: 39, kernel: secnext }
-    env:
-      FEDORA_VERSION: ${{ matrix.env.version }}
-      KERNEL_TYPE: ${{ matrix.env.kernel }}
-      ROOT_DOMAIN: ${{ matrix.domain }}
+        arch: [x86_64, aarch64]
+        kernel: [latest, secnext]
     steps:
-      - name: Install GNU coreutils
-        run: brew install coreutils
-      - uses: actions/checkout@v4
-      # macOS sometimes allows symlinks to have permissions other than 777,
-      # so change all symlink perms to match the Linux convention. Otherwise
-      # the rsync run by Vagrant will complain that it can't copy over the
-      # perms.
-      - name: Fix symlink permissions
-        run: find . -type link -exec chmod -h 777 \{\} \;
-      - name: Treat compiler warnings as errors
-        run: sed -i '' 's/-Wall/-Wall -Werror/' tests/Makefile
-      - name: Create a Vagrant VM
-        run: vagrant up
-      - name: Wait for the machine to come up if rebooting (max 5m)
-        run: gtimeout 5m "$SHELL" -c 'while ! vagrant ssh -- true; do sleep 1; done'
-      - name: Show Vagrant VM details
-        run: |
-          vagrant ssh -- uname -a
-          vagrant ssh -- cat /proc/cmdline
-      - name: Run SELinux testsuite
-        run: vagrant ssh -- sudo make -C /root/testsuite test
-      - name: Check unwanted denials
-        run: vagrant ssh -- '! sudo ausearch -m avc -i </dev/null | grep ${{ matrix.domain }}'
-      - name: Check .gitignore coverage
-        run: test "$(vagrant ssh -- sudo git -C /root/testsuite ls-files -o --exclude-standard | wc -l)" -eq 0
+      - name: Schedule test on Testing Farm
+        uses: sclorg/testing-farm-as-github-action@main
+        with:
+          api_key: ${{ secrets.TESTING_FARM_API_TOKEN }}
+          arch: ${{ matrix.arch }}
+          variables: STS_ROOT_DOMAIN=${{ matrix.domain }}; STS_KERNEL=${{ matrix.kernel }}
+          tmt_plan_filter: tag:ci

tmt/README.md

@@ -0,0 +1,19 @@
+# TMT test plans for selinux-testsuite
+
+This directory contains basic "test plans" for running the selinux-testsuite via the [TMT tool](https://tmt.readthedocs.io/en/stable/). They are primarily intended for the GitHub-Actions-driven CI, but they can be also used directly through `tmt`:
+
+```bash
+tmt run [-e STS_ROOT_DOMAIN=...] [-e STS_KERNEL=...] \
+    plans -f 'tag:-ci' --all provision -h ...
+```
+
+See `tmt run provision --help` for information about possible provisioning methods (most useful are `local`, `connect`, or `virtual.testcloud`).
+
+Possible values for the `STS_ROOT_DOMAIN` env parameter are:
+* `unconfined_t` - run the testsuite as an unconfined root.
+* `sysadm_t` - run the testsuite as a `sysadm_u:sysadm_r:sysadm_t:...` root.
+
+Possible values for the `STS_KERNEL` env parameter are:
+* `default` - try to use the kernel currently booted on the test machine.
+* `latest` - update to the latest kernel available in the repos and boot it.
+* `secnext` - install the "secnext" kernel from https://repo.paul-moore.com/ and boot it.

tmt/plans.fmf

@@ -0,0 +1,14 @@
+/run:
+  summary: Run the testsuite
+  discover:
+    how: fmf
+    filter: tag:-ci
+  execute:
+    how: tmt
+/run-ci:
+  summary: Run the CI for the testsuite
+  tag: [ci]
+  discover:
+    how: fmf
+  execute:
+    how: tmt

tmt/tests.fmf

@@ -0,0 +1,133 @@
+# can't do this in the prepare step because of
+# https://github.com/teemtee/tmt/issues/902
+/prepare-system:
+  summary: Prepare the system for testing
+  order: 1
+  require: policycoreutils-python-utils
+  test: |
+    set -ex
+
+    if [ "$TMT_REBOOT_COUNT" -eq 0 ]; then
+      reboot=0
+
+      case "$STS_ROOT_DOMAIN" in
+      unconfined_t|'')
+        ;;
+      sysadm_t)
+        semanage boolean --modify --on ssh_sysadm_login
+        semanage login --modify -s sysadm_u root
+
+        reboot=1
+        ;;
+      *)
+        echo "Invalid STS_ROOT_DOMAIN value!"
+        exit 1
+        ;;
+      esac
+
+      case "$STS_KERNEL" in
+      default|'')
+        dnf install -y kernel-modules-$(uname -r) kernel-devel-$(uname -r)
+        ;;
+      latest)
+        dnf install -y kernel-modules kernel-devel
+        ;;
+      secnext)
+        dnf install -y --disablerepo testing-farm-tag-repository \
+          --nogpgcheck --releasever rawhide \
+          --repofrompath 'kernel-secnext,https://repo.paul-moore.com/rawhide/$basearch' \
+          kernel-modules kernel-devel
+        ;;
+      *)
+        echo "Invalid STS_KERNEL value!"
+        exit 1
+        ;;
+      esac
+
+      if [ "$(grubby --default-kernel)" != "/boot/vmlinuz-$(uname -r)" ]; then
+        reboot=1
+      fi
+
+      if [ "$reboot" -ne 0 ]; then
+        tmt-reboot
+      fi
+    fi
+
+    # reset the audit log
+    :>/var/log/audit.log
+    rm -f /var/log/audit.log.*
+/machine-info:
+  summary: Show machine info
+  order: 2
+  test: |
+    set -x
+
+    uname -r
+    id -Z
+    rpm -q libselinux
+    rpm -q selinux-policy
+    lscpu
+/run:
+  summary: Run the testsuite
+  order: 3
+  duration: 20m
+  require:
+    - make
+    - perl-Test
+    - perl-Test-Harness
+    - perl-Test-Simple
+    - perl-lib
+    - selinux-policy-devel
+    - gcc
+    - libselinux-devel
+    - net-tools
+    - netlabel_tools
+    - iptables
+    - nftables
+    - lksctp-tools-devel
+    - attr
+    - libbpf-devel
+    - keyutils-libs-devel
+    - quota
+    - xfsprogs-devel
+    - libuuid-devel
+    - e2fsprogs
+    - jfsutils
+    - dosfstools
+    - rdma-core-devel
+  test: make -C .. test
+/run-nfs:
+  summary: Run the NFS tests
+  order: 3
+  duration: 25m
+  require: nfs-utils
+  test: env -C .. bash -x ./tools/nfs.sh
+/avc-check:
+  summary: Check unwanted denials
+  order: 4
+  tag: [ci]
+  test: '! ausearch -m avc -i </dev/null | grep "$STS_ROOT_DOMAIN"'
+/gitignore-coverage:
+  summary: Check .gitignore coverage
+  order: 4
+  tag: [ci]
+  require: git-core
+  test: |
+    # exclude tmt's litter
+    mkdir -p "$HOME/.config/git"
+    echo "tmt-*" >"$HOME/.config/git/ignore"
+
+    git -C .. ls-files -o --exclude-standard
+    test "$(git -C .. ls-files -o --exclude-standard | wc -l)" -eq 0
+/unprepare:
+  summary: Undo the preparation
+  order: 5
+  require: policycoreutils-python-utils
+  test: |
+    set -ex
+
+    make -C ../policy unload || true
+    if [ "$STS_ROOT_DOMAIN" = sysadm_t ]; then
+      semanage boolean --modify --off ssh_sysadm_login
+      semanage login --modify -s unconfined_u root
+    fi