SSDF PW5.2: Documentation of testing for infra tools

[Haroon-Khel]

PW.5.2: Conduct unit testing
1. Adoptium maintain process documentation detailing their unit testing process.

Though for some reason this ssdf work item is not mentioned in any of the epics, I thought it would be most appropriate to use adoptium/adoptium#124 as a reference.

This is issue will be used to discuss the documentation of the testing being done for the tools used in this repository (any current documentation and ideas/improvements to the documentation).

[Haroon-Khel]

• VPC (Vagrant Playbook Check) https://ci.adoptopenjdk.net/job/VagrantPlaybookCheck
  • VMs of various operating systems to test changes made to the windows and linux playbooks.
  • Supported OSs include
    • Windows 2012
    • Debian8
    • Debian10
    • Ubuntu1604
    • Ubuntu1804
    • Ubuntu2004
    • Ubuntu2104
    • CentOS6
    • CentOS7
    • CentOS8
    • Solaris10
• QPC (Qemu Playbook Check) https://ci.adoptopenjdk.net/job/QEMUPlaybookCheck/
  • VMs of various operating systems and architecture combinations, to test changes made to the linux playbook
  • Supported OS/arch combinations include
    • Ubuntu1804 PPC64LE
    • Ubuntu1804 s390x
    • Ubuntu1804 aarch64
    • Debian11 RISCV
    • Debian10 aarch64
    • Debian8 arm32

[Haroon-Khel]

Github workflows

• VPC solaris
  • Uses https://github.com/adoptium/infrastructure/blob/master/.github/workflows/build_vagrant.yml
  • Spins up a Solaris 10 vagrant box, runs the playbook on it
• MacOS 10.15
  • Uses https://github.com/adoptium/infrastructure/blob/master/.github/workflows/build_mac.yml
  • Runs the playbook on a MacOS 10.15 image
• Docker build image, uses docker files here, workflow file here
  • Builds a centos 6 docker image, runs the playbook on it, pushes the image to dockerhub
  • Builds a alpine 3 docker image, runs the playbook on it, pushes the image to dockerhub
• Windows Playbook Checker workflow
  • Workflow file https://github.com/adoptium/infrastructure/blob/master/.github/workflows/build_wsl.yml
  • Builds and checks pr on windows 2019 and 2022 vm
• QEMU-Playbook-Check pr workflow
  • Workflow file https://github.com/adoptium/infrastructure/blob/master/.github/workflows/build_qemu.yml
  • When the QEMU-Playbook-Check label is added to a pr, the pr changes are run through aarch64, s390x and ppc64le vms.
  • Similar functionality to https://ci.adoptopenjdk.net/job/QEMUPlaybookCheck/ but in a github workflow

[Haroon-Khel]

Docker Static Workflows

• Uses https://github.com/adoptium/infrastructure/blob/master/.github/workflows/check_dockerstatic.yml
• If changes are made to https://github.com/adoptium/infrastructure/tree/master/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/DockerStatic/Dockerfiles, changes are tested on our alpine, ubuntu, centos and fedora docker static containers

[Haroon-Khel]

AWX

• AWX server hosted at https://awx2.adoptopenjdk.net/
• Infrastructure members use the service to test any new playbook changes by running their changes onto our infrastructure machines
• Not automated like the pr workflows

[sxa]

@Haroon-Khel Can we get this moved into a file in the repository somewhere? Once that's done we can close it off.

I was initially thinking about having it directly in the README.md but since there's quite a lot perhaps having a separate "docs/Testing.md" that's referenced from other places would be preferable. We should also include information on the linter checks (I believe this is covered in @steelhead31's document referenced in #2971 (comment) but I'm not sure if that's public somewhere)

I'll probably add a link from my PR with updates to the CONTRIBUTING.md once this is done :-)

[Haroon-Khel]

Closing this as #3291 is merged

[sxa]

Noting that we will expect to be looking at checks using the semgrep tool as an extra set of testing that we can perform at the project.