This project provides a set of scripts intended to accelerate the snappy validation process over the diferent platforms and devices supported.
The scripts are ready to run spread tests for the different scenarios on actual devices, test flinger devices, local vms and to create vms with ubuntu core inside test flinger machines.
The default configuration of the project is ready to run beta validation process.
Smoke test suite is used to validate a core snap is working on an specific channel.
The initial use case for this test suite is to validate the core snap when it is promoted to stable release, but it can be used for validating other channels too.
To run this test suite it is needed to specify the core version and channel. The following line shows how to run it:
SPREAD_CORE_VERSION=2.37.4 SPREAD_CORE_CHANNEL=stable spread google:tasks/smoke/
The current systems are using base images with no dependencies installed on google compute engine.
Next step is to extend the suite to run the same tests but on devices.
In this section it is explained how to run beta validation by using this project. To complete beta validation all the following tests have to be executed on dragonboard, rpi2, rpi3, core64 and core32.
By running the following lines, the script will create the images for beta validation for all the supported devices.
git clone https://github.com/sergiocazzolato/validator.git
cd validator
sudo ./create.sh beta <core-number> <snaps-by-channel> <extra-snaps> <models>
with . core-number = Ubuntu Core system version, currently supported: 16 or 18 . snaps-by-channel = select specific snap by channel to install, for example core=edge . extra-snaps = select local snaps to be installed during the image creation, for example core18_782.snap . models: if not defined then images for all the models are created. Otherwise one of these can be selected: . dragonboard . pc-amd64 . pc-i386 . pi3 . pi2
This example shows how to create a core16 image using stable channel but core from beta with amd64 architecture
sudo ./create.sh stable 16 "core=beta" "" pc-amd64
Download the core 16 images from: http://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ Download the core 18 images from: http://cdimage.ubuntu.com/ubuntu-core/18/stable/current/ Download the core 20 images from: http://cdimage.ubuntu.com/ubuntu-core/20/stable/current/
It is possible to use older images too to validate the refresh scenario.
sudo dd if=<PAT_TO_IMG> of=/dev/mmcblk0 bs=4M oflag=sync status=noxfer
old kvm:
amd64: kvm -snapshot -smp 2 -m 1500 -redir tcp:8022::22 -nographic -serial mon:stdio <PATH_TO_VM_IMAGE>
i386: kvm -snapshot -smp 2 -m 1500 -redir tcp:8023::22 -nographic -serial mon:stdio <PATH_TO_VM_IMAGE>
new kvm:
amd64: sudo kvm -snapshot -smp 2 -m 1500 -net nic,model=virtio -net user,hostfwd=tcp::8022-:22 -nographic -serial mon:stdio <PATH_TO_VM_IMAGE>
i386: sudo kvm -snapshot -smp 2 -m 1500 -net nic,model=virtio -net user,hostfwd=tcp::8023-:22 -nographic -serial mon:stdio <PATH_TO_VM_IMAGE>
amd64: sudo kvm -snapshot -smp 2 -m 2000 -net nic,model=virtio -net user,hostfwd=tcp::8022-:22 -serial mon:stdio <PATH_TO_VM_IMAGE>
i386: sudo kvm -snapshot -smp 2 -m 2000 -net nic,model=virtio -net user,hostfwd=tcp::8023-:22 -serial mon:stdio <PATH_TO_VM_IMAGE>
Note: it could be needed to provide initial entropy by moving the mouse and using the keyboard in the kvm window.
amd64:
sudo cp /usr/share/OVMF/OVMF_VARS.ms.fd .
sudo qemu-system-x86_64 -smp 2 -m 4096 -snapshot -machine ubuntu-q35,accel=kvm -global ICH9-LPC.disable_s3=1 -netdev user,id=mynet0,hostfwd=tcp::8022-:22 -device virtio-net-pci,netdev=mynet0 -drive file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=./OVMF_VARS.ms.fd,if=pflash,format=raw,unit=1 -drive file=<PATH_TO_VM_IMAGE>,cache=none,format=raw,id=disk1,if=none -device virtio-blk-pci,drive=disk1,bootindex=1
Note: it is needed to install ovmf package as dependency
sudo screen /dev/ttyUSB0 115200
To complete beta validation is requested to run all the following scenario. For all of them, it is needed to save the results, analize and track test failures:
Db: DEVICE_IP=<DEVICE_IP> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_db
Amd64: DEVICE_PORT=<DEVICE_PORT> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_amd64
I386: DEVICE_PORT=<DEVICE_PORT> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_i386
Pi2: DEVICE_IP=<DEVICE_IP> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_pi
Pi3: DEVICE_IP=<DEVICE_IP> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_pi
Db: DEVICE_IP=<DEVICE_IP> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_db_refresh
Amd64: DEVICE_PORT=<DEVICE_PORT> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_amd64_refresh
I386: DEVICE_PORT=<DEVICE_PORT> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_i386_refresh
Pi2: DEVICE_IP=<DEVICE_IP> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_pi_refresh
Pi3: DEVICE_IP=<DEVICE_IP> DEVICE_USER=<DEVICE_USER> BRANCH=<BRANCH> scripts/run_external_device.sh dev_snapd_pi_refresh
Db: DEVICE_USER=<DEVICE_USER> WIFI_SSID=<WIFI_SSID> WIFI_PASSWORD=<WIFI_PASSWORD> DEVICE_IP=<DEVICE_IP> scripts/run_external_device.sh dev_cconf_db
Amd64: DEVICE_PORT=<DEVICE_PORT> DEVICE_USER=<DEVICE_USER> ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh tf_cconf_amd64
I386: DEVICE_PORT=<DEVICE_PORT> DEVICE_USER=<DEVICE_USER> ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh tf_cconf_i386
Pi2: DEVICE_USER=<DEVICE_USER> DEVICE_IP=<DEVICE_IP> scripts/run_external_device.sh dev_cconf_pi2
Pi3: DEVICE_USER=<DEVICE_USER> WIFI_SSID=<WIFI_SSID> WIFI_PASSWORD=<WIFI_PASSWORD> DEVICE_IP=<DEVICE_IP> scripts/run_external_device.sh dev_cconf_pi3
In order to run console-conf tests, it is needed to run console-conf manually once in order to get the device ip to use it on the tests. Make sure the network conections are setup for different devices, i.e. on pi3 the ethernet and wifi need to be setup initially.
16.04-64: BRANCH=<BRANCH> scripts/run_google.sh goo_snapd_amd64_core_revert
It is occasionally useful to re-run a specific test (or set of tests), or skip a test. This is supported by SPREAD_TESTS and SKIP_TESTS variables passed to the scripts along with the standard variables, e.g. SPREAD_TESTS=external:ubuntu-core-16-arm-64:tests/main/interfaces-hostname-control,external:ubuntu-core-16-arm-64:tests/main/snap-logs SKIP_TESTS=tests/main/revert-devmode
<DEVICE_IP> = The ip of the device that's gonna be tested, in case there are two ips (wifi and lan), it is more reliable to use the lan.
<DEVICE_USER> = The user for whom the device to test is registered.
<DEVICE_PORT> = The port used to ssh to the device, it is mostly used when the device is a local vm.
<BRANCH> = The branch corresponding to the release to validate. Note: beta and edge can be used when testing core and core18.
<WIFI_SSID> = The ssid for the wifi that is gonna used to validate
<WIFI_PASSWORD> = The password to connect the device to the wifi
In case of dragonboard it can be tested on testflinger. For that it is needed to install the testflinger-cli snap and run the tests connected to the vpn. See the examples section to see how to run it.
The following section shows the examples that are used to execution beta validation for core 16 and core 18
DEVICE_USER=sergio-j-cazzolato DEVICE_PORT=8022 BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_amd64
DEVICE_USER=sergio-j-cazzolato DEVICE_PORT=8022 BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_amd64_18
DEVICE_USER=sergio-j-cazzolato DEVICE_PORT=8022 BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_amd64_20
DEVICE_USER=sergio-j-cazzolato DEVICE_PORT=8023 BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_i386
DEVICE_USER=sergio-j-cazzolato DEVICE_PORT=8023 BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_i386_18
DEVICE_USER=sergio-j-cazzolato DEVICE_PORT=8023 BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_i386_20
DEVICE_IP=10.42.0.67 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_pi
DEVICE_IP=10.42.0.67 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_pi_18
DEVICE_IP=10.42.0.67 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_pi_20
DEVICE_IP=192.168.1.8 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_db
DEVICE_IP=192.168.1.8 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_db_18
DEVICE_IP=192.168.1.8 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_db_20
BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_device.sh tf_snapd_db
BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_device.sh tf_snapd_cm3
IMAGE_URL=http://<GCE-IP>/validator/images/pc-amd64-16-beta/pc.img BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_vm.sh tf_snapd_amd64
IMAGE_URL=http://<GCE-IP>/validator/images/pc-amd64-18-beta/pc.img BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_vm.sh tf_snapd_amd64_18
IMAGE_URL=http://<GCE-IP>/validator/images/pc-amd64-18-beta/pc.img BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_vm.sh tf_snapd_amd64_20
IMAGE_URL=http://<GCE-IP>/validator/images/pc-i386-16-beta/pc.img BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_vm.sh tf_snapd_i386
IMAGE_URL=http://<GCE-IP>/validator/images/pc-i386-18-beta/pc.img BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_vm.sh tf_snapd_i386_18
IMAGE_URL=http://<GCE-IP>/validator/images/pc-i386-18-beta/pc.img BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_vm.sh tf_snapd_i386_20
DEVICE_PORT=8022 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_amd64_refresh
DEVICE_PORT=8022 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_amd64_refresh_18
DEVICE_PORT=8022 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_amd64_refresh_20
DEVICE_PORT=8023 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_i386_refresh
DEVICE_PORT=8023 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_i386_refresh_18
DEVICE_PORT=8023 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_i386_refresh_20
DEVICE_IP=10.42.0.67 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_pi_refresh
DEVICE_IP=10.42.0.67 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_pi_refresh_18
DEVICE_IP=10.42.0.67 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_pi_refresh_20
DEVICE_IP=192.168.1.8 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_db_refresh
DEVICE_IP=192.168.1.8 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_db_refresh_18
DEVICE_IP=192.168.1.8 DEVICE_USER=sergio-j-cazzolato BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_snapd_db_refresh_20
BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_device.sh tf_snapd_db_refresh
BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_device.sh tf_snapd_cm3_refresh
BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_vm.sh tf_snapd_amd64_refresh
BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_tf_vm.sh tf_snapd_i386_refresh
BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_google.sh goo_snapd_amd64_core_revert
DEVICE_USER=sergio-j-cazzolato DEVICE_IP=localhost DEVICE_PORT=8022 ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh tf_cconf_amd64
DEVICE_USER=sergio-j-cazzolato DEVICE_IP=localhost DEVICE_PORT=8023 ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh tf_cconf_i386
DEVICE_USER=sergio-j-cazzolato DEVICE_IP=10.42.0.24 ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_cconf_pi2
DEVICE_USER=sergio-j-cazzolato WIFI_SSID=put-your-ssid WIFI_PASSWORD=put-the-pwd DEVICE_IP=192.168.1.6 ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_cconf_pi3
DEVICE_USER=sergio-j-cazzolato WIFI_SSID=put-your-ssid WIFI_PASSWORD=put-the-pwd DEVICE_IP=192.168.1.8 ~/workspace/snappy-qa-jobs/scripts/run_external_device.sh dev_cconf_db
BRANCH=beta ~/workspace/snappy-qa-jobs/scripts/run_google_sru.sh goo_snapd_sru_validation
DEVICE_USER=ubuntu DEVICE_PASS=pass DEVICE_IP=192.168.1.8 ~/workspace/snappy-qa-jobs/scripts/run_external_device_sru.sh dev_snapd_sru_validation
This promotion can be done manually through the snap store site or by using the validator project which automates the process.
git clone https://github.com/sergiocazzolato/validator.git
./validator/promote.sh <SNAP_NAME> <FROM_CHANNEL> <TO_CHANNEL>
The promote.sh script will promote the different revisions (all the architectures) for the snap <SNAP_NAME> from the channel <FROM_CHANNEL> to the channel <TO_CHANNEL>
These are the typical examples:
./validator/promote.sh core beta candidate
./validator/promote.sh snapd beta candidate
For amd64 architecture due to the number of devices affected the progressive release is executed. The idea of this is to deliver the core snap to a predefined number of devices.
For all the architectures but pc-amd64 the release can be done by runnign 'snapdcraft release core <REV_NUM> stable' In case of architecture pc-amd64 a progressive release is nedeed. For that the following steps need to be executed in that order, repeating the step 2 with different percentages until arrive to 100.
Steps:
- surl -s production -a sca-production -e --force -p package_access -p package_release
- surl -a sca-production -e -X POST -d '{"name": "core", "revision": <REV_NUM>, "channels": ["stable"], "progressive": {"key": "progressive-core-16-", "percentage": , "paused": false}}' https://dashboard.snapcraft.io/dev/api/snap-release/
- snapcraft release core <REV_NUM> stable
With:
<EMAIL> = The email which is used to login in the snap store
<REV_NUM> = The rev number for the snap to do progressive release
<VERVION> = Id used to identify the progressive release
<PERCENTAGE> = % of devices which will get the new release. The % should be increase by 25 every 6 hours when the snap store team validates the release is going well. The values for this should be: 25, 50, 75 and 100.
This promotion can be done manually through the snap store site or by using the validator project which automates the process.
./validator/promote.sh snapd candidate stable