From e036472d6a2da2cc63db63ee94aae3ff7d9ac980 Mon Sep 17 00:00:00 2001 From: Seunghun Lee <45145778+seunghun1ee@users.noreply.github.com> Date: Fri, 11 Oct 2024 13:32:05 +0100 Subject: [PATCH 1/5] Update package update testing instructions (#830) Co-authored-by: Alex-Welsh --- doc/source/contributor/package-updates.rst | 43 ++++++++++++++-------- 1 file changed, 28 insertions(+), 15 deletions(-) diff --git a/doc/source/contributor/package-updates.rst b/doc/source/contributor/package-updates.rst index 9a6c74589..49e2b0852 100644 --- a/doc/source/contributor/package-updates.rst +++ b/doc/source/contributor/package-updates.rst @@ -63,18 +63,20 @@ The following steps describe the process to test the new package and container r Creating the multinode environments ----------------------------------- -There is a comprehensive guide to setting up a multinode environment with Terraform, found here: https://github.com/stackhpc/terraform-kayobe-multinode. There are some things to note: +The `Multinode deployment workflow `_ can be used to automatically test changes. + +To manually test the changes, there is a comprehensive guide to set up a Multinode environment with Terraform, found here: https://github.com/stackhpc/terraform-kayobe-multinode. There are some things to note: * OVN is enabled by default, you should override it under ``etc/kayobe/environments/ci-multinode/kolla.yml kolla_enable_ovn: false`` for the OVS multinode environment. -* Remember to set different vxlan_vnis for each. +* Remember to set a different ``vxlan_vni`` for each. -* Before starting any tests, run ``dnf distro-sync`` on each host to ensure you are using the same snapshots as in the release train. You can do this using the following commands: +* Before starting any tests, run ``dnf distro-sync -y`` on each host to ensure you are using the same snapshots as in the release train. Option ``-y`` is used to prevent hosts hang waiting for the confirmation input. You can do this using the following commands: .. code-block:: console - kayobe seed host command run -b --command "dnf distro-sync" - kayobe overcloud host command run -b --command "dnf distro-sync" + kayobe seed host command run -b --command "dnf distro-sync -y" + kayobe overcloud host command run -b --command "dnf distro-sync -y" * This may have installed a new kernel version. If so, you will need to reboot the overcloud hosts. You can check the installed kernels and the currently running kernel with the following commands. If the latest listed version is not running, you will need to reboot. @@ -85,7 +87,7 @@ There is a comprehensive guide to setting up a multinode environment with Terraf kayobe playbook run --limit seed,overcloud $KAYOBE_CONFIG_PATH/ansible/reboot.yml -* The tempest tests run automatically at the end of deploy-openstack.sh. If you have the time, it is worth fixing any failing tests you can so that there is greater coverage for the package updates. (Also remember to propose these fixes in the relevant repos where applicable.) +* The tempest tests run automatically at the end of the multinode deployment script. If you have the time, it is worth fixing any failing tests you can so that there is greater coverage for the package updates. (Also remember to propose these fixes in the relevant repos where applicable.) Upgrading host packages ----------------------- @@ -102,6 +104,7 @@ Bump the snapshot versions in /etc/yum/repos.d with: .. code-block:: console + kayobe seed host configure -t dnf -kt none kayobe overcloud host configure -t dnf -kt none Install new packages: @@ -112,22 +115,32 @@ Install new packages: Perform a rolling reboot of hosts: +.. note:: + In the Multinode environment, the seed-hypervisor cannot access control + plane instances with the Openstack client. To use Openstack client, connect + to the Seed instance via SSH first. For authentication, use scp to copy + ``public-openrc.sh`` to the Seed + .. code-block:: console - export ANSIBLE_SERIAL=1 - kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit controllers - kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit compute[0] + # Check your hypervisor hostname + (seed) openstack hypervisor list + + # Reboot controller instances and zeroth compute instance + (seed-hypervisor) export ANSIBLE_SERIAL=1 + (seed-hypervisor) kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit controllers + (seed-hypervisor) kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit compute[0] # Test live migration - openstack server create --image cirros --flavor m1.tiny --network external --hypervisor-hostname wallaby-pkg-refresh-ovs-compute-02.novalocal --os-compute-api-version 2.74 server1 - openstack server migrate --live-migration server1 - watch openstack server show server1 + (seed) openstack server create --image cirros --flavor m1.tiny --network external --hypervisor-hostname --os-compute-api-version 2.74 server1 + (seed) openstack server migrate --live-migration server1 + (seed) watch openstack server show server1 - kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit compute[1] + (seed-hypervisor) kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit compute[1] # Try and migrate back - openstack server migrate --live-migration server1 - watch openstack server show server1 + (seed) openstack server migrate --live-migration server1 + (seed) watch openstack server show server1 Upgrading containers within a release ------------------------------------- From d229d41a84011d8f650de7526a443b5a328c686c Mon Sep 17 00:00:00 2001 From: Jack Hodgkiss Date: Mon, 4 Nov 2024 15:06:53 +0000 Subject: [PATCH 2/5] fix!: manage the `physical` interface in `ci-aio` Treat the physical interface as network interface inside of the `ci-aio` environment. This is to ensure the network interface files are created correctly with the appropriate permissions otherwise we will lose connection after the DHCP lease expires. Also this approach enables us to include the interface within the firewall configuration ensuring we maintain SSH access through the firewall via this interface. --- .github/workflows/stackhpc-all-in-one.yml | 19 +++++-------------- .../environments/ci-aio/automated-setup.sh | 4 ---- .../environments/ci-aio/controllers.yml | 3 +++ .../group_vars/controllers/network-interfaces | 5 +++++ etc/kayobe/environments/ci-aio/networks.yml | 6 ++++++ 5 files changed, 19 insertions(+), 18 deletions(-) diff --git a/.github/workflows/stackhpc-all-in-one.yml b/.github/workflows/stackhpc-all-in-one.yml index 1909129ac..7f426d3f8 100644 --- a/.github/workflows/stackhpc-all-in-one.yml +++ b/.github/workflows/stackhpc-all-in-one.yml @@ -202,23 +202,14 @@ jobs: - name: Write Terraform network config run: | cat << EOF > etc/kayobe/environments/$KAYOBE_ENVIRONMENT/tf-networks.yml - - admin_oc_net_name: admin - admin_cidr: "{{ access_cidr.value }}" - admin_allocation_pool_start: 0.0.0.0 - admin_allocation_pool_end: 0.0.0.0 - admin_gateway: "{{ access_gw.value }}" - admin_bootproto: dhcp - admin_ips: + admin_oc_net_name: ethernet + ethernet_cidr: "{{ access_cidr.value }}" + ethernet_allocation_pool_start: 0.0.0.0 + ethernet_allocation_pool_end: 0.0.0.0 + ethernet_ips: controller0: "{{ access_ip_v4.value }}" EOF - - name: Write Terraform network interface config - run: | - cat << EOF > etc/kayobe/environments/$KAYOBE_ENVIRONMENT/inventory/group_vars/controllers/tf-network-interfaces - admin_interface: "{{ access_interface.value }}" - EOF - - name: Write all-in-one scenario config run: | cat << EOF > etc/kayobe/environments/$KAYOBE_ENVIRONMENT/zz-aio-scenario.yml diff --git a/etc/kayobe/environments/ci-aio/automated-setup.sh b/etc/kayobe/environments/ci-aio/automated-setup.sh index 84b9b5f09..5129db015 100644 --- a/etc/kayobe/environments/ci-aio/automated-setup.sh +++ b/etc/kayobe/environments/ci-aio/automated-setup.sh @@ -72,10 +72,6 @@ fi sudo ip l set dummy1 up sudo ip l set dummy1 master breth1 -if type apt; then - sudo cp /run/systemd/network/* /etc/systemd/network -fi - export KAYOBE_VAULT_PASSWORD=$(cat $BASE_PATH/vault-pw) pushd $BASE_PATH/src/kayobe-config source kayobe-env --environment ci-aio diff --git a/etc/kayobe/environments/ci-aio/controllers.yml b/etc/kayobe/environments/ci-aio/controllers.yml index b34536705..b67cb68f1 100644 --- a/etc/kayobe/environments/ci-aio/controllers.yml +++ b/etc/kayobe/environments/ci-aio/controllers.yml @@ -6,6 +6,9 @@ # to setup the Kayobe user account. Default is {{ os_distribution }}. controller_bootstrap_user: "{{ os_distribution if os_distribution == 'ubuntu' else 'cloud-user' }}" +controller_extra_network_interfaces: + - ethernet + # Controller lvm configuration. See intentory/group_vars/controllers/lvm.yml # for the exact configuration. controller_lvm_groups: diff --git a/etc/kayobe/environments/ci-aio/inventory/group_vars/controllers/network-interfaces b/etc/kayobe/environments/ci-aio/inventory/group_vars/controllers/network-interfaces index 2f8d30103..85f318f42 100644 --- a/etc/kayobe/environments/ci-aio/inventory/group_vars/controllers/network-interfaces +++ b/etc/kayobe/environments/ci-aio/inventory/group_vars/controllers/network-interfaces @@ -2,6 +2,11 @@ ############################################################################### # Network interface definitions for the controller group. +# Ethernet interface is the `primary` or `physical` interface associated +# with the instance that the AIO deployment runs inside of. It is the interface used +# to reach the instance. +ethernet_interface: "{{ ansible_facts['default_ipv4']['interface'] }}" + # Controller interface on all-in-one network. aio_interface: breth1 # Use dummy1 if it exists, otherwise the bridge will have no ports. diff --git a/etc/kayobe/environments/ci-aio/networks.yml b/etc/kayobe/environments/ci-aio/networks.yml index 216696eaa..f09e1cbc2 100644 --- a/etc/kayobe/environments/ci-aio/networks.yml +++ b/etc/kayobe/environments/ci-aio/networks.yml @@ -80,6 +80,12 @@ cleaning_net_name: aio ############################################################################### # Network definitions. +# This network is required to be defined within `ci-aio` environment to ensure that +# the network interface files are created appropriately and to provide easy inclusion +# within the firewall configuration. +ethernet_bootproto: dhcp +ethernet_zone: trusted + # All-in-one network. aio_cidr: 192.168.33.0/24 aio_allocation_pool_start: 192.168.33.3 From f4f88993ee936bb8eccbf8c5d1476f22387f11be Mon Sep 17 00:00:00 2001 From: Jack Hodgkiss Date: Mon, 4 Nov 2024 15:06:53 +0000 Subject: [PATCH 3/5] fix!: manage the `physical` interface in `ci-aio` Treat the physical interface as network interface inside of the `ci-aio` environment. This is to ensure the network interface files are created correctly with the appropriate permissions otherwise we will lose connection after the DHCP lease expires. Also this approach enables us to include the interface within the firewall configuration ensuring we maintain SSH access through the firewall via this interface. --- .github/workflows/stackhpc-all-in-one.yml | 19 +++++-------------- .../environments/ci-aio/automated-setup.sh | 4 ---- .../environments/ci-aio/controllers.yml | 3 +++ .../group_vars/controllers/network-interfaces | 5 +++++ etc/kayobe/environments/ci-aio/networks.yml | 6 ++++++ 5 files changed, 19 insertions(+), 18 deletions(-) diff --git a/.github/workflows/stackhpc-all-in-one.yml b/.github/workflows/stackhpc-all-in-one.yml index 45a33f47d..0ec9b6d34 100644 --- a/.github/workflows/stackhpc-all-in-one.yml +++ b/.github/workflows/stackhpc-all-in-one.yml @@ -225,23 +225,14 @@ jobs: - name: Write Terraform network config run: | cat << EOF > etc/kayobe/environments/$KAYOBE_ENVIRONMENT/tf-networks.yml - - admin_oc_net_name: admin - admin_cidr: "{{ access_cidr.value }}" - admin_allocation_pool_start: 0.0.0.0 - admin_allocation_pool_end: 0.0.0.0 - admin_gateway: "{{ access_gw.value }}" - admin_bootproto: dhcp - admin_ips: + admin_oc_net_name: ethernet + ethernet_cidr: "{{ access_cidr.value }}" + ethernet_allocation_pool_start: 0.0.0.0 + ethernet_allocation_pool_end: 0.0.0.0 + ethernet_ips: controller0: "{{ access_ip_v4.value }}" EOF - - name: Write Terraform network interface config - run: | - cat << EOF > etc/kayobe/environments/$KAYOBE_ENVIRONMENT/inventory/group_vars/controllers/tf-network-interfaces - admin_interface: "{{ access_interface.value }}" - EOF - - name: Write all-in-one scenario config run: | cat << EOF > etc/kayobe/environments/$KAYOBE_ENVIRONMENT/zz-aio-scenario.yml diff --git a/etc/kayobe/environments/ci-aio/automated-setup.sh b/etc/kayobe/environments/ci-aio/automated-setup.sh index f7d34db86..82e642e2b 100644 --- a/etc/kayobe/environments/ci-aio/automated-setup.sh +++ b/etc/kayobe/environments/ci-aio/automated-setup.sh @@ -72,10 +72,6 @@ fi sudo ip l set dummy1 up sudo ip l set dummy1 master breth1 -if type apt; then - sudo cp /run/systemd/network/* /etc/systemd/network -fi - export KAYOBE_VAULT_PASSWORD=$(cat $BASE_PATH/vault-pw) pushd $BASE_PATH/src/kayobe-config source kayobe-env --environment ci-aio diff --git a/etc/kayobe/environments/ci-aio/controllers.yml b/etc/kayobe/environments/ci-aio/controllers.yml index b34536705..b67cb68f1 100644 --- a/etc/kayobe/environments/ci-aio/controllers.yml +++ b/etc/kayobe/environments/ci-aio/controllers.yml @@ -6,6 +6,9 @@ # to setup the Kayobe user account. Default is {{ os_distribution }}. controller_bootstrap_user: "{{ os_distribution if os_distribution == 'ubuntu' else 'cloud-user' }}" +controller_extra_network_interfaces: + - ethernet + # Controller lvm configuration. See intentory/group_vars/controllers/lvm.yml # for the exact configuration. controller_lvm_groups: diff --git a/etc/kayobe/environments/ci-aio/inventory/group_vars/controllers/network-interfaces b/etc/kayobe/environments/ci-aio/inventory/group_vars/controllers/network-interfaces index 2f8d30103..85f318f42 100644 --- a/etc/kayobe/environments/ci-aio/inventory/group_vars/controllers/network-interfaces +++ b/etc/kayobe/environments/ci-aio/inventory/group_vars/controllers/network-interfaces @@ -2,6 +2,11 @@ ############################################################################### # Network interface definitions for the controller group. +# Ethernet interface is the `primary` or `physical` interface associated +# with the instance that the AIO deployment runs inside of. It is the interface used +# to reach the instance. +ethernet_interface: "{{ ansible_facts['default_ipv4']['interface'] }}" + # Controller interface on all-in-one network. aio_interface: breth1 # Use dummy1 if it exists, otherwise the bridge will have no ports. diff --git a/etc/kayobe/environments/ci-aio/networks.yml b/etc/kayobe/environments/ci-aio/networks.yml index 216696eaa..f09e1cbc2 100644 --- a/etc/kayobe/environments/ci-aio/networks.yml +++ b/etc/kayobe/environments/ci-aio/networks.yml @@ -80,6 +80,12 @@ cleaning_net_name: aio ############################################################################### # Network definitions. +# This network is required to be defined within `ci-aio` environment to ensure that +# the network interface files are created appropriately and to provide easy inclusion +# within the firewall configuration. +ethernet_bootproto: dhcp +ethernet_zone: trusted + # All-in-one network. aio_cidr: 192.168.33.0/24 aio_allocation_pool_start: 192.168.33.3 From b9fde8b40708f685499c95a1f9bd0ea172258edb Mon Sep 17 00:00:00 2001 From: Pierre Riteau Date: Wed, 20 Nov 2024 23:00:02 +0100 Subject: [PATCH 4/5] docs: fix link to release train page --- doc/source/configuration/release-train.rst | 2 ++ doc/source/operations/upgrading.rst | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/doc/source/configuration/release-train.rst b/doc/source/configuration/release-train.rst index 0d62fadfd..f77109aff 100644 --- a/doc/source/configuration/release-train.rst +++ b/doc/source/configuration/release-train.rst @@ -1,3 +1,5 @@ +.. _stackhpc_release_train: + ====================== StackHPC Release Train ====================== diff --git a/doc/source/operations/upgrading.rst b/doc/source/operations/upgrading.rst index 218c39bb1..b8d468b44 100644 --- a/doc/source/operations/upgrading.rst +++ b/doc/source/operations/upgrading.rst @@ -363,9 +363,8 @@ To upgrade the Ansible control host: Syncing Release Train artifacts ------------------------------- -New `StackHPC Release Train <../configuration/release-train>`_ content should -be synced to the local Pulp server. This includes host packages (Deb/RPM) and -container images. +New :ref:`stackhpc_release_train` content should be synced to the local Pulp +server. This includes host packages (Deb/RPM) and container images. .. _sync-rt-package-repos: From 6975cb113330edb28d0dae8839e8ffb89f336a99 Mon Sep 17 00:00:00 2001 From: Pierre Riteau Date: Fri, 22 Nov 2024 09:47:22 +0100 Subject: [PATCH 5/5] Bump kayobe-automation This is required to fix some CI jobs [1]. [1] https://github.com/stackhpc/kayobe-automation/pull/69 --- .automation | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.automation b/.automation index b00f285be..a7de3abb3 160000 --- a/.automation +++ b/.automation @@ -1 +1 @@ -Subproject commit b00f285be240e34c643c4bd93a877e56587f71fa +Subproject commit a7de3abb3f0bf529e78c4ba9ad1cbe26d356dd3b