From 06800ad10a5f362979e8e8b7d6aa8db0ec1f3f33 Mon Sep 17 00:00:00 2001 From: Patryk Matuszak Date: Tue, 9 Jul 2024 12:52:22 +0200 Subject: [PATCH] move details of microshift-tuned to dedicated section --- .../low-latency-workloads-on-microshift.md | 61 +++++++++++++------ 1 file changed, 41 insertions(+), 20 deletions(-) diff --git a/enhancements/microshift/low-latency-workloads-on-microshift.md b/enhancements/microshift/low-latency-workloads-on-microshift.md index 0f1110e4c4..d994a201ba 100644 --- a/enhancements/microshift/low-latency-workloads-on-microshift.md +++ b/enhancements/microshift/low-latency-workloads-on-microshift.md @@ -106,14 +106,8 @@ Workflow consists of two parts: 1. User builds the blueprint 1. User deploys the commit / installs the system. 1. System boots -1. `microshift-tuned.service` starts (after `tuned.service`, before `microshift.service`): - - Compares active profile with requested profile - - If requested profile is already active: - - Compare checksum of requested profile with cached checksum. - - If checksums are the same - exit. - - Apply requested profile - - Calculate checksum of the profile and the variables file and save it - - If `reboot_after_apply` is True, then reboot the host +1. `microshift-tuned.service` runs (after `tuned.service`, before `microshift.service`) and optionally reboots the system. + See "microshift-tuned service" section below for more information. 1. Host boots again, everything for low latency is in place, `microshift.service` can continue start up. @@ -213,14 +207,8 @@ RUN systemctl enable microshift-tuned.service 1. Production environment - User creates `/etc/microshift/microshift-tuned.yaml` to configure `microshift-tuned.service` - User enables `microshift.service` - - User enables and starts `microshift-tuned.service` which: - - Compares active profile with requested profile - - If requested profile is already active: - - Compare checksum of requested profile with cached checksum. - - If checksums are the same - exit. - - Apply requested profile - - Calculate checksum of the profile and the variables file and save it - - If `reboot_after_apply` is True, then reboot the host + - User enables and starts `microshift-tuned.service` which activates the TuneD profile and optionally reboot the host.: + See "microshift-tuned service" section below for more information. - Host is rebooted: MicroShift starts because it was enabled - Host doesn't need reboot: - User starts `microshift.service` @@ -232,7 +220,7 @@ RUN systemctl enable microshift-tuned.service - Setting Pod's memory limit and memory request to the same value, and setting CPU limit and CPU request to the same value to ensure Pod has guaranteed QoS class. - Use annotations to get desired behavior - (unless link to a documentation is present, these annotations only take two values: enabled and disabled): + (unless link to a documentation is present, these annotations only take two values: `enabled` and `disabled`): - `cpu-load-balancing.crio.io: "disable"` - disable CPU load balancing for Pod (only use with CPU Manager `static` policy and for Guaranteed QoS Pods using whole CPUs) - `cpu-quota.crio.io: "disable"` - disable Completely Fair Scheduler (CFS) @@ -326,16 +314,49 @@ hugepages = 0 additional_args = ``` -#### `microshift-tuned.service` configuration +#### microshift-tuned service -Config file to specify which profile to re-apply each boot and if host should be rebooted if -the kargs before and after applying profile are mismatched. +`microshift-tuned.service` will be responsible for activating TuneD profile specified by user in the config. +User will also need to specify if host should be rebooted after activating the profile. ```yaml profile: microshift-baseline reboot_after_apply: True ``` +Rationale for creating microshift-tuned.service: +1. Automatic apply of a TuneD profile and reboot if requested - helps with unattended install of a fleet of devices. +2. TuneD does not reboot the system in case of profile change. TuneD daemon will reapply the profile, + but there were changes to kernel arguments, they will be inactive until host is rebooted which would a manual operation. + +To address 1. microshift-tuned will, on each start, apply specified by user TuneD profile and reboot the host if user commanded so (`reboot_after_apply`). +To address 2. microshift-tuned will calculate checksum of content of TuneD profile and checksum of variables file referrenced in the profile. +These checksums will be persisted on disk and used on next start of the microshift-tuned to decide +if the profile changed and should be re-applied, optionally followed with reboot. + +`microshift-tuned.service`'s workflow is as follows: +- Compares active profile (according to TuneD) with requested profile (in the config file) +- If requested profile is already active: + - Compare checksum of requested profile with cached checksum. + - If checksums are the same - exit. +- Apply requested profile +- Calculate checksum of the profile and the variables file and save it +- If `reboot_after_apply` is True, then reboot the host + +In case of errors: +- Checksum cache cannot be loaded + - Checksum cache is loaded when active and desired profiles are the same. + - If checksum cannot be loaded, it means that profile was activated outside `microshift-tuned.service` + - Checksum should be calculated and stored in the cache + - microshift-tuned.service exits with success + - Why not reapply the profile and reboot? Maybe the cache is not there, because it could not be written. If on missing cache we would potentially reboot, this could mean a boot loop. +- If any operation fails not reaching the `tuned-adm profile $PROFILE`, then the currently active profile will stay active. +- If `tuned-adm profile $PROFILE` fails, it is very likely that the active profile is unloaded and new profile is not active resulting in no active profile. + - This can be further investigated by interacting with TuneD, e.g. + - `sudo tuned-adm active` to get active profile + - Inspecting `/var/log/tuned/tuned.log` for errors + + #### CRI-O configuration ```ini