neonvm: introduce CPU sysfs state scaling flow based on the vmSpec.cpuScalingMode #1111

mikhail-sakhnov · 2024-10-14T13:37:50Z

Introduce separate CPU scaling flow based on the vmSpec.cpuScalingMode

If vmSpec.cpuScalingMode is equal to qmpScaling the logic of the
scaling is preserved as before:

Scale, if required the amount of CPUs using qmp commands.
If it is required to scale cgroups, call vm-runner /cpu_change endpoint

if vmSpec.cpuScalingMode is equal to sysfsScaling all cpu
scaling requests go directly to the vm-runner /cpu_change, which in
that configuration goes to the neonvm-daemon to reconcile required
amount of online CPUs.

Value cpuSysfsStateScaling also modifies the qemu and the kernel
arguments to enable plug all CPUs but mark as online only first one.

mikhail-sakhnov · 2024-10-15T12:51:09Z

FYI, @sharnoff
We discussed with @Omrigan that it would be great to reflect the currently used cpuScalingMode in status or annotations for better observability plus probably we want to restart pod if the cpuScalingMode was changed - that actually would allow us to perform gradual testing manually without switching default mode.

sharnoff

Partial review, haven't yet gotten to the neonvm-controller changes

neonvm-controller/cmd/main.go

Makefile

neonvm-controller/deployment.yaml

neonvm-daemon/pkg/cpuscaling/sys_fs_state_scaling.go

neonvm-runner/cmd/main.go

neonvm/apis/neonvm/v1/virtualmachine_types.go

pkg/api/types.go

Omrigan

A lot of suggestions around naming, and other.

neonvm-controller/cmd/main.go

neonvm/apis/neonvm/v1/virtualmachine_types.go

pkg/neonvm/controllers/vm_controller.go

pkg/neonvm/controllers/vm_controller_handle_cpu_scaling.go

pkg/neonvm/cpuscaling/sysfsscaling.go

neonvm-daemon/pkg/cpuscaling/sys_fs_state_scaling.go

neonvm-runner/cmd/main.go

neonvm-daemon/cmd/main.go

neonvm-runner/cmd/main.go

sharnoff

Looks great! 🚀

Left just a couple final comments

pkg/neonvm/cpuscaling/cpuscaler.go

pkg/neonvm/cpuscaling/sysfs.go

…xplicitly default the field value if it is not set Signed-off-by: Misha Sakhnov <[email protected]>

…nd mark vmSpec.Guest.CPUs.Min as online during the boot Signed-off-by: Misha Sakhnov <[email protected]>

Signed-off-by: Misha Sakhnov <[email protected]>

introduce separate CPU scaling flow based on the vmSpec.cpuScalingMode If vmSpec.cpuScalingMode is equal to `qmp_scaling` the logic of the scaling is preserved as before: - Scale, if required the amount of CPUs using qmp commands. - If it is required to scale cgroups, call vm-runner /cpu_change endpoint if vmSpec.cpuScalingMode is equal to `cpuSysfsStateScaling` all cpu scaling requests go directly to the vm-runner /cpu_change, which in that configuration goes to the neonvm-daemon to reconcile required amount of online CPUs. Value `cpuSysfsStateScaling` also modifies the qemu and the kernel arguments to enable plug all CPUs but mark as online only first one. Signed-off-by: Misha Sakhnov <[email protected]>

pass cpuScalingMode as argument to the vm-runner rename arguments, constants and functions here and there drop unused code move default cpu scaling mode to controller argument Signed-off-by: Misha Sakhnov <[email protected]>

Co-authored-by: Oleg Vasilev <[email protected]>

Signed-off-by: Misha Sakhnov <[email protected]>

Co-authored-by: Oleg Vasilev <[email protected]>

Signed-off-by: Misha Sakhnov <[email protected]>

change CpuScalingMode to be a enum type drop obsolete comments Signed-off-by: Misha Sakhnov <[email protected]>

Improve readability in pkg/neonvm/controllers/vm_controller_cpu_scaling.go by adding empty lines and comments Signed-off-by: Misha Sakhnov <[email protected]>

Change scaling logic to work with ranges instead of slices Split sys fs scaling struture to allow unit testing Add unit tests for the main reconcilation logic Signed-off-by: Misha Sakhnov <[email protected]>

Get current state through aggregated /cpu/online file instead of using per-cpu files Signed-off-by: Misha Sakhnov <[email protected]>

Co-authored-by: Em Sharnoff <[email protected]> Signed-off-by: Misha Sakhnov <[email protected]>

Change cpuscaling logic to work with collection of IDs Signed-off-by: Misha Sakhnov <[email protected]>

Omrigan

LGTM, one suggestion.

neonvm-runner/cmd/main.go

Co-authored-by: Oleg Vasilev <[email protected]>

mikhail-sakhnov force-pushed the misha/cpu-scaling branch 4 times, most recently from 85608dd to dfe79f1 Compare October 14, 2024 13:47

mikhail-sakhnov changed the title ~~Misha/cpu scaling~~ neonvm-controller, neonvm-runner, neonvm-daemon: introduce CPU sysfs state scaling flow based on the vmSpec.cpuScalingMode Oct 14, 2024

mikhail-sakhnov changed the title ~~neonvm-controller, neonvm-runner, neonvm-daemon: introduce CPU sysfs state scaling flow based on the vmSpec.cpuScalingMode~~ neonvm: introduce CPU sysfs state scaling flow based on the vmSpec.cpuScalingMode Oct 14, 2024

mikhail-sakhnov force-pushed the misha/cpu-scaling branch 3 times, most recently from 94bdcba to 002f978 Compare October 14, 2024 13:57

sharnoff mentioned this pull request Oct 14, 2024

neonvm-controller: add spec.cpuScalingMode #1104

Closed

mikhail-sakhnov requested review from sharnoff, Omrigan and petuhovskiy October 14, 2024 14:05

mikhail-sakhnov mentioned this pull request Oct 14, 2024

Replace hotplug cpu by online/offline CPU #1082

Closed

mikhail-sakhnov force-pushed the misha/cpu-scaling branch 3 times, most recently from 508920e to e48f902 Compare October 14, 2024 18:32

mikhail-sakhnov marked this pull request as ready for review October 14, 2024 18:36

mikhail-sakhnov mentioned this pull request Oct 14, 2024

e2e-tests: fix ps path in vm-neonvmd test #1113

Merged

sharnoff self-assigned this Oct 14, 2024

mikhail-sakhnov force-pushed the misha/cpu-scaling branch from e48f902 to 6156c36 Compare October 15, 2024 09:56

sharnoff reviewed Oct 17, 2024

View reviewed changes

mikhail-sakhnov force-pushed the misha/cpu-scaling branch 3 times, most recently from 935c4fa to 5a094ae Compare October 17, 2024 12:32

mikhail-sakhnov requested a review from sharnoff October 17, 2024 12:32

mikhail-sakhnov force-pushed the misha/cpu-scaling branch from 5a094ae to 82a4827 Compare October 17, 2024 12:45

Omrigan reviewed Oct 18, 2024

View reviewed changes

mikhail-sakhnov force-pushed the misha/cpu-scaling branch from f1af40d to 0658f05 Compare October 21, 2024 08:45

sharnoff approved these changes Nov 20, 2024

View reviewed changes

pkg/neonvm/cpuscaling/cpuscaler.go Outdated Show resolved Hide resolved

pkg/neonvm/cpuscaling/sysfs.go Outdated Show resolved Hide resolved

mikhail-sakhnov force-pushed the misha/cpu-scaling branch 2 times, most recently from d0af69f to fe10eef Compare November 20, 2024 16:03

sharnoff assigned mikhail-sakhnov and unassigned sharnoff Nov 21, 2024

mikhail-sakhnov and others added 16 commits November 26, 2024 13:12

neonvm-controller: add spec.cpuScalingMode, change vm-controller to e…

2fb4c6c

…xplicitly default the field value if it is not set Signed-off-by: Misha Sakhnov <[email protected]>

neonvmrunner,neonvmcontroller: allocate all possible CPUs to the VM a…

dd9b40c

…nd mark vmSpec.Guest.CPUs.Min as online during the boot Signed-off-by: Misha Sakhnov <[email protected]>

neonvmd: expose interface to control online/offline state of the CPUs

c2f1e12

Signed-off-by: Misha Sakhnov <[email protected]>

neonvm: apply code review fixes

32de84e

pass cpuScalingMode as argument to the vm-runner rename arguments, constants and functions here and there drop unused code move default cpu scaling mode to controller argument Signed-off-by: Misha Sakhnov <[email protected]>

neonvm-controller: apply lint and regenerate code

d628a9a

Co-authored-by: Oleg Vasilev <[email protected]>

neonvm-controller: rename runner cpu functions

37be0db

Signed-off-by: Misha Sakhnov <[email protected]>

neonvm-controller: make spec.cpuScalingMode immutable if already set

1261b88

Signed-off-by: Misha Sakhnov <[email protected]>

Update pkg/neonvm/controllers/vm_controller_cpu_scaling.go

9e56963

Co-authored-by: Oleg Vasilev <[email protected]>

neonvm-runner: apply code review fixes and lint

475efde

Signed-off-by: Misha Sakhnov <[email protected]>

neonvm: apply code review fixes

28ab9c1

change CpuScalingMode to be a enum type drop obsolete comments Signed-off-by: Misha Sakhnov <[email protected]>

neonvm-controller: code review fixes

d6ff7c7

Improve readability in pkg/neonvm/controllers/vm_controller_cpu_scaling.go by adding empty lines and comments Signed-off-by: Misha Sakhnov <[email protected]>

neonvm-daemon: apply code review fixes

09f0375

Change scaling logic to work with ranges instead of slices Split sys fs scaling struture to allow unit testing Add unit tests for the main reconcilation logic Signed-off-by: Misha Sakhnov <[email protected]>

neonvm-daemon: apply code review fix

37e8226

Get current state through aggregated /cpu/online file instead of using per-cpu files Signed-off-by: Misha Sakhnov <[email protected]>

neonvm: apply code review fixes

4532840

Co-authored-by: Em Sharnoff <[email protected]> Signed-off-by: Misha Sakhnov <[email protected]>

neonvm: apply code review fixes

79721a2

Change cpuscaling logic to work with collection of IDs Signed-off-by: Misha Sakhnov <[email protected]>

mikhail-sakhnov force-pushed the misha/cpu-scaling branch from fe10eef to 79721a2 Compare November 26, 2024 12:14

Omrigan approved these changes Nov 26, 2024

View reviewed changes

neonvm-runner/cmd/main.go Outdated Show resolved Hide resolved

mikhail-sakhnov and others added 2 commits November 26, 2024 17:07

Update neonvm-runner/cmd/main.go

fabb7ab

Co-authored-by: Oleg Vasilev <[email protected]>

Merge branch 'main' into misha/cpu-scaling

327e7fb

Omrigan enabled auto-merge (squash) November 27, 2024 11:19

Omrigan merged commit 0bc5fe1 into main Nov 27, 2024
22 checks passed

Omrigan deleted the misha/cpu-scaling branch November 27, 2024 14:52

mikhail-sakhnov mentioned this pull request Dec 5, 2024

Publish neonvm-daemon image as a part of the build-images.yml #1167

Closed

mikhail-sakhnov mentioned this pull request Dec 13, 2024

neonvm: handle sysfs scaling mode in migrations #1179

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

neonvm: introduce CPU sysfs state scaling flow based on the vmSpec.cpuScalingMode #1111

neonvm: introduce CPU sysfs state scaling flow based on the vmSpec.cpuScalingMode #1111

mikhail-sakhnov commented Oct 14, 2024 •

edited

Loading

mikhail-sakhnov commented Oct 15, 2024

sharnoff left a comment

Omrigan left a comment

sharnoff left a comment

Omrigan left a comment

neonvm: introduce CPU sysfs state scaling flow based on the vmSpec.cpuScalingMode #1111

neonvm: introduce CPU sysfs state scaling flow based on the vmSpec.cpuScalingMode #1111

Conversation

mikhail-sakhnov commented Oct 14, 2024 • edited Loading

mikhail-sakhnov commented Oct 15, 2024

sharnoff left a comment

Choose a reason for hiding this comment

Omrigan left a comment

Choose a reason for hiding this comment

sharnoff left a comment

Choose a reason for hiding this comment

Omrigan left a comment

Choose a reason for hiding this comment

mikhail-sakhnov commented Oct 14, 2024 •

edited

Loading