CP-52709: use timeslices shorter than 50ms #6177

edwintorok · 2024-12-12T22:40:17Z

Changing the default OCaml thread switch timeslice from 50ms

The default OCaml 4.x timeslice for switching between threads is 50ms: if there is more than 1 active OCaml threads each one is let to run up to 50ms, and then (at various safepoints) it can switch to another running thread.
When the runtime lock is released (and C code or syscalls run) then another OCaml thread is immediately let to run if any.

However 50ms is too long, and it inserts large latencies into the handling of API calls.

OTOH if a timeslice is too short then we waste CPU time:

overhead of Thread.yield system call, and the cost of switching threads at the OS level
potentially higher L1/L2 cache misses if we switch on the same CPU between multiple OCaml threads
potentially losing branch predictor history
potentially higher L3 cache misses (but on a hypervisor with VMs running L3 will be mostly taken up by VMs anyway, we can only rely on L1/L2 staying with us)

A microbenchmark has shown that timeslices as small as 0.5ms might strike an optimal balance between latency and overhead: values lower than that lose performance due to increased overhead, and values higher than that lose performance due to increased latency:

(the microbenchmark measures the number of CPU cycles spent simulating an API call with various working set sizes and timeslice settings)

This is all hardware dependent though, and a future PR will introduce an autotune service that measures the yield overhead and L1/L2 cache refill overhead and calculates an optimal timeslice for that particular hardware/Xen/kernel combination.
(and while we're at it, we can also tweak the minor heap size to match ~half of CPU L2 cache).

Timeslice change mechanism

Initially I used Unix.set_itimer using virtual timers, to switch a thread only when it has been actively using CPU for too long. However that relies on delivering a signal to the process, and XAPI is very bad at handling signals.
In fact XAPI is not allowed to receive any signals, because it doesn't handle EINTR well (a typical problem, that affects C programs too sometimes). Although this is a well understood problem (described in the OCaml Unix book, and some areas of XAPI make an effort to handle it, others just assert that they never receive one. Fixing that would require changes in all of XAPI (and its dependencies).

So instead I don't use signals at all, but rely on Statmemprof to trigger a hook to be executed "periodically", but not based purely on time, but on allocation activity (i.e. at places the GC could run). The hook checks the elapsed time since the last time it got called, and if too much then calls Thread.yield.
Yield is smart enough to be a no-op if there aren't any other runnable OCaml threads.

Yield isn't always beneficial though at reducing latencies, e.g. if we are holding locks then we're just increasing latency for everyone who waits for that lock.
So a mechanism is introduced to notify the periodic function when any highly contended locks are held, and the yield is skipped in this instance (e.g. the XAPI DB lock).

Plotting code

This PR only includes a very simplified version of the microbenchmark, a separate one will introduce the full cache plotting code (which is useful for development/troubleshooting purposes but won't be needed at runtime).

Default timeslice value

Set to 5ms for now, just a bit above 4ms = 1/HZ in our Dom0 kernel, the autotuner from a future PR can change this to a more appropriate value.
(the autotuner needs more validation on a wider range of hardware)

Results

The cache measurements needs to be repeated on a wider variety of hardware, but the timeslice changes here have already proven useful in reducing XAPI DB lock hold times (together with other optimizations).

Uses Gc.Memprof to run a hook function periodically. This checks whether we are holding any locks, and if not and sufficient time has elapsed since the last, then we yield. POSIX timers wouldn't work here, because they deliver signals, and there are too many places in XAPI that don't handle EINTR properly. Signed-off-by: Edwin Török <[email protected]>

And apply on startup. Signed-off-by: Edwin Török <[email protected]>

Signed-off-by: Edwin Török <[email protected]>

robhoes · 2024-12-13T11:01:03Z

ocaml/libs/timeslice/timeslice.ml

@@ -0,0 +1,48 @@
+(* avoid allocating an extra option every time *)


Please add the copyright headers.

This applies to all the files, not just this.

robhoes · 2024-12-13T11:04:20Z

ocaml/xapi-idl/lib/xcp_service.ml

+  let interval = !timeslice in
+  D.debug "Setting timeslice to %.3fs" interval ;
+  if interval >= 0.05 then
+    D.debug "Timeslice same as or larger than default: not setting"


Then where is Xapi_timeslice.Timeslice.set called for the default case?

Oh you mean the OCaml default of 50 ms.

Yeah the default timeslice is always there, this is in addition to that, it is not possible to turn off the default 50 ms switching.
I should probably document that.

So the idea is to add additional yields; would suggest to add __FUNCTION__ to log messages.

lindig · 2024-12-16T09:04:39Z

ocaml/libs/timeslice/timeslice.ml

@@ -0,0 +1,48 @@
+(* avoid allocating an extra option every time *)


This applies to all the files, not just this.

lindig · 2024-12-16T09:06:41Z

ocaml/xapi-idl/lib/xcp_service.ml

+  let interval = !timeslice in
+  D.debug "Setting timeslice to %.3fs" interval ;
+  if interval >= 0.05 then
+    D.debug "Timeslice same as or larger than default: not setting"


So the idea is to add additional yields; would suggest to add __FUNCTION__ to log messages.

lindig · 2024-12-16T09:07:36Z

ocaml/xapi-idl/lib/xcp_service.ml

@@ -454,6 +478,7 @@ let configure_common ~options ~resources arg_parse_fn =
        failwith (String.concat "\n" lines)
    )
    resources ;
+  apply_timeslice () ;


Would adjust_timeslice be a better name?

lindig · 2024-12-16T09:12:24Z

ocaml/xapi-idl/lib/xcp_service.ml

@@ -236,6 +255,11 @@ let common_options =
    , (fun () -> !config_dir)
    , "Location of directory containing configuration file fragments"
    )
+  ; ( "timeslice"
+    , Arg.Set_float timeslice
+    , (fun () -> Printf.sprintf "%.3f" !timeslice)


Not sure seconds is the best unit here when we mostly talk about milliseconds. However, this has enough resolution down to 1ms.

lindig · 2024-12-17T09:03:08Z

So instead I don't use signals at all, but rely on Statmemprof to trigger a hook to be executed "periodically", but not based purely on time, but on allocation activity (i.e. at places the GC could run). The hook checks the elapsed time since the last time it got called, and if too much then calls Thread.yield.

This seem to be crucial as the mechanism to insert additional yields. Could you elaborate how Statmemprof works such that a thread periodically executes a hook that may yield? As this is not xapi specific, is there a way to release this as an opam package to engage the OCaml community?

edwintorok added 5 commits December 12, 2024 13:46

CP-52709: add timeslice configuration to all services

4d1b7d2

And apply on startup. Signed-off-by: Edwin Török <[email protected]>

CP-52709: add simple measurement code

764cbf1

Signed-off-by: Edwin Török <[email protected]>

CP-52709: recommended measurement

b30e768

Signed-off-by: Edwin Török <[email protected]>

CP-52709: Enable timeslice setting during unit tests by default

572559c

Signed-off-by: Edwin Török <[email protected]>

edwintorok force-pushed the pr/CP-52709 branch from b24f60b to 572559c Compare December 12, 2024 22:43

edwintorok mentioned this pull request Dec 12, 2024

CP-49141: mark the DB lock as high priority #6180

Draft

robhoes reviewed Dec 13, 2024

View reviewed changes

lindig approved these changes Dec 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CP-52709: use timeslices shorter than 50ms #6177

CP-52709: use timeslices shorter than 50ms #6177

edwintorok commented Dec 12, 2024 •

edited

Loading

robhoes Dec 13, 2024

lindig Dec 16, 2024

robhoes Dec 13, 2024 •

edited

Loading

robhoes Dec 13, 2024

edwintorok Dec 13, 2024 •

edited

Loading

lindig Dec 16, 2024

lindig Dec 16, 2024

lindig Dec 16, 2024

lindig Dec 16, 2024

lindig Dec 16, 2024

lindig commented Dec 17, 2024

		@@ -0,0 +1,48 @@
		(* avoid allocating an extra option every time *)

CP-52709: use timeslices shorter than 50ms #6177

Are you sure you want to change the base?

CP-52709: use timeslices shorter than 50ms #6177

Conversation

edwintorok commented Dec 12, 2024 • edited Loading

Changing the default OCaml thread switch timeslice from 50ms

Timeslice change mechanism

Plotting code

Default timeslice value

Results

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robhoes Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edwintorok Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lindig commented Dec 17, 2024

edwintorok commented Dec 12, 2024 •

edited

Loading

robhoes Dec 13, 2024 •

edited

Loading

edwintorok Dec 13, 2024 •

edited

Loading