Sync from master to feature/easier-pool-join (#6170)

.github/workflows/generate-and-build-sdks.yml

@@ -24,6 +24,14 @@ jobs:
         shell: bash
         run: opam exec -- make sdk
 
+      # sdk-ci runs some Go unit tests.
+      # This setting ensures that SDK date time 
+      # tests are run on a machine that
+      # isn't using UTC
+      - name: Set Timezone to Tokyo for datetime tests
+        run: |
+          sudo timedatectl set-timezone Asia/Tokyo
+
       - name: Run CI for SDKs
         uses: ./.github/workflows/sdk-ci
 
@@ -54,6 +62,7 @@ jobs:
           path: |
             _build/install/default/share/go/*
             !_build/install/default/share/go/dune
+            !_build/install/default/share/go/**/*_test.go
 
       - name: Store Java SDK source
         uses: actions/upload-artifact@v4
@@ -110,6 +119,14 @@ jobs:
           java-version: '17'
           distribution: 'temurin'
 
+      # Java Tests are run at compile time.
+      # This setting ensures that SDK date time 
+      # tests are run on a machine that
+      # isn't using UTC 
+      - name: Set Timezone to Tokyo for datetime tests
+        run: |
+          sudo timedatectl set-timezone Asia/Tokyo
+
       - name: Build Java SDK
         shell: bash
         run: |
@@ -138,6 +155,21 @@ jobs:
           name: SDK_Source_CSharp
           path: source/
 
+        # All tests builds and pipelines should
+        # work on other timezones. This setting ensures that
+        # SDK date time tests are run on a machine that
+        # isn't using UTC 
+      - name: Set Timezone to Tokyo for datetime tests
+        shell: pwsh
+        run: Set-TimeZone -Id "Tokyo Standard Time"
+
+      - name: Test C# SDK
+        shell: pwsh
+        run: |
+          dotnet test source/XenServerTest `
+          --disable-build-servers `
+          --verbosity=normal
+
       - name: Build C# SDK
         shell: pwsh
         run: |

.github/workflows/go-ci/action.yml

@@ -14,6 +14,11 @@ runs:
         working-directory: ${{ github.workspace }}/_build/install/default/share/go/src
         args: --config=${{ github.workspace }}/.golangci.yml
 
+    - name: Run Go Tests
+      shell: bash
+      working-directory: ${{ github.workspace }}/_build/install/default/share/go/src
+      run: go test -v
+
     - name: Run CI for Go SDK
       shell: bash
       run: |

doc/content/design/plugin-protocol-v2.md

@@ -20,7 +20,7 @@ DATASOURCES
 000001e4
 dba4bf7a84b6d11d565d19ef91f7906e
 {
-  "timestamp": 1339685573,
+  "timestamp": 1339685573.245,
   "data_sources": {
     "cpu-temp-cpu0": {
       "description": "Temperature of CPU 0",
@@ -62,7 +62,7 @@ reported datasources.
 ### Example
 ```
 {
-  "timestamp": 1339685573,
+  "timestamp": 1339685573.245,
   "data_sources": {
     "cpu-temp-cpu0": {
       "description": "Temperature of CPU 0",
@@ -96,7 +96,7 @@ Protocol V2
 |data checksum        |32               |int32 |binary-encoded crc32 of the concatenation of the encoded timestamp and datasource values|
 |metadata checksum    |32               |int32 |binary-encoded crc32 of the metadata string (see below)                                 |
 |number of datasources|32               |int32 |only needed if the metadata has changed - otherwise RRDD can use a cached value         |
-|timestamp            |64               |int64 |Unix epoch                                                                              |
+|timestamp            |64               |double|Unix epoch                                                                              |
 |datasource values    |n * 64           |int64 \| double |n is the number of datasources exported by the plugin, type dependent on the setting in the metadata for value_type [int64\|float]  |
 |metadata length      |32               |int32 |                                                                                        |
 |metadata             |(string length)*8|string|                                                                                        |
@@ -193,6 +193,3 @@ This means that for a normal update, RRDD will only have to read the header plus
 the first (16 + 16 + 4 + 8 + 8*n) bytes of data, where n is the number of
 datasources exported by the plugin. If the metadata changes RRDD will have to
 read all the data (and parse the metadata).
-
-n.b. the timestamp reported by plugins is not currently used by RRDD - it uses
-its own global timestamp.

doc/content/toolstack/features/NUMA/index.md

@@ -49,7 +49,7 @@ There is also I/O NUMA where a cost is similarly associated to where a PCIe is p
 
 NUMA does have advantages though: if each node accesses only its local memory, then each node can independently achieve maximum throughput.
 
-For best performance we should:
+For best performance, we should:
    - minimize the amount of interconnect bandwidth we are using
    - run code that accesses memory allocated on the closest NUMA node
    - maximize the number of NUMA nodes that we use in the system as a whole
@@ -62,39 +62,59 @@ The Xen scheduler supports 2 kinds of constraints:
 * hard pinning: a vCPU may only run on the specified set of pCPUs and nowhere else
 * soft pinning: a vCPU is *preferably* run on the specified set of pCPUs, but if they are all busy then it may run elsewhere
 
-The former is useful if you want strict separation, but it can potentially leave part of the system idle while another part is bottlenecked with lots of vCPUs all competing for the same limited set of pCPUs.
+Hard pinning can be used to partition the system. But, it can potentially leave part of the system idle while another part is bottlenecked by many vCPUs competing for the same limited set of pCPUs.
 
-Xen does not migrate workloads between NUMA nodes on its own (the Linux kernel does), although it is possible to achieve a similar effect with explicit migration.
-However migration introduces additional delays and is best avoided for entire VMs.
+Xen does not migrate workloads between NUMA nodes on its own (the Linux kernel can). Although, it is possible to achieve a similar effect with explicit migration.
+However, migration introduces additional delays and is best avoided for entire VMs.
 
-The latter (soft pinning) is preferred: running a workload now, even on a potentially suboptimal pCPU (higher NUMA latency) is still better than not running it at all and waiting until a pCPU is freed up.
+Therefore, soft pinning is preferred: Running on a potentially suboptimal pCPU that uses remote memory could still be better than not running it at all until a pCPU is free to run it.
 
-Xen will also allocate memory for the VM according to the vCPU (soft) pinning: if the vCPUs are pinned only to NUMA nodes A and B, then it will allocate the VM's memory from NUMA nodes A and B (in a round-robin way, resulting in interleaving).
+Xen will also allocate memory for the VM according to the vCPU (soft) pinning: If the vCPUs are pinned to NUMA nodes A and B, Xen allocates memory from NUMA nodes A and B in a round-robin way, resulting in interleaving.
 
-By default (no pinning) it will interleave memory from all NUMA nodes, which provides average performance, but individual tasks' performance may be significantly higher or lower depending on which NUMA node the application may have "landed" on.
-Furthermore restarting processes will speed them up or slow them down as address space randomization picks different memory regions inside a VM.
+### Current default: No vCPU pinning
+
+By default, when no vCPU pinning is used, Xen interleaves memory from all NUMA nodes. This averages the memory performance, but individual tasks' performance may be significantly higher or lower depending on which NUMA node the application may have "landed" on.
+As a result, restarting processes will speed them up or slow them down as address space randomization picks different memory regions inside a VM.
+
+This uses the memory bandwidth of all memory controllers and distributes the load across all nodes.
+However, the memory latency is higher as the NUMA interconnects are used for most memory accesses and vCPU synchronization within the Domains.
 
 Note that this is not the worst case: the worst case would be for memory to be allocated on one NUMA node, but the vCPU always running on the furthest away NUMA node.
 
 ## Best effort NUMA-aware memory allocation for VMs
 
-By default Xen stripes the VM's memory accross all NUMA nodes of the host, which means that every VM has to go through all the interconnects. 
+
+### Summary
+
+The best-effort mode attempts to fit Domains into NUMA nodes and to balance memory usage.
+It soft-pins Domains on the NUMA node with the most available memory when adding the Domain.
+Memory is currently allocated when booting the VM (or while constructing the resuming VM).
+
+Parallel boot issue: Memory is not pre-allocated on creation, but allocated during boot.
+The result is that parallel VM creation and boot can exhaust the memory of NUMA nodes.
+
+### Goals
+
+By default, Xen stripes the VM's memory across all NUMA nodes of the host, which means that every VM has to go through all the interconnects.
 The goal here is to find a better allocation than the default, not necessarily an optimal allocation.
-An optimal allocation would require knowing what VMs you would start/create in the future, and planning across hosts too.
+An optimal allocation would require knowing what VMs you would start/create in the future, and planning across hosts.
+This allows the host to use all NUMA nodes to take advantage of the full memory bandwidth available on the pool hosts.
 
-Overall we want to balance the VMs across NUMA nodes, such that we use all NUMA nodes to take advantage of the maximum memory bandwidth available on the system.
+Overall, we want to balance the VMs across NUMA nodes, such that we use all NUMA nodes to take advantage of the maximum memory bandwidth available on the system.
 For now this proposed balancing will be done only by balancing memory usage: always heuristically allocating VMs on the NUMA node that has the most available memory.
-Note that this allocation has a race condition for now when multiple VMs are booted in parallel, because we don't wait until Xen has constructed the domain for each one (that'd serialize domain construction, which is currently parallel). 
+For now, this allocation has a race condition: This happens when multiple VMs are booted in parallel, because we don't wait until Xen has constructed the domain for each one (that'd serialize domain construction, which is currently parallel).
 This may be improved in the future by having an API to query Xen where it has allocated the memory, and to explicitly ask it to place memory on a given NUMA node (instead of best_effort).
 
 If a VM doesn't fit into a single node then it is not so clear what the best approach is.
 One criteria to consider is minimizing the NUMA distance between the nodes chosen for the VM.
-Large NUMA systems may not be fully connected in a mesh requiring multiple hops to each a node, or even have assymetric links, or links with different bitwidth.
-These tradeoff should be approximatively reflected in the ACPI SLIT tables, as a matrix of distances between nodes.
+Large NUMA systems may not be fully connected in a mesh requiring multiple hops to each a node, or even have asymmetric links, or links with different bandwidth.
+The specific NUMA topology is provided by the ACPI SLIT table as the matrix of distances between nodes.
 It is possible that 3 NUMA nodes have a smaller average/maximum distance than 2, so we need to consider all possibilities.
 
 For N nodes there would be 2^N possibilities, so [Topology.NUMA.candidates] limits the number of choices to 65520+N (full set of 2^N possibilities for 16 NUMA nodes, and a reduced set of choices for larger systems).
 
+### Implementation
+
 [Topology.NUMA.candidates] is a sorted sequence of node sets, in ascending order of maximum/average distances.
 Once we've eliminated the candidates not suitable for this VM (that do not have enough total memory/pCPUs) we are left with a monotonically increasing sequence of nodes.
 There are still multiple possibilities with same average distance.
@@ -110,19 +130,19 @@ See page 13 in [^AMD_numa] for a diagram of an AMD Opteron 6272 system.
 
 * Booting multiple VMs in parallel will result in potentially allocating both on the same NUMA node (race condition) 
 * When we're about to run out of host memory we'll fall back to striping memory again, but the soft affinity mask won't reflect that (this needs an API to query Xen on where it has actually placed the VM, so we can fix up the mask accordingly)
-* XAPI is not aware of NUMA balancing across a pool, and choses hosts purely based on total amount of free memory, even if a better NUMA placement could be found on another host
+* XAPI is not aware of NUMA balancing across a pool. Xenopsd chooses NUMA nodes purely based on amount of free memory on the NUMA nodes of the host, even if a better NUMA placement could be found on another host
 * Very large (>16 NUMA nodes) systems may only explore a limited number of choices (fit into a single node vs fallback to full interleaving)
 * The exact VM placement is not yet controllable
 * Microbenchmarks with a single VM on a host show both performance improvements and regressions on memory bandwidth usage: previously a single VM may have been able to take advantage of the bandwidth of both NUMA nodes if it happened to allocate memory from the right places, whereas now it'll be forced to use just a single node.
    As soon as you have more than 1 VM that is busy on a system enabling NUMA balancing should almost always be an improvement though.
-* it is not supported to combine hard vCPU masks with soft affinity: if hard affinities are used then no NUMA scheduling is done by the toolstack and we obey exactly what the user has asked for with hard affinities.
+* It is not supported to combine hard vCPU masks with soft affinity: if hard affinities are used, then no NUMA scheduling is done by the toolstack, and we obey exactly what the user has asked for with hard affinities.
    This shouldn't affect other VMs since the memory used by hard-pinned VMs will still be reflected in overall less memory available on individual NUMA nodes.
 * Corner case: the ACPI standard allows certain NUMA nodes to be unreachable (distance `0xFF` = `-1` in the Xen bindings).
    This is not supported and will cause an exception to be raised.
    If this is an issue in practice the NUMA matrix could be pre-filtered to contain only reachable nodes.
-   NUMA nodes with 0 CPUs *are* accepted (it can result from hard affinity pinnings)
+   NUMA nodes with 0 CPUs *are* accepted (it can result from hard affinity pinning)
 * NUMA balancing is not considered during HA planning
-* Dom0 is a single VM that needs to communicate with all other VMs, so NUMA balancing is not applied to it (we'd need to expose NUMA topology to the Dom0 kernel so it can better allocate processes)
+* Dom0 is a single VM that needs to communicate with all other VMs, so NUMA balancing is not applied to it (we'd need to expose NUMA topology to the Dom0 kernel, so it can better allocate processes)
 * IO NUMA is out of scope for now
 
 ## XAPI datamodel design
@@ -139,7 +159,7 @@ Meaning of the policy:
 * `best_effort`: the algorithm described in this document, where soft pinning is used to achieve better balancing and lower latency
 * `default_policy`: when the admin hasn't expressed a preference
 
-* Currently `default_policy` is treated as `any`, but the admin can change it, and then the system will remember that change across upgrades.
+* Currently, `default_policy` is treated as `any`, but the admin can change it, and then the system will remember that change across upgrades.
    If we didn't have a `default_policy` then changing the "default" policy on an upgrade would be tricky: we either risk overriding an explicit choice of the admin, or existing installs cannot take advantage of the improved performance from `best_effort`
 * Future XAPI versions may change `default_policy` to mean `best_effort`.
    Admins can still override it to `any` if they wish on a host by host basis.
@@ -149,7 +169,7 @@ It is not expected that users would have to change `best_effort`, unless they ru
 There is also no separate feature flag: this host flag acts as a feature flag that can be set through the API without restarting the toolstack.
 Although obviously only new VMs will benefit.
 
-Debugging the allocator is done by running `xl vcpu-list` and investigating the soft pinning masks, and by analyzing xensource.log.
+Debugging the allocator is done by running `xl vcpu-list` and investigating the soft pinning masks, and by analyzing `xensource.log`.
 
 ### Xenopsd implementation
 
@@ -166,18 +186,18 @@ This avoids exponential state space explosion on very large systems (>16 NUMA no
 * [Topology.NUMA.choose] will choose one NUMA node deterministically, while trying to keep overall NUMA node usage balanced.
 * [Domain.numa_placement] builds a [NUMARequest] and uses the above [Topology] and [Softaffinity] functions to compute and apply a plan.
 
-We used to have a `xenopsd.conf` configuration option to enable numa placement, for backwards compatibility this is still supported, but only if the admin hasn't set an explicit policy on the Host.
+We used to have a `xenopsd.conf` configuration option to enable NUMA placement, for backwards compatibility this is still supported, but only if the admin hasn't set an explicit policy on the Host.
 It is best to remove the experimental `xenopsd.conf` entry though, a future version may completely drop it.
 
 Tests are in [test_topology.ml] which checks balancing properties and whether the plan has improved best/worst/average-case access times in a simulated test based on 2 predefined NUMA distance matrixes (one from Intel and one from an AMD system).
 
 ## Future work
 
-* enable 'best_effort' mode by default once more testing has been done
-* an API to query Xen where it has actually allocated the VM's memory.
-   Currently only an `xl debug-keys` interface exists which is not supported in production as it can result in killing the host via the watchdog, and is not a proper API, but a textual debug output with no stability guarantees.
-* more host policies  (e.g. `strict`).
-   Requires the XAPI pool scheduler to be NUMA aware and consider it as part of chosing hosts.
+* Enable 'best_effort' mode by default once more testing has been done
+* Add an API to query Xen for the NUMA node memory placement (where it has actually allocated the VM's memory).
+   Currently, only the `xl debug-keys` interface exists which is not supported in production as it can result in killing the host via the watchdog, and is not a proper API, but a textual debug output with no stability guarantees.
+* More host policies, e.g. `strict`.
+   Requires the XAPI pool scheduler to be NUMA aware and consider it as part of choosing hosts.
 * VM level policy that can set a NUMA affinity index, mapped to a NUMA node modulo NUMA nodes available on the system (this is needed so that after migration we don't end up trying to allocate vCPUs to a non-existent NUMA node)
 * VM level anti-affinity rules for NUMA placement (can be achieved by setting unique NUMA affinity indexes)