Skip to content

Latest commit

 

History

History
250 lines (199 loc) · 10.3 KB

cpu-templates.md

File metadata and controls

250 lines (199 loc) · 10.3 KB
Raw

CPU templates

Firecracker allows users to customise how the vCPUs are represented to the guest software by changing the following configuration:

  • CPUID (x86_64 only)
  • MSRs (Model Specific Registers, x86_64 only)
  • ARM registers (aarch64 only)
  • vCPU features (aarch64 only)
  • KVM capabilities (both x86_64 and aarch64)

A combination of the changes to the above entities is called a CPU template.

The functionality can be used when a user wants to mask a feature from the guest. A real world use case for this is representing a heterogeneous fleet (a fleet consisting of multiple CPU models) as a homogeneous fleet, so the guests will experience a consistent feature set supported by the host.

Note Representing one CPU vendor as another CPU vendor is not supported.

Note CPU templates shall not be used as a security protection against malicious guests. Disabling a feature in a CPU template does not generally make it completely unavailable to the guest. For example, disabling a feature related to an instruction set will indicate to the guest that the feature is not supported, but the guest may still be able to execute corresponding instructions if it does not obey the feature bit.

Firecracker supports two types of CPU templates:

  • Static CPU templates - a set of built-in CPU templates for users to choose from
  • Custom CPU templates - users can create their own CPU templates in json format and pass them to Firecracker

Note Static CPU templates are deprecated starting from v1.5.0 and will be removed in accordance with our deprecation policy. Even after the removal, custom CPU templates are available as an improved iteration of static CPU templates. For more information about the transition from static CPU templates to custom CPU templates, please refer to this GitHub discussion.

Note CPU templates for ARM (both static and custom) require the following patch to be available in the host kernel: Support writable CPU ID registers from userspace. Otherwise KVM will fail to write to the ARM registers.

Static CPU templates

At the moment the following set of static CPU templates are supported:

CPU template CPU vendor CPU model
C3 Intel any
T2 Intel any
T2A AMD Milan
T2CL Intel Cascade Lake or newer
T2S Intel any
V1N1 ARM Neoverse V1

T2 and C3 templates are mapped as close as possible to AWS T2 and C3 instances in terms of CPU features. Note that on a microVM that is lauched with the C3 template and running on processors that do not enumerate FBSDP_NO, PSDP_NO and SBDR_SSDP_NO on IA32_ARCH_CAPABILITIES MSR, the kernel does not apply the mitigation against MMIO stale data vulnerability.

The T2S template is designed to allow migrating snapshots between hosts with Intel Skylake and Intel Cascade Lake securely by further restricting CPU features for the guest, however this comes with a performance penalty. Users are encouraged to carry out a performance assessment if they wish to use the T2S template. Note that Firecracker expects the host to always be running the latest version of the microcode.

The T2CL template is mapped to be close to Intel Cascade Lake. It is not safe to use it on Intel CPUs older than Cascade Lake (such as Skylake).

The only AMD template is T2A. It is considered safe to be used with AMD Milan.

Intel T2CL and AMD T2A templates together aim to provide instruction set feature parity between CPUs running them, so they can form a heterogeneous fleet exposing the same instruction sets to the application.

The V1N1 template is designed to represent ARM Neoverse V1 as ARM Neoverse N1.

Configuring static CPU templates

Configuration of a static CPU template is performed via the /machine-config API endpoint:

curl --unix-socket /tmp/firecracker.socket -i  \
  -X PUT 'http://localhost/machine-config' \
  -H 'Accept: application/json'            \
  -H 'Content-Type: application/json'      \
  -d '{
           "vcpu_count": 2,
           "mem_size_mib": 1024,
           "cpu_template": "T2CL"
  }'

Custom CPU templates

Users can create their own CPU templates by creating a json file containing modifiers for CPUID, MSRs or ARM registers.

Note Creating custom CPU templates requires expert knowledge of CPU architectures. Custom CPU templates must be tested thoroughly before use in production. An inappropriate configuration may lead to guest crashes or making guests vulnerable to security attacks. For example, if a CPU template signals a hardware vulnerability mitigation to the guest while the mitigation is in fact not supported by the hardware, the guest may decide to disable corresponding software mitigations which will make the guest vulnerable.

Note Having MSRs or ARM registers in the custom CPU template does not affect access permissions that guests will have to those registers. The access control is handled by KVM and is not influenced by CPU templates.

Note When setting guest configuration, KVM may reject setting some bits quietly. This is user's responsibility to make sure that their custom CPU template is applied as expected even if Firecracker does not report an error.

In order to assist with creation and usage of CPU templates, there exists a CPU template helper tool. More details can be found here.

Configuring custom CPU templates

Configuration of a custom CPU template is performed via the /cpu-config API endpoint.

An example of configuring a custom CPU template on x86_64:

curl --unix-socket /tmp/firecracker.socket -i  \
  -X PUT 'http://localhost/cpu-config' \
  -H 'Accept: application/json'            \
  -H 'Content-Type: application/json'      \
  -d '{
        "kvm_capabilities": ["!56"],
        "cpuid_modifiers": [
          {
            "leaf": "0x1",
            "subleaf": "0x0",
            "flags": 0,
            "modifiers": [
              {
                "register": "eax",
                "bitmap": "0bxxxx000000000011xx00011011110010"
              }
            ]
          }
        ],
        "msr_modifiers": [
          {
            "addr": "0x10a",
            "bitmap": "0b0000000000000000000000000000000000000000000000000000000000000000"
          }
        ]
      }'

This CPU template will do the following:

  • removes check for KVM capability: KVM_CAP_XCRS. This allows Firecracker to run on old cpus. See this discussion.
  • in leaf 0x1, subleaf 0x0, register eax:
    • clear bits 0b00001111111111000011100100001101
    • set bits 0b00000000000000110000011011110010
    • leave bits 0b11110000000000001100000000000000 intact.
  • in MSR 0x10, it will clear all bits.

An example of configuring a custom CPU template on ARM:

curl --unix-socket /tmp/firecracker.socket -i  \
  -X PUT 'http://localhost/cpu-config' \
  -H 'Accept: application/json'            \
  -H 'Content-Type: application/json'      \
  -d '{
        "kvm_capabilities": ["171", "172"],
        "vcpu_features": [{ "index": 0, "bitmap": "0b11xxxxx" }]
        "reg_modifiers": [
          {
            "addr": "0x603000000013c020",
            "bitmap": "0bxxxxxxxxxxxx0000xxxxxxxxxxxx0000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          }
        ]
      }'

This CPU template will do the following:

  • add checks for KVM capabilities: KVM_CAP_ARM_PTRAUTH_ADDRESS and KVM_CAP_ARM_PTRAUTH_GENERIC. These checks are to ensure that the host have capabilities needed for the vCPU features.
  • enable additional vCPU features: KVM_ARM_VCPU_PTRAUTH_ADDRESS and KVM_ARM_VCPU_PTRAUTH_GENERIC
  • modify ARM register 0x603000000013c020:
    • clear bits 0b0000000000001111000000000000111100000000000000000000000000000000
    • leave bits 0b1111111111110000111111111111000011111111111111111111111111111111 intact.

Information about KVM capabilities can be found in the kernel source. Information about vCPU features on aarch64 can be found in the kernel source. Information on how the ARM register addresses are constructed can be found in the KVM API documentation.

Custom CPU templates language schema

The full description of the custom CPU templates language can be found here.

Note You can also use _ to visually separate parts of a bitmap. So instead of writing: 0b0000xxxx, it can be 0b0000_xxxx.

Expansion of contracted bitmaps

If a contracted version of a bitmap is given, for example, 0b101 where a 32-bit bitmap is expected, missing characters are implied to be x (0bxxxxxxxxxxxxxxxxxxxxxxxxxxxxx101).

CPUID normalization and boot protocol register settings

Some of the configuration set by a custom CPU template may be overwritten by Firecracker. More details can be found here and here.

Information about architecture-specific settings

For detailed information when working with custom CPU templates, please refer to hardware specifications from CPU vendors, for example:

A note about configuration of both static and custom CPU templates

If a user configured both a static CPU template (via /machine-config) and a custom CPU template (via /cpu-config) in the same Firecracker process, only the configuration that was performed the last is applied. This means that if a static CPU template was configured first and a custom CPU template was configured later, only the custom CPU template configuration will be applied when starting a microVM.