Apple Silicon M3 + MacOS 15: nested virtualization more easier

[Fred78290]

Description

After installing the version 10.0.0-beta to test nested virtualization on M3; the default template doesn't offer nested virtualization on MacOS 15 + M3 or higher CPU.

After reading concerned code (fork of Code-VZ) the ability of nested virtualization must be true by default if required conditions are present.

On intel plateform the nested virtualization is always active.

Else add a command line option.

Regards

[AkihiroSuda]

On intel plateform the nested virtualization is always active.

Did you mean inactive?
Intel doesn't seem mentioned in https://developer.apple.com/documentation/virtualization/vzgenericplatformconfiguration/4360553-isnestedvirtualizationsupported

I tried it on MacBook Pro 2020 (2.3 GHz Quad-Core Intel Core i7) with macOS 15.1, but it didn't work: kvm_intel: VMX not supported by CPU 3

[Fred78290]

No you misunterstanded me. I don't say that VZ by default enabled nested virtualization, i just said that by default on any intel plateform (linux, windows...) it's is enabled by default with other virtualization engine like qemu and Hyper-V and other... with native host feature...

I think somewhere in the code you must have this: swift code

		if #available(macOS 15.0, *) {
			if VZGenericPlatformConfiguration.isNestedVirtualizationSupported {
				platform.isNestedVirtualizationEnabled = true
			}
		}

[jandubois]

I think the code you are looking for is

lima/pkg/vz/vm_darwin.go

Lines 240 to 258 in 021c974

	// nested virt
	if *driver.Instance.Config.NestedVirtualization {
	macOSProductVersion, err := osutil.ProductVersion()
	if err != nil {
	return fmt.Errorf("failed to get macOS product version: %w", err)
	}
	if macOSProductVersion.LessThan(*semver.New("15.0.0")) {
	return errors.New("nested virtualization requires macOS 15 or newer")
	}
	if !vz.IsNestedVirtualizationSupported() {
	return errors.New("nested virtualization is not supported on this device")
	}
	if err := platformConfig.SetNestedVirtualizationEnabled(true); err != nil {
	return fmt.Errorf("cannot enable nested virtualization: %w", err)
	}
	}

What do you suggest needs changing? I don't understand your feedback.

[Fred78290]

@jandubois @AkihiroSuda

Hello, I suggest that:

	// nested virt
	if macOSProductVersion, err := osutil.ProductVersion(); err == nil {
		if macOSProductVersion.Compare(*semver.New("15.0.0")) >= 0 && vz.IsNestedVirtualizationSupported() {
			if driver.Instance.Config.NestedVirtualization == nil {
				driver.Instance.Config.NestedVirtualization = ptr.Of(true)
			}

			if *driver.Instance.Config.NestedVirtualization {
				if err := platformConfig.SetNestedVirtualizationEnabled(true); err != nil {
					return fmt.Errorf("cannot enable nested virtualization: %w", err)
				}
			}
		}
	} else {
		return fmt.Errorf("failed to get macOS product version: %w", err)
	}

and remove in default.go

	if y.NestedVirtualization == nil {
		y.NestedVirtualization = ptr.Of(false)
	}

In this case, the nested feature is always enabled by default except if we decide to disabled it.

By the way we don't need to pass a yml config or modify existing config.

Your actual implementation mean that the user must check before running limactl if nested is supported else he got an error...

[balajiv113]

@Fred78290
I understand that you want a easier way to enable Nested Virtualization.

Problem in enabling by default is, it may have performance impact. For Instance, when we enable rosetta for vz there will be a performance impact which is expected. Similarly there may be some here as well.

If you want a one-liner to enable via command line argument, this should work

limactl create --set='.nestedVirtualization = true'

[Fred78290]

@Fred78290 I understand that you want a easier way to enable Nested Virtualization.

Problem in enabling by default is, it may have performance impact. For Instance, when we enable rosetta for vz there will be a performance impact which is expected. Similarly there may be some here as well.

If you want a one-liner to enable via command line argument, this should work

limactl create --set='.nestedVirtualization = true'

The problem is if I use this on MAc with unsupported nested virtualization, I got an error :(

Except Rosetta2 emulation, I'm not sure of performance impact. So if rosetta is enabled, by default nested could be disabled.

Well running VM X86 on MacOS silicon, I think that the user has already accepted degraded performance. And typically it's for developement environment. Isn't it?

[balajiv113]

The problem is if I use this on MAc with unsupported nested virtualization, I got an error :(

Can you share the error ?? After error is VM getting created or failed ?

Except Rosetta2 emulation, I'm not sure of performance impact.

My point is, similar to rosetta we might have performance impact here as well. Not mixing rosetta with nested Virtualization

[Fred78290]

Can you share the error ?? After error is VM getting created or failed ?

Just by reading the code :)

if *driver.Instance.Config.NestedVirtualization {
...
    if macOSProductVersion.LessThan(*semver.New("15.0.0")) { 
        return errors.New("nested virtualization requires macOS 15 or newer") 
    } 
 
    if !vz.IsNestedVirtualizationSupported() { 
        return errors.New("nested virtualization is not supported on this device") 
    }
...

[jandubois]

I think the best way to deal with this would be to create ~/.lima/_config/default.yaml on your M3 machine running macOS Sequoia:

nestedVirtualization: true

That way any machines created on that host will have nested virtualization enabled by default, unless you override it in the lima.yaml.

[afbjorklund]

And half of the point of using containers is to avoid things like nested virtualization, so it seems OK to have it opt-in.

[afbjorklund]

Well running VM X86 on MacOS silicon, I think that the user has already accepted degraded performance.

That's not really the use case of the Rosetta, it's more like running a few legacy binaries that haven't been rebuilt

Similar to how we used to have multilib support, to be able to run some old i386 binaries that weren't amd64 yet?

You can still emulate the entire system if you want to, like with RISC-V, and then you accept like 10x slower...