Apple deployment target, SDK root and SDK version

[madsmtm]

I've been looking into our handling of the deployment targets and SDK paths on Apple platforms, and I've found it to be somewhat inconsistent and in some places outright incorrect. I'm opening this issue to give context to the PRs I've been (and will be) opening to fix it.

Deployment targets

The "deployment target" is the minimum operating system version that the final binary will work on. It can be configured with the *_DEPLOYMENT_TARGET environment variables, and setting it allows us to enable certain optimizations. Currently this is done primarily in the codegen backend LLVM, though it could also be done by Cranelift or GCC if they wanted to, and also by std or other library crates.

By default, Clang takes the default deployment target from the SDK, which is usually quite high. rustc chooses a different approach here, and compiles by default for the minimum supported version, which I believe is the right choice, as it forces users to use the correct *_DEPLOYMENT_TARGET if they need to use newer features.

I've aligned the default/minimum versions of this variable with Clang in #129367, and made sure we rebuild when the user changes it in #129342.

SDKROOT

The SDK root contains system header files and linker Text-Based stub files (.tbd). The SDK path is passed to the static linker (ld) so that it can tell which system library a given symbol comes from, and in turn write this information in the final Mach-O, so that it can be read by the dynamic linker (dyld) at runtime.

System libraries have been "hidden away" in the dyld cache since macOS 11 Big Sur, and since Rust does not distribute the linker stubs (unlike e.g. zig cc does on macOS), the SDK root is effectively always required to link anything (whether it's a cross-compile or not).

The correct way to invoke e.g. Clang, then, is using xcrun to pass the desired SDK in the SDKROOT environment variable, e.g.:

$ xcrun --sdk iphoneos clang -target aarch64-apple-ios-macabi foo.c -o foo

This is not the full story, however: The binary at /usr/bin/clang is actually a trampoline that (effectively) invokes xcrun and then calls out to the actual clang binary distributed with Xcode, which means that a plain clang foo.c command usually works, at least when compiling for the host macOS. (In the case where you've compiled Clang from source instead, then this won't work, and you have to provide the SDK root).

rustc is in a bit of a trickier position than Clang though:

• We're not a system built-in under /usr/bin, and as such don't get the affordances of automatically having SDKROOT set.
• The design of Cargo means that rustc will be invoked for different targets, but with the same SDKROOT. E.g. compiling build scripts when running Cargo under Xcode, SDKROOT will usually be set for an iOS SDK, while the build script will be targetting the host macOS.
• We try to make it easy to cross-compile by default, so we want to figure out the SDK root from the target instead of forcing the user to specify it.

Some of this already works (and kudos to the people that have implemented this in the past), but it's currently incomplete, which means we as a stop-gap end up shelling out to xcrun to let it figure out a SDK root for us. I believe that rustc should, when linking, re-implement the SDK discovery logic that xcrun does, see #131433, and should always set the SDK root when invoking the linker, see #131477.

SDK version and LC_BUILD_VERSION

The SDK root is also used for something else in Clang: To find the SDK version, and embed it, together with the deployment target, in the LC_BUILD_VERSION of the produced (Mach-O) binary. It is quite important that this load command is present, otherwise the linker may refuse to link the binary (see #114114 and #111384), as it cannot reliably figure out the target OS and ABI.

The SDK version is used by dyld to emit more errors on newer binaries (see here and here), but also by system frameworks internally to change behaviour when compiled for an older SDK, for example -[NSView wantsBestResolutionOpenGLSurface]. I suspect the deployment target to be used to similar purposes.

#129369 ensures that we consistently set the deployment target for all produced binaries.

The SDK version is set differently by rustc depending on what kind of binary is being produced:

• Raw object files: Delegate to the codegen backend (LLVM sets it to 0 or n/a).
• Other object files: Currently a hard-coded value, but I've changed it to 0 in Apple: Do not specify an SDK version in rlib object files #131016 to match LLVM.
• Everything linked with cc: Give Clang the SDK root, and let them figure it out.
• Everything linked with ld: Set to be equal to the deployment target. I think a better approach here would be to read the SDK version from the SDKSettings.json, to match what Clang does. I have implemented this in WIP: Parse Apple SDK versions #131478.

Conclusion

Within the constraints that we have (Cargo is target agnostic, and won't deal with this, though it's better positioned to do so IMO), I think that the approach that rustc takes is a fairly good, and would be even better with a few fixes ;).

Just to clarify, in the end, rustc's dependency on these variables (i.e. the effect on incremental compilation) would be roughly:

• *_DEPLOYMENT_TARGET: Codegen.
• SDKROOT/DEVELOPER_DIR//var/db/xcode_select_link/...: Linking.

That is, ideally cargo check shouldn't be influenced by them at all, and cargo build only re-link the last binaries if the SDK root changed, and should recompile all binaries if the deployment target changed.

We will also need to ensure that the cc crate also handles all of this correctly too (see for example #128419 for a current issue) (it's much more difficult there, since they have backwards compatibility issues).

---

@rustbot label O-apple

CC the Apple experts I know of:
@simlay, @BlackHoleFox, @thomcc, @shepmaster