WIP: Parse Apple SDK versions

[madsmtm]

High-level goals of this PR:

1. Easy cross compile on macOS host to iOS, tvOS, watchOS and visionOS.
   • If SDKROOT is specified for such a target, we need to find a macOS SDK to compile build scripts etc. for the host.
2. Support cross compile to older macOS targets.
   • Similar to above, we are potentially given an SDK that is not suitable for the host machine (e.g. Aarch64 to x86 with the macOS 10.13 SDK that does not support Aarch64).
3. Support cross compile on non-macOS host.
   • Give a warning if no SDK root is set, and guide the user towards setting it.
   • Should still support zig cc (though that maybe only works on macOS? I don't think Zig provides the .tdbs and headers to build and link iOS?)

---

Previously, we did a simplistic check against the SDKROOT path to try to figure out whether we should use the environment variable, or try to find a better matching SDK instead.

This is brittle since the user might want their SDK to be in a different place in the file system (that's kinda the whole idea of the SDKROOT variable), and also didn't work properly with SDKs in the Xcode Command Line Tools.

Instead, we now:

1. Parse the SDKSettings.json file under the SDKROOT, to determine whether it is suitable for the target we're compiling for (which it might not be when e.g. using Cargo, and targetting iOS but compiling build scripts on macOS).
2. If it's not, attempt to find another SDK.

This allows us to give much more detailed error messages, to better support cross-compile scenarios, and allows using parameters from the SDK, such as the SDK version, which we need to pass to ld64 for correctness (this fixes the remaining difference between linking with cc and ld64).

Final part of #129432.

Additionally, this fixes #80120 by introducing better error messages, so that it's possible to more clearly see when /Library/Developer/CommandLineTools is invalid.

Test

Tested with:

./x test tests/run-make/apple-* --target="aarch64-apple-darwin,aarch64-apple-ios,aarch64-apple-ios-macabi,aarch64-apple-ios-sim,aarch64-apple-tvos,aarch64-apple-tvos-sim,aarch64-apple-visionos,aarch64-apple-visionos-sim,aarch64-apple-watchos,aarch64-apple-watchos-sim,arm64_32-apple-watchos,armv7k-apple-watchos,armv7s-apple-ios,x86_64-apple-darwin,x86_64-apple-ios,x86_64-apple-ios-macabi,x86_64-apple-tvos,x86_64-apple-watchos-sim,x86_64h-apple-darwin"
IPHONEOS_DEPLOYMENT_TARGET=10.0 ./x test tests/run-make/apple-* --target=i386-apple-ios
MACOSX_DEPLOYMENT_TARGET=10.12 SDKROOT=$(pwd)/MacOSX10.13.sdk ./x test tests/run-make/apple-* --target=i686-apple-darwin

Which includes every Apple target except for arm64e.

Note that using the MacOSX10.13.sdk now works (and should work even more reliably once this change hits beta).

TODO

• Manually test a few more scenarios:
  • Using cargo +stage1 inside Xcode for a project with both build scripts and proc macros (namely Bevy's).
  • Use zig cc without any SDKs available.
  • Cross compile with zig cc from a non macOS host.
  • Set SDKROOT to different crazy values.
  • Test running xcrun --sdk XYZ rustc +stage1.
• Clean up diagnostic setup (and use subdiagnostics for certain things?).
• Fix bootstrap so that it doesn't pass SDKROOT by default
  • Not sure if this is possible, since we also want the user to be able to override SDKROOT when building with bootstrap?
• Test properly with i686

CC @BlackHoleFox @thomcc

Currently an unholy git combination of #131037, #131433 and #131477, so:
@rustbot blocked

[rustbot]

r? @jieyouxu

rustbot has assigned @jieyouxu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rust-log-analyzer]

The job mingw-check-tidy failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/

# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
           --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---
#13 2.789 Building wheels for collected packages: reuse
#13 2.790   Building wheel for reuse (pyproject.toml): started
#13 3.036   Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 3.037   Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132715 sha256=dfa09868353292d98f811d3efdb0d54d07389e808efc71d68e3b93c514bf8bec
#13 3.037   Stored in directory: /tmp/pip-ephem-wheel-cache-z57sj4k1/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#13 3.040 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#13 3.429 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#13 3.430 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#13 3.943 Collecting virtualenv
#13 3.943 Collecting virtualenv
#13 4.003   Downloading virtualenv-20.26.6-py3-none-any.whl (6.0 MB)
#13 4.294      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 20.8 MB/s eta 0:00:00
#13 4.352 Collecting filelock<4,>=3.12.2
#13 4.362   Downloading filelock-3.16.1-py3-none-any.whl (16 kB)
#13 4.394 Collecting platformdirs<5,>=3.9.1
#13 4.403   Downloading platformdirs-4.3.6-py3-none-any.whl (18 kB)
#13 4.425 Collecting distlib<1,>=0.3.7
#13 4.447      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 45.4 MB/s eta 0:00:00
#13 4.447      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 45.4 MB/s eta 0:00:00
#13 4.525 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 4.712 Successfully installed distlib-0.3.9 filelock-3.16.1 platformdirs-4.3.6 virtualenv-20.26.6
#13 DONE 4.8s

#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
---
DirectMap4k:      235456 kB
DirectMap2M:    11298816 kB
DirectMap1G:     7340032 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test            --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
    Finished `dev` profile [unoptimized] target(s) in 0.05s
##[endgroup]
downloading https://static.rust-lang.org/dist/2024-09-22/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
extracting /checkout/obj/build/cache/2024-09-22/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz to /checkout/obj/build/x86_64-unknown-linux-gnu/rustfmt
---
fmt check
Diff in /checkout/compiler/rustc_codegen_ssa/src/apple.rs:384:
     /// This is needed by the linker.
     ///
     /// settings["Version"] or settings["VersionMap"]["macOS_iOSMac"][settings["Version"]].
-    pub(crate) fn sdk_version(&self, target: &Target, sdkroot: &Path) -> Result<AppleOSVersion, AppleSdkError> {
+    pub(crate) fn sdk_version(
+        target: &Target,
+        sdkroot: &Path,
+        sdkroot: &Path,
+    ) -> Result<AppleOSVersion, AppleSdkError> {
         if target.abi == "macabi" {
                 .version_map
                 .version_map
fmt error: Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/rustfmt/bin/rustfmt" "--config-path" "/checkout" "--edition" "2021" "--unstable-features" "--skip-children" "--check" "/checkout/compiler/rustc_infer/src/infer/region_constraints/mod.rs" "/checkout/compiler/rustc_infer/src/infer/region_constraints/leak_check.rs" "/checkout/compiler/rustc_infer/src/infer/lexical_region_resolve/mod.rs" "/checkout/compiler/rustc_infer/src/infer/type_variable.rs" "/checkout/compiler/rustc_codegen_ssa/src/base.rs" "/checkout/compiler/rustc_codegen_ssa/src/assert_module_sources.rs" "/checkout/compiler/rustc_codegen_ssa/src/debuginfo/mod.rs" "/checkout/compiler/rustc_codegen_ssa/src/debuginfo/type_names.rs" "/checkout/compiler/rustc_infer/src/infer/canonical/instantiate.rs" "/checkout/compiler/rustc_infer/src/infer/canonical/mod.rs" "/checkout/compiler/rustc_infer/src/infer/canonical/query_response.rs" "/checkout/compiler/rustc_infer/src/infer/canonical/canonicalizer.rs" "/checkout/compiler/rustc_infer/src/infer/opaque_types/mod.rs" "/checkout/compiler/rustc_infer/src/infer/opaque_types/table.rs" "/checkout/compiler/rustc_infer/src/lib.rs" "/checkout/compiler/rustc_codegen_ssa/src/apple/tests.rs" "/checkout/compiler/rustc_codegen_ssa/src/common.rs" "/checkout/compiler/rustc_codegen_ssa/src/mono_item.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/coverageinfo.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/locals.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/mod.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/rvalue.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/statement.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/analyze.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/intrinsic.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/debuginfo.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/operand.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/place.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/constant.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/block.rs" "/checkout/compiler/rustc_codegen_ssa/src/codegen_attrs.rs" "/checkout/compiler/rustc_codegen_ssa/src/size_of_val.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/coverageinfo.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/misc.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/abi.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/mod.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/type_.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/backend.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/builder.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/declare.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/asm.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/intrinsic.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/debuginfo.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/statics.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/consts.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/write.rs" "/checkout/compiler/rustc_codegen_ssa/src/errors.rs" "/checkout/compiler/rustc_codegen_ssa/src/apple.rs" "/checkout/compiler/rustc_codegen_ssa/src/meth.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/link.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/archive.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/mod.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/rpath/tests.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/metadata.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/rpath.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/command.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/symbol_export.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/lto.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/write.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/linker.rs" "/checkout/compiler/rustc_codegen_ssa/src/target_features.rs" "/checkout/compiler/rustc_codegen_ssa/src/lib.rs" "/checkout/compiler/rustc_mir_transform/src/mentioned_items.rs" "/checkout/compiler/rustc_infer/src/infer/resolve.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
  local time: Thu Oct 10 01:38:15 UTC 2024
  network time: Thu, 10 Oct 2024 01:38:15 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

[jieyouxu]

• Fix bootstrap so that it doesn't pass SDKROOT by default
  • Not sure if this is possible, since we also want the user to be able to override SDKROOT when building with bootstrap?

bootstrap can possibly handle that as some kind of target specific config option, depending on the specifics of what you want.

[bors]

☔ The latest upstream changes (presumably #132497) made this pull request unmergeable. Please resolve the merge conflicts.