Firefox does not build with "gentoo" or mixed "gentoo"/"unknown" rust targets

[stefson]

x86_64-unknown-linux-musl became x86_64_gentoo_linux_musl
armv7a-unknown-linux-musleabihf became armv7a_gentoo_linux_musleabihf

Now, it is introduced by this patch: 0022-Add-gentoo-target-specs.patch

There isn't much of a description in that patch header, why this was introduced and so on. Is this related to the use of system_llvm? Can I override it with the old rusthost?

The reason for asking is, that the firefox config scripts know how to detect the old rusthost. With the new rusthost the scripts fail to detect rust and hang.

[smaeul]

The reason is that the previous patchset was not acceptable to upstream. Historically, the *-unknown-linux-musl* targets have been static-linking only. After some work I did for 1.21.0, they allowed dynamic linking, but still defaulted to static linking. The patchset used for the Gentoo builds changed that default; so the behavior for rustc --target x86_64-unknown-linux-musl was different on Gentoo compared to everywhere else.

So to avoid patching the generic -unknown- target, I had to add a new target with the patched behavior. It also removed the need for all of the string mangling in the ebuild. It looks like unfortunately this will require patching Firefox again. However, the patch is quite simple: avoid the string mangling there as well, and use CHOST (host_or_target in rust.configure) directly as the rust triple.

[smaeul]

Would it be helpful to ship the official static rust targets in addition to the default dynamic ones? I would guess that the static targets would work fine for firefox, since it's only compiling libraries anyway.

[stefson]

I've been using this patch for some time now, and a similar one for armv7:

--- a/build/moz.configure/rust.configure
+++ b/build/moz.configure/rust.configure
@@ -190,9 +190,12 @@
             die("Don't know how to translate {} for rustc".format(
                 host_or_target.alias))
 
+        # Create a gentoo alias to match the rustc target on musl
+        gentoo_alias = rustc_target.alias.replace('unknown','gentoo')
+
         # Check to see whether our rustc has a reasonably functional stdlib
         # for our chosen target.
-        target_arg = '--target=' + rustc_target.alias
+        target_arg = '--target=' + gentoo_alias
         in_fd, in_path = mkstemp(prefix='conftest', suffix='.rs')
         out_fd, out_path = mkstemp(prefix='conftest', suffix='.rlib')
         os.close(out_fd)
@@ -229,7 +232,7 @@
             os.remove(out_path)
 
         # This target is usable.
-        return rustc_target.alias
+        return gentoo_alias
 
     return rust_target

In the longterm I should file a bug at mozilla and try to rework the detection. Since the patch is small and not a burden to maintain, I don't see any need to press this one.

About your question, I'm actually not sure how your rust stages behave, if they link statically or dynamically, but it gives a working firefox binary :)

[stefson]

@smaeul

I'm having serious trouble to get things going on cross compiling with rust-std.

librustc_target/spec/mod.rs says:

    ("aarch64-gentoo-linux-musl", aarch64_gentoo_linux_musl),
    ("arm-unknown-linux-musleabi", arm_gentoo_linux_musleabi),
    ("armv7a-unknown-linux-musleabihf", armv7a_gentoo_linux_musleabihf),
    ("i686-gentoo-linux-musl", i686_gentoo_linux_musl),
    ("powerpc-gentoo-linux-musl", powerpc_gentoo_linux_musl),
    ("powerpc64-gentoo-linux-musl", powerpc64_gentoo_linux_musl),
    ("x86_64-gentoo-linux-musl", x86_64_gentoo_linux_musl),

the missmatch between the chost syntax is making my head spin, so can you provide support for armv7a-gentoo-linux-musleabihf, please?

I'm going to try the following approach, do you think it may work?

diff --git a/src/librustc_target/spec/mod.rs b/src/librustc_target/spec/mod.rs
index 42bd26c..07d68e7 100644
--- a/src/librustc_target/spec/mod.rs
+++ b/src/librustc_target/spec/mod.rs
@@ -332,7 +332,7 @@ mod arm_unknown_linux_musleabi;
 supported_targets! {
     ("aarch64-gentoo-linux-musl", aarch64_gentoo_linux_musl),
     ("arm-unknown-linux-musleabi", arm_gentoo_linux_musleabi),
-    ("armv7a-unknown-linux-musleabihf", armv7a_gentoo_linux_musleabihf),
+    ("armv7-gentoo-linux-musleabihf", armv7a_gentoo_linux_musleabihf),
     ("i686-gentoo-linux-musl", i686_gentoo_linux_musl),
     ("powerpc-gentoo-linux-musl", powerpc_gentoo_linux_musl),
     ("powerpc64-gentoo-linux-musl", powerpc64_gentoo_linux_musl),

P.S: anything related to cargo build --target armv7a-unknown-linux-musleabihf is working, but firefox (and librsvg in the future) are more complicated.

[smaeul]

@stefson The target names there come straight from the gentoo profiles. You can change them to whatever you want if that works better (your patch looks fine). Is the issue because firefox doesn't recognize "armv7a" as your architecture?

I mentioned before that I want to ship the official targets alongside the distro targets, and I think that will fix your issues. Sorry, I haven't gotten a chance to implement that yet.

[stefson]

The problem is the rust.configure script shipped by firefox. During detection of rust and cargo, it can't deal with the missmatch of armv7a-unknown-linux-musleabihf and x86_64-gentoo-linux-musl.

aarch64 for instance should be much easier to deal with: aarch64-gentoo-linux-musl vs x86_64-gentoo-linux-musl

The script can be easily patched to accept gentoo instead of unknown, my silly little patch from above does that and it works in this regard. (alpine does this too with their x86_64-alpine-linux-musl rusthost)

[smaeul]

Please let me know if my latest changes help any (and if the thumb code works).

[stefson]

I can compile test the new thumbv7neon target, but can you explain to me how to runtime test it? It's not going to work with firefox out of the box for sure :) and also I don't really see that the missmatch in the rust chost has been dealed with?

[smaeul]

If I understand the problem correctly, the change should solve the mismatch because the unknown target (the official upstream one) is now always available. So ideally you can drop the patch that was replacing unknown with gentoo.

[stefson]

ok, I'll give it a shot, but it's going to take a while. In the meantime, do you know how to tell if the neon and thumb stuff are working? A simple hello world type of thing? Also: which gcc are you using to build the stage0's? There's a llvm codegen bug in upstream's rust-1.38.0 stage0's, affecting armv7 only.

[stefson]

I'm sorry, but it still doesn't work with the new patches:

 0:05.66 checking whether the C++ compiler supports -Wformat-overflow=2... yes
 0:05.68 checking whether the C compiler supports -Wno-gnu-zero-variadic-macro-arguments... no
 0:05.70 checking whether the C++ compiler supports -Wno-gnu-zero-variadic-macro-arguments... no
 0:05.72 checking whether the C++ compiler supports -fno-sized-deallocation... yes
 0:05.72 checking for rustc... /usr/bin/rustc
 0:05.72 checking for cargo... /usr/bin/cargo
 0:05.88 checking rustc version... 1.38.0
 0:05.90 checking cargo version... 1.38.0
 0:06.04 checking for rust target triplet... armv7a-unknown-linux-musleabihf
 0:06.12 checking for rust host triplet... x86_64-unknown-linux-musl
 0:06.12 ERROR: The rust compiler host (x86_64-gentoo-linux-musl) is not suitable for the configure host (x86_64-gentoo-linux-musl/x86_64-unknown-linux-musl).
 0:06.16 *** Fix above errors and then restart with\
 0:06.16                "./mach build"
 0:06.16 gmake: *** [client.mk:115: configure] Error 1
 * ERROR: www-client/firefox-68.2.0::gentoo failed (configure phase):

it really seems that you will have to harmonize the rust host's: either x86_64-gentoo-linux-musl and armv7a-gentoo-linux-musl, or to stick to unknown instead. And well, not sure yet how to deal with the thumbv7neon target -.-

[smaeul]

Looks like I'll have to set up a Firefox cross-build environment to figure this out. I don't have one at the moment. It'll be at least a couple of weeks before I have a chance to look at this again, sorry.

[stefson]

If you want to I can send you a copy of my /usr/armv7a-unknown-linux-musleabihf/ rootfs, tared up it's some 250mb. In any case, the problem here is the ambiguity of the rust chost names. All of the targets can be easily renamed, but I'm unaware of the consequences.

[stefson]

two things I forgot: you'll need a binary pkg of dev-libs/nss compiled on real harddware (gentoo #699696) - if you don't have that, I can send you one. And I've got a little overlay for my personal stuff, my firefox ebuild works in terms of cross compiling. Those from tree propably not so much.

[anarchpenguin]

This is moot with current firefox builds, since 72.0 the detection has been properly handled in firefox. This is only an issue in esr and mozilla project has no plans to backport fixes for detection, the next esr will handle it properly.

[stefson]

I doubt it's fixed for armv7+neon, but will give it a try nonetheless.

[stefson]

not working with neon instructions, as of today, since it's looking for the gnueabihf thumbv7neon bindings

These are the packages that would be merged, in order:

[ebuild     U #] www-client/firefox-72.0.2::gentoo [68.3.0::gentoo] USE="system-icu system-jpeg (system-libevent) system-sqlite system-webp -bindist -clang -custom-cflags -custom-optimization -debug -eme-free -geckodriver -gmp-autoupdate (-hardened) -hwaccel -jack (-lto) (-pgo) -pulseaudio -screenshot (-selinux) -startup-notification -system-av1 -system-libvpx -test -wayland -wifi (-dbus%) (-system-harfbuzz%*)" CPU_FLAGS_ARM="neon*" L10N="de -ach -af -an -ar -ast -az -be -bg -bn -br -bs -ca -cak -cs -cy -da -dsb -el -en-CA -en-GB -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -ia -id -is -it -ja -ka -kab -kk -km -kn -ko -lij -lt -lv -mk -mr -ms -my -nb -nl -nn -oc -pa -pl -pt-BR -pt-PT -rm -ro -ru -si -sk -sl -son -sq -sr -sv -ta -te -th -tr -uk -ur -uz -vi -xh -zh-CN -zh-TW" 0 KiB

Total: 1 package (1 upgrade), Size of downloads: 0 KiB

 1:31.36 checking rustc version... 1.39.0
 1:31.51 checking cargo version... 1.39.0
 1:32.79 checking for rust target triplet...
 1:32.80 DEBUG: Creating `/var/tmp/portage/www-client/firefox-72.0.2/temp/conftestxHKlZk.rs` with content:
 1:32.80 DEBUG: | pub extern fn hello() { println!("Hello world"); }
 1:32.81 DEBUG: Executing: `/usr/bin/rustc --crate-type staticlib --target=thumbv7neon-unknown-linux-gnueabihf -o /var/tmp/portage/www-client/firefox-72.0.2/temp/conftestbbr0Ca.rlib /var/tmp/portage/www-client/firefox-72.0.2/temp/conftestxHKlZk.rs`
 1:32.81 DEBUG: The command returned non-zero exit status 1.
 1:32.82 DEBUG: Its error output was:
 1:32.82 DEBUG: | error[E0463]: can't find crate for `std`
 1:32.83 DEBUG: |   |
 1:32.83 DEBUG: |   = note: the `thumbv7neon-unknown-linux-gnueabihf` target may not be installed
 1:32.84 DEBUG: |
 1:32.84 DEBUG: | error: aborting due to previous error
 1:32.85 DEBUG: |
 1:32.85 DEBUG: | For more information about this error, try `rustc --explain E0463`.
 1:32.86 ERROR: Cannot compile for armv7a-unknown-linux-musleabihf with /usr/bin/rustc
 1:32.86 The target may be unsupported, or you may not have
 1:32.86 a rust std library for that target installed. Try:
 1:32.87 
 1:32.87   rustup target add thumbv7neon-unknown-linux-gnueabihf
 1:32.88 
 1:33.12 *** Fix above errors and then restart with\
 1:33.13                "./mach build"
 1:33.13 gmake: *** [client.mk:115: configure] Error 1

[anarchpenguin]

Open a proper bug for arm breakage, this is not broken on ppc32 ppc64 arm64 or amd64 using smaeul's overlay.

0:11.11 checking for rustc... /usr/bin/rustc
0:11.11 checking for cargo... /usr/bin/cargo
0:11.65 checking rustc version... 1.39.0
0:11.67 checking cargo version... 1.39.0
0:12.26 checking for rust target triplet... x86_64-gentoo-linux-musl
0:12.34 checking for rust host triplet... x86_64-gentoo-linux-musl

[stefson]

This isn't your overlay, and neither is this your bug. So I'm not sure why you even bother to comment. But if you need to know: there's #684896 for adding additional thumbv7neon target

[smaeul]

@stefson It looks like you added this target spec change in 1.49.0. Does that resolve the issue for you?

[stefson]

I tested spidermonkey, which ships the same patchset as firefox, with rust-1.49

it works fine for rust_host x86_64-gentoo-linux-musl, but doesn't for armv7a-unknown-linux-musleabihf

need more time to solve this