`mix firmware` failure during squashfs #876

jjcarstens · 2023-06-02T15:08:45Z

With squashfs 4.6+ on the host machine, some mix firmware builds may fail after building the rootfs and then squashing it

|nerves_bootstrap| Building OTP Release...

* skipping runtime configuration (config/runtime.exs not found)
* creating _build/rpi0_dev/rel/my_fw/releases/0.1.0/vm.args
Updating base firmware image with Erlang release...
Copying rootfs_overlay: .../_build/rpi0_dev/nerves/rootfs_overlay
Copying rootfs_overlay: .../my_fw/rootfs_overlay
** (Mix) Nerves encountered an error. %IO.Stream{device: :standard_io, line_or_bytes: :line, raw: true}

The workaround is to downgrade squashfs to 4.5 and things should work again.

We should investigate the breakage with squashfs 4.6 and how to mitigate

The text was updated successfully, but these errors were encountered:

LostKobrakai · 2023-06-02T15:19:07Z

The bit of debugging I did seemed to fail in a unsquashfs call of ~/.nerves/artifacts/our-fw/scripts/merge-squashfs

fhunleth · 2023-06-02T15:34:20Z

I could use some help reproducing this. I have had Squashfs 4.6.1 installed on two MBP's for quite a while now and haven't seen this at all. Does this affect all projects? Aside from work projects, I build Elixir Circuits Quickstart and Nerves Livebook frequently. Does it reproduce for either of you with those?

jjcarstens · 2023-06-02T15:36:11Z

It is a bit stalled until we have some reproduction. A few others are looking into it and we'll report once we have more

Part of nerves-project/nerves#876

Part of nerves-project/nerves#877 for adding more useful output during the firmware build process. nerves-project/nerves#876 shows cases where `mix firmware` fails to build. In reproducing, I found that `merge-squashfs` was failing silently. Specifically at this line because `2>/dev/null` was forwarding stderr to be ignored. Instead, this changes to let stderr through so more helpful messages can be reviewed

jjcarstens · 2023-06-27T14:28:55Z

I was able to reproduce this with a private repo. Once digging through the scripts and commands being run during squashing, this is what I ran into

FATAL ERROR: write_file: file squashfs-root/usr/lib/xtables/libip6t_hl.so already exists

Frank gave me the idea to check for files without case-sensitivity and sure enough, there are 2

> my_fw √ % grep -i libip6t_hl /var/folders/l9/61kntssd7g19yw5h4l9qxsr00000gn/T/tmp.P40vXDGq/pseudofile
/usr/lib/xtables/libip6t_HL.so m 755 root wheel
/usr/lib/xtables/libip6t_hl.so m 755 root wheel

MacOS is case-insensitive by default. This actually hits at a lots of random times. For cases during squashfs, it's probably running into files like this that are provided by the underlying Nerves system and is going to vary widely.

I need to think about how to handle this. I'm assuming the recent squashfs update has some sort of change to ignore case and just need to see how to work around it

Part of nerves-project/nerves#877 for adding more useful output during the firmware build process. nerves-project/nerves#876 shows cases where `mix firmware` fails to build. In reproducing, I found that `merge-squashfs` was failing silently. Specifically at this line because `2>/dev/null` was forwarding stderr to be ignored. Instead, this changes to let stderr through so more helpful messages can be reviewed

schrockwell · 2023-08-01T18:23:58Z

For folks who need to downgrade squashfs on macOS using Homebrew:

brew tap-new zappy/edith
brew extract --version="4.5.1" squashfs zappy/edith
brew install [email protected]
brew link --overwrite [email protected]
unsquashfs -version # => should report 4.5.1

SteffenDE · 2023-11-19T13:07:39Z

Just a warning that building a firmware on macOS actually leads to a somewhat corrupted system.

I recently spent a couple of hours trying to find out why a specific iptables command failed on a nerves firmware built on macOS using the downgraded version of squashfs that has been suggested. I later noticed that a system built on linux works just fine.

If you're seeing logs like these it could happen that some things will not work as expected:

Pseudo modify file "lib/modules/5.15.138/kernel/net/netfilter/xt_dscp.ko" does not exist in source filesystem.  Ignoring.
Pseudo modify file "lib/modules/5.15.138/kernel/net/netfilter/xt_hl.ko" does not exist in source filesystem.  Ignoring.
Pseudo modify file "lib/modules/5.15.138/kernel/net/netfilter/xt_rateest.ko" does not exist in source filesystem.  Ignoring.
Pseudo modify file "lib/modules/5.15.138/kernel/net/netfilter/xt_tcpmss.ko" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libip6t_hl.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libipt_ttl.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_connmark.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_dscp.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_mark.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_rateest.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_set.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_tcpmss.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_tos.so" does not exist in source filesystem.  Ignoring.

salseeg · 2023-12-16T11:38:50Z

I was able to reproduce this with a private repo. Once digging through the scripts and commands being run during squashing, this is what I ran into
FATAL ERROR: write_file: file squashfs-root/usr/lib/xtables/libip6t_hl.so already exists
Frank gave me the idea to check for files without case-sensitivity and sure enough, there are 2
> my_fw √ % grep -i libip6t_hl /var/folders/l9/61kntssd7g19yw5h4l9qxsr00000gn/T/tmp.P40vXDGq/pseudofile
/usr/lib/xtables/libip6t_HL.so m 755 root wheel
/usr/lib/xtables/libip6t_hl.so m 755 root wheel
MacOS is case-insensitive by default. This actually hits at a lots of random times. For cases during squashfs, it's probably running into files like this that are provided by the underlying Nerves system and is going to vary widely.

I need to think about how to handle this. I'm assuming the recent squashfs update has some sort of change to ignore case and just need to see how to work around it

@jjcarstens Have just added issue about this - #937
I worked around this by replacing mktemp and moving project to case sensitive volume

schrockwell · 2024-02-06T15:54:33Z

I went down this road again and came to the same conclusion as @salseeg. Would it make sense to add a $NERVES_TMP_DIR environment variable that can be passed to mktemp?

I also found that mktemp -d -p '' will honor $TMPDIR which should already exist, so there would be no need to define a special environment variable.

joserpintuitivo · 2024-04-19T20:56:29Z

For folks who need to downgrade squashfs on macOS using Homebrew:

brew tap-new zappy/edith
brew extract --version="4.5.1" squashfs zappy/edith
brew install [email protected]
brew link --overwrite [email protected]
unsquashfs -version # => should report 4.5.1

This is not working for me, I changed my Mac and installed version 4.6.1 on homebrew, I am not being able to downgrade to 4.5.1

This is a temporary fix for an issue that prevents building firmware on macOS if the Nerves system contains any files with names that collide on a case-insensitive filesystem like APFS. See nerves-project/nerves#876. When running `mix firmware`, warnings in the following format may indicate that some things may not work as expected: ``` Pseudo modify file "path/to/file" does not exist in source filesystem. Ignoring. ```

jjcarstens mentioned this issue Jun 2, 2023

Provide more helpful script output with mix firmware task errors #877

Closed

jjcarstens added a commit to nerves-project/nerves_system_br that referenced this issue Jun 27, 2023

Don't consume stderr in merge-squashfs

17cf6ca

Part of nerves-project/nerves#876

jjcarstens mentioned this issue Jun 27, 2023

Don't consume stderr in merge-squashfs nerves-project/nerves_system_br#736

Merged

This was referenced Feb 6, 2024

Force mktemp to honor $TMPDIR in merge-squashfs script nerves-project/nerves_system_br#778

Merged

Add docs for building systems on macOS, and $TMPDIR #951

Open

bjyoungblood mentioned this issue Jul 30, 2024

Run unsquashfs with force flag when merging rootfs nerves-project/nerves_system_br#807

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`mix firmware` failure during squashfs #876

`mix firmware` failure during squashfs #876

jjcarstens commented Jun 2, 2023

LostKobrakai commented Jun 2, 2023

fhunleth commented Jun 2, 2023

jjcarstens commented Jun 2, 2023

jjcarstens commented Jun 27, 2023

schrockwell commented Aug 1, 2023 •

edited

Loading

SteffenDE commented Nov 19, 2023

salseeg commented Dec 16, 2023 •

edited

Loading

schrockwell commented Feb 6, 2024

joserpintuitivo commented Apr 19, 2024

mix firmware failure during squashfs #876

mix firmware failure during squashfs #876

Comments

jjcarstens commented Jun 2, 2023

LostKobrakai commented Jun 2, 2023

fhunleth commented Jun 2, 2023

jjcarstens commented Jun 2, 2023

jjcarstens commented Jun 27, 2023

schrockwell commented Aug 1, 2023 • edited Loading

SteffenDE commented Nov 19, 2023

salseeg commented Dec 16, 2023 • edited Loading

schrockwell commented Feb 6, 2024

joserpintuitivo commented Apr 19, 2024

`mix firmware` failure during squashfs #876

`mix firmware` failure during squashfs #876

schrockwell commented Aug 1, 2023 •

edited

Loading

salseeg commented Dec 16, 2023 •

edited

Loading