poudriere bulk fails to proceed a long list of dependencies on aarch64

[fluffykhv]

Got a reproducible bug, only with poudriere-devel but not release package

try to build something big

poudriere bulk -j 150aarch64 -tr x11/kde5
[skip]
[00:00:39] Recording filesystem state for prepkg... done
[00:00:39] Processing PRIORITY_BOOST
Error: (78387) mkdir:pkgqueue_move_ready_to_pool:269: 115Z114Z113Z112Z111Z110Z109Z108Z107Z106Z105Z104Z103Z102Z101Z100Z99Z98Z97Z96Z95Z94Z93Z92Z91Z90Z89Z88Z87Z86Z85Z84Z83Z82Z81Z80Z79Z78Z77Z76Z75Z74Z73Z72Z71Z70Z69Z68Z67Z66Z65Z64Z63Z62Z61Z60Z59Z58Z57Z56Z55Z54Z53Z52Z51Z50Z49Z48Z47Z46Z45Z44Z43Z42Z41Z40Z39Z38Z37Z36Z35Z34Z33Z32Z31Z30Z29Z28Z27Z26Z25Z24Z23Z22Z21Z20Z19Z18Z17Z16Z15Z14Z13Z12Z11Z10Z9Z8Z7Z6Z5Z4Z3Z2Z1Z0: File name too long
Error: (32676) /usr/local/share/poudriere/bulk.sh:pkgqueue_move_ready_to_pool:269: set -e error: status = 1
[00:00:40] Cleaning up

Environment

• Host OS [15.0 aarch64]:
• Jail OS [15.0 aarch64]:
• Poudriere Version [poudriere-git-3.4.99.20240811]:
• Ports branch and revision [head]:

[fluffykhv]

Looks like it fails at

poudriere/src/share/poudriere/include/pkgqueue.sh

Lines 566 to 570 in fca4bc3

	set)
	POOL_BUCKET_DIRS="$(echo "${PKGQUEUE_PRIORITIES}" |
	tr ' ' '\n' | LC_ALL=C sort -run |
	paste -d ' ' -s -)"
	;;

If I comment this block, build runs fine.

Reproducible only on aarch64, amd64/i386 machines are fine.

@bdrewery @bapt please take a look

[fluffykhv]

Indeed, on long lists paste -d ' ' -s - produces borged result with 'Z' symbol instead of space (' ') as mentioned in first message in thread.

Replace it with pr -m -t -s' ' - resolves issue and produce a valid list of numbers

patch-src_share_poudriere_include_pkgqueue.sh.patch

[bdrewery]

The 'ZZZ' thing was showing up for OPTIONS for @bapt too. If paste is giving Z then it has a bug that needs to be fixed, or we need to pass a proper LANG value. Switching to another tool isn't the right fix.

Can you try passing LC_ALL=C to paste too?

[fluffykhv]

The 'ZZZ' thing was showing up for OPTIONS for @bapt too. If paste is giving Z then it has a bug that needs to be fixed, or we need to pass a proper LANG value. Switching to another tool isn't the right fix.

Can you try passing LC_ALL=C to paste too?

Tried this now but no luck, bug is reproducible :(

LC_ALL=C paste -d gives me the same ZZZZ error. C.UTF-8 is also fails.

[bdrewery]

Someone pointed out that Z is 0x5a which is jemalloc's uninitialized memory canary. So there's a real problem here with paste(1). I'll keep this issue open for discussion as that is looked into.

[bdrewery]

If it only happens in poudriere-devel then it may be a problem with the way we have paste(1) imported into sh as a builtin.

[bdrewery]

Looks like this is only in poudriere-devel. 4a3460c

commit 4a3460c62616ee643e14cdabdc96620d4ad4ec9f
Author: Bryan Drewery <[email protected]>
Date:   Tue Jan 24 14:13:52 2023 -0800

    sh: Import paste(1) builtin from FreeBSD main 9bdb5158d

[bdrewery]

I think I see the problem. Trying to find a way to test it.

[bdrewery]

I definitely recreated some problems and will prioritize a fix for this ASAP.