Consider symlinks as potential provides for so: #1662

smoser · 2024-11-21T16:59:08Z

Some packages provide symlinks in /usr/lib/ that point to the library installed in another path.
Examples:

lua5.3 usr/lib/liblua-5.3.so.0 -> lua5.3/liblua-5.3.so.0
lua5.4 usr/lib/liblua-5.4.so.0 -> lua5.4/liblua-5.4.so.0
llvm17 usr/lib/libRemarks.so.16 -> ../lib/llvm17/lib/libRemarks.so.17

The total number of these cases in wolfi is only 12 links from a system lib dir to elsewhere
across 5 origins:

llvm15-15.0.7-r7
llvm16-16.0.6-r3
llvm17-17.0.6-r2
lua5.3-5.3.6-r5
lua5.4-5.4.7-r1

The result was that those packages don't get listed as providing the library that the do provide.

The target of the symlink above previously would get considered itself, but the fact that the target was not in isInDir(libDirs) meant it would get marked as vendored.

It meant that packages that depend on liblua-5.4.so.0 can't really resolve their dependency.

smoser · 2024-11-21T17:05:17Z

not all the fails seem related to may change. #1663 is a test rebuild of main.

imjasonh

This definitely needs a test. Changes to SCA-generated deps tend to break in unexpected and confusing ways, and historically we haven't had very good tests on hand to diagnose that.

xnox

In general, yes approved.

A file, or symlink, in public location is a publicly accessible library that generally linked libraries can find.

But yes, need to mount full wolfi; and execute melange scan --diff across all packages that provide .so files.

imjasonh · 2024-11-21T17:27:26Z

Fixed the test failures in #1664 , rebasing should see some green.

Some packages provide symlinks in /usr/lib/ that point to the library installed in another path. Examples: lua5.3 usr/lib/liblua-5.3.so.0 -> lua5.3/liblua-5.3.so.0 lua5.4 usr/lib/liblua-5.4.so.0 -> lua5.4/liblua-5.4.so.0 llvm17 usr/lib/libRemarks.so.16 -> ../lib/llvm17/lib/libRemarks.so.17 The total number of these cases in wolfi is only 12 links across 7 origins: llvm-18-18.1.8-r3 llvm-19-19.1.4-r0 llvm15-15.0.7-r7 llvm16-16.0.6-r3 llvm17-17.0.6-r2 lua5.3-5.3.6-r5 lua5.4-5.4.7-r1 The result was that those packages don't get listed as providing the library that the _do_ provide. The target of the symlink above previously would get considered itself, but the fact that the target was not in isInDir(libDirs) meant it would get marked as vendored. It meant that packages that depend on liblua-5.4.so.0 can't really resolve their dependency.

smoser · 2024-11-21T19:13:10Z

OK. Gave this a pretty good test, here is what I found.

the wonderfully named 'go-now' program is:

#!/bin/sh
if [ -z "$PROCESS" ]; then
    jobs=32
    outd=$1
    shift
    mkdir -p "$outd"
    ${MELANGE:-melange} version > "$outd/melange-version.txt"
    
    PROCESS=true exec parallel \
       "--jobs=$jobs" --line-buffer -- \
       "$0" "$outd" ::: "$@"
fi


outd="$1"
pkg="$2"

pkg=${pkg%.yaml}
out=$outd/$pkg.out
err=$outd/$pkg.err
rcf=$outd/$pkg.rc
tmpout="$outd/.$pkg.out"
tmperr="$outd/.$pkg.err"
tmprcf="$outd/.$pkg.rc"

cleanup() {
    rm -f "$tmpout" "$tmperr" "$tmprcf"
}
fail() { echo "$@" 1>&2; exit 99; }

trap cleanup EXIT
if [ -f "$out" ] && [ -f "$err" ] && [ -f "$rcf" ]; then
    read rc < $rcf
    echo "$pkg - $rc [already done]"
    exit 0
fi

melange=${MELANGE:-melange}
$melange scan --diff "$pkg.yaml" >"$tmpout" 2>"$tmperr"
rc=$?
echo "$rc" > "$tmprcf" || fail "failed writing to tmprcf"
echo "$pkg - $rc"

mv "$tmpout" "$out" &&
    mv "$tmperr" "$err" &&
    mv "$tmprcf" "$rcf" ||
    fail "odd: failed renames - $pkg"
exit $rc

I used it to run melange scan --diff against *.yaml in wolfi at the commit listed.

$ git log HEAD^.. --oneline --no-decorate 
a3e5a2aa8 php-8.3-pecl-http - enable updates and update to 4.2.6 (#34749)

$ ls -l packages
lrwxrwxrwx    1 user     docker          50 Nov 21 12:18 packages -> /gcsfuse/wolfi-production-registry-destination/os/

$ for m in main mine; do d=out.$m; time env MELANGE=$HOME/src/melange/melange.$m ./go-now $d *.yaml; done

Then collect the differences in output, using

diff -ru '--exclude=*.err' out.main/llvm15.out out.mine/llvm15.out
--- out.main/llvm15.out	2024-11-21 13:55:58.107295357 -0500
+++ out.mine/llvm15.out	2024-11-21 14:00:30.680188771 -0500
@@ -0,0 +1,10 @@
+diff llvm15-15.0.7-r7.apk llvm15.yaml
+--- llvm15-15.0.7-r7.apk
++++ llvm15.yaml
+@@ -14,4 +14,6 @@
+ depend = so:libm.so.6
+ depend = so:libstdc++.so.6
+ depend = so:libz.so.1
++provides = so:libLTO.so.15=15
++provides = so:libRemarks.so.15=15
+ datahash = 23a2e286edcd4ef174d5890756c775faa548ee8d4407be91d3513f9a78e9c85c
diff -ru '--exclude=*.err' out.main/llvm15.rc out.mine/llvm15.rc
--- out.main/llvm15.rc	2024-11-21 13:56:05.995350151 -0500
+++ out.mine/llvm15.rc	2024-11-21 14:00:30.684188799 -0500
@@ -1 +1 @@
-0
+1
diff -ru '--exclude=*.err' out.main/llvm16.out out.mine/llvm16.out
--- out.main/llvm16.out	2024-11-21 13:55:58.159295718 -0500
+++ out.mine/llvm16.out	2024-11-21 14:00:30.181185305 -0500
@@ -0,0 +1,10 @@
+diff llvm16-16.0.6-r3.apk llvm16.yaml
+--- llvm16-16.0.6-r3.apk
++++ llvm16.yaml
+@@ -14,4 +14,6 @@
+ depend = so:libm.so.6
+ depend = so:libstdc++.so.6
+ depend = so:libz.so.1
++provides = so:libLTO.so.16=16
++provides = so:libRemarks.so.16=16
+ datahash = e64275b4d81210efd3d49e90bd68515060eedc84babd8881a0e976a8cbf4b52a
diff -ru '--exclude=*.err' out.main/llvm16.rc out.mine/llvm16.rc
--- out.main/llvm16.rc	2024-11-21 13:56:05.772348602 -0500
+++ out.mine/llvm16.rc	2024-11-21 14:00:30.187185346 -0500
@@ -1 +1 @@
-0
+1
diff -ru '--exclude=*.err' out.main/llvm17.out out.mine/llvm17.out
--- out.main/llvm17.out	2024-11-21 13:55:58.183295885 -0500
+++ out.mine/llvm17.out	2024-11-21 14:00:30.898190285 -0500
@@ -0,0 +1,10 @@
+diff llvm17-17.0.6-r2.apk llvm17.yaml
+--- llvm17-17.0.6-r2.apk
++++ llvm17.yaml
+@@ -14,4 +14,6 @@
+ depend = so:libm.so.6
+ depend = so:libstdc++.so.6
+ depend = so:libz.so.1
++provides = so:libLTO.so.17=17
++provides = so:libRemarks.so.17=17
+ datahash = 8e204146f4942b643f784bc537ef09a660440025a681cd955dade63296094da8
diff -ru '--exclude=*.err' out.main/llvm17.rc out.mine/llvm17.rc
--- out.main/llvm17.rc	2024-11-21 13:56:06.480353520 -0500
+++ out.mine/llvm17.rc	2024-11-21 14:00:30.902190313 -0500
@@ -1 +1 @@
-0
+1
diff -ru '--exclude=*.err' out.main/lua5.3.out out.mine/lua5.3.out
--- out.main/lua5.3.out	2024-11-21 13:56:02.648326901 -0500
+++ out.mine/lua5.3.out	2024-11-21 14:00:27.565167133 -0500
@@ -7,3 +7,12 @@
  license = MIT
 +depend = man-db
  datahash = a6012234cbc881270340b93e19549a651b7af066b197c69783a9209dd6f96f54
+diff lua5.3-5.3.6-r5.apk lua5.3.yaml
+--- lua5.3-5.3.6-r5.apk
++++ lua5.3.yaml
+@@ -14,4 +14,5 @@
+ depend = so:libm.so.6
+ provides = cmd:lua5.3=5.3.6-r5
+ provides = cmd:luac5.3=5.3.6-r5
++provides = so:liblua-5.3.so.0=0
+ datahash = 619f77b14ec3d7b42df5e1b6c3992b513ee6fa94002bcf4990e4904e29779ad4
diff -ru '--exclude=*.err' out.main/lua5.4.out out.mine/lua5.4.out
--- out.main/lua5.4.out	2024-11-21 13:56:02.669327047 -0500
+++ out.mine/lua5.4.out	2024-11-21 14:00:27.453166355 -0500
@@ -7,3 +7,12 @@
  license = MIT
 +depend = man-db
  datahash = 015a53ac8eeef09a15e34c9445cc641b9f96c2cae3b5a327da46aa9638af62cd
+diff lua5.4-5.4.7-r1.apk lua5.4.yaml
+--- lua5.4-5.4.7-r1.apk
++++ lua5.4.yaml
+@@ -14,4 +14,5 @@
+ depend = so:libreadline.so.8
+ provides = cmd:lua5.4=5.4.7-r1
+ provides = cmd:luac5.4=5.4.7-r1
++provides = so:liblua-5.4.so.0=0
+ datahash = 44212eda3b6802569d8ee669f3bf16b9bdfca45256dcec54b7733d7f8113c8a3
diff -ru '--exclude=*.err' out.main/melange-version.txt out.mine/melange-version.txt
--- out.main/melange-version.txt	2024-11-21 13:54:46.686799236 -0500
+++ out.mine/melange-version.txt	2024-11-21 13:59:09.468624638 -0500
@@ -5,10 +5,10 @@
  |_|  |_| |_____| |_____| /_/   \_\ |_| \_|  \____| |_____|
 melange
 
-GitVersion:    v0.15.12-2-gc0f7f40
-GitCommit:     c0f7f40012d053dde760758b15d80c3220cfcdd0
+GitVersion:    v0.15.12-3-g4c43b40
+GitCommit:     4c43b40b6bd327921e0d9fa3e0be1d3b5b4b3f56
 GitTreeState:  clean
-BuildDate:     '2024-11-21T17:24:19Z'
+BuildDate:     '2024-11-21T17:27:54Z'
 GoVersion:     go1.23.3
 Compiler:      gc
 Platform:      linux/amd64

smoser · 2024-11-21T20:05:47Z

The diff above is almost exactly what i was expecting.
I had expected 7 origins to have changed. Instead there were 5. The two that didn't change as I expected were llvm-18-18.1.8-r3
llvm-19-19.1.4-r0. Looking further, those two did have symlinks but the source was not in /usr/lib/ so they were not considered.

So... this is exactly the changes I expect after that further research.

A build of lua5.4 right now will not 'provide: so:liblua5.4.so' due to /usr/lib/liblua* being symlinks. We need a build chainguard-dev/melange#1662 to fix that. Other things here are: * build 5.4 with V=5.4 (not V=5.3) * build 5.4 with compath for 5.3 not the non-existant 5.2 https://gitlab.alpinelinux.org/alpine/aports/-/issues/13257 * split libs out from the main package (main package will get a dep on the libs so that will be fine) * rename patches in lua5.4/ to not be redundantly lua-5.4-thing * define and use a var 'luaMM' for the major.minor everywhere. * use one patch section with 3 patches rather than 3 patch sections * do not install liblua.la * improve tests a bit to check for version in '-v' output * add a test to make sure lua5.4-libs provides the desired so: * fix typo in configure.ac (top_builddir, not 'top_buildir').

- lua5.4 - updates to package and a rebuild. A build of lua5.4 right now will not 'provide: so:liblua5.4.so' due to /usr/lib/liblua* being symlinks. We need a build chainguard-dev/melange#1662 to fix that. Other things here are: * build 5.4 with V=5.4 (not V=5.3) * build 5.4 with compath for 5.3 not the non-existant 5.2 https://gitlab.alpinelinux.org/alpine/aports/-/issues/13257 * split libs out from the main package (main package will get a dep on the libs so that will be fine) * rename patches in lua5.4/ to not be redundantly lua-5.4-thing * define and use a var 'luaMM' for the major.minor everywhere. * use one patch section with 3 patches rather than 3 patch sections * do not install liblua.la * improve tests a bit to check for version in '-v' output * add a test to make sure lua5.4-libs provides the desired so: * fix typo in configure.ac (top_builddir, not 'top_buildir'). - lua5.3 - apply the same changes, but leave it building with linenoise instead of readline.

smoser mentioned this pull request Nov 21, 2024

smoser test rebuild #1663

Closed

imjasonh reviewed Nov 21, 2024

View reviewed changes

xnox requested changes Nov 21, 2024

View reviewed changes

smoser force-pushed the fix/let-links-provide-so branch from 62b411a to 4c43b40 Compare November 21, 2024 17:28

smoser requested a review from xnox November 21, 2024 19:34

smoser mentioned this pull request Nov 21, 2024

Withdraw lua-rrd-1.8.0-r0 due to it providing liblua-5.4.so wolfi-dev/os#34786

Closed

smoser enabled auto-merge November 21, 2024 21:20

xnox approved these changes Nov 21, 2024

View reviewed changes

smoser merged commit 907019c into chainguard-dev:main Nov 21, 2024
36 checks passed

smoser mentioned this pull request Nov 22, 2024

lua5.3, lua5.4 - package cleanups and rebuild wolfi-dev/os#34985

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consider symlinks as potential provides for so: #1662

Consider symlinks as potential provides for so: #1662

smoser commented Nov 21, 2024 •

edited

Loading

smoser commented Nov 21, 2024

imjasonh left a comment

xnox left a comment

imjasonh commented Nov 21, 2024

smoser commented Nov 21, 2024

smoser commented Nov 21, 2024 •

edited

Loading

Consider symlinks as potential provides for so: #1662

Consider symlinks as potential provides for so: #1662

Conversation

smoser commented Nov 21, 2024 • edited Loading

smoser commented Nov 21, 2024

imjasonh left a comment

Choose a reason for hiding this comment

xnox left a comment

Choose a reason for hiding this comment

imjasonh commented Nov 21, 2024

smoser commented Nov 21, 2024

smoser commented Nov 21, 2024 • edited Loading

smoser commented Nov 21, 2024 •

edited

Loading

smoser commented Nov 21, 2024 •

edited

Loading