Library not loaded: libpact_ffi.dylib

[stan-is-hate]

Apologies in advance for a long post.
This issue started happening to me seemingly from the latest MacOS update, but might've been caused by some other changes on the system.
I'm wondering if this is local to my system or if others have experienced this too?

Basically just trying to run go test ... with pact-go causes issues loading libpact_ffi.dylib:

dyld[13198]: Library not loaded: libpact_ffi.dylib
  Referenced from: <097B1871-BF1B-3A20-B6CE-B3EE8D6C100B> /private/var/folders/vk/9xmgp9w16dj3ty41013x0zt80000gn/T/go-build92553240/b001/consumer.test
  Reason: tried: 'libpact_ffi.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpact_ffi.dylib' (no such file), 'libpact_ffi.dylib' (no such file), '/Users/stanislav.vodetskyi/github/pact-go/consumer/libpact_ffi.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/stanislav.vodetskyi/github/pact-go/consumer/libpact_ffi.dylib' (no such file), '/Users/stanislav.vodetskyi/github/pact-go/consumer/libpact_ffi.dylib' (no such file)
FAIL    github.com/pact-foundation/pact-go/v2/consumer  0.474s

Note how the log doesn't mention looking at /usr/local/lib or /usr/lib.
I'm not 100% sure, but I suspect that previously DYLD_FALLBACK_LIBRARY_PATH was set to a default value of /usr/local/lib:/usr/lib, or dyld had these defaults hardcoded if the variable was unset, but not anymore.
So it used to be able to find libpact_ffi.dylib in fallback path, but not anymore.

Some more details.

pact-go installs libpact_ffi.dylib under /usr/local/lib by default.
pact-go also invokes install_name_tool -id libpact_ffi.dylib /usr/local/lib/libpact_ffi.dylib.
This sets the id / install name of the lib to libpact_ffi.dylib. So whenever I link against this library (e.g. via go test -c ...) it looks like this (interestingly linking works ok, not sure which vars control the lookup paths there):

otool -L pact.test
pact.test:
        libpact_ffi.dylib (compatibility version 0.0.0, current version 0.0.0)

(not sure why the version is 0.0.0, looks weird and suspicious, but I've tried two latest libpact_ffi releases and both look like that)
so when dyld actually links to it at runtime (and assuming dyld_library_path is not set), it will search by this name first (since it's not absolute path, it'll check current dir) or check fallback path.

Search by name fails, because our current work dir is not /usr/local/lib and there's no such file. And fallback path is empty by default in my case, I suspect because of the latest MacOS update, so it doesn't even try to search there. This corresponds to the error message I'm seeing, where it only searches current path (and some weird paths that ultimately look like current path + some security nonsense)

I was able to resolve it locally by:

• either ensure DYLD_FALLBACK_LIBRARY_PATH is set to /usr/local/lib - tricky to do for go test, since it requires using -exec flag, as described here - https://forum.golangbridge.org/t/go-test-with-cgo-on-macos-and-dyld-library-path/32274/2
• or calling install_name_tool -id /usr/local/lib/libpact_ffi.dylib /usr/local/lib/libpact_ffi.dylib - now when linking to this lib, the resulting binary would link to a full path, rather than just the file name, removing the need to search in fallback dir

I don't have an older MacOS version to confirm if DYLD_FALLBACK_LIBRARY_PATH was set (or if dyld had a hardcoded value to use when it wasn't set).

I can send a PR to pact-go to change install_name_tool invocation, but I'm not sure why it was setting the name without a full path in the first place, and since I don't fully understand it, I'm worried I might break some use case unknown to me. Could it be that older pact ffi dylibs had a wrong id set and the call to install_name_tool is just to correct it?

One more thing that I'm still confused about is how it was supposed to work when custom --libDir value was passed to pact-go install - how would that custom library be found without modifying DYLD_* variables or providing a full path in install_name_tool? E.g. in the Makefile of pact-go itself. Is /tmp searched by linker and loader by default?

Software versions

• OS: e.g. Mac OSX 13.6 (22G120)
• Golang Version: go1.20.8 (also tried with go1.21)
• Golang environment:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/stanislav.vodetskyi/Library/Caches/go-build"
GOENV="/Users/stanislav.vodetskyi/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/stanislav.vodetskyi/go/pkg/mod"
GONOPROXY="github.com/confluentinc/*"
GONOSUMDB="github.com/confluentinc/*"
GOOS="darwin"
GOPATH="/Users/stanislav.vodetskyi/go"
GOPRIVATE="github.com/confluentinc/*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.20.8"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/vk/9xmgp9w16dj3ty41013x0zt80000gn/T/go-build2646031252=/tmp/go-build -gno-record-gcc-switches -fno-common"

Relevent log files

pact-go install is successful:

2023/09/27 00:17:15 [INFO] package libpact_ffi not found
2023/09/27 00:17:15 [INFO] downloading library from https://github.com/pact-foundation/pact-reference/releases/download/libpact_ffi-v0.4.5/libpact_ffi-osx-x86_64.dylib.gz to /usr/local/lib/libpact_ffi.dylib
&{}
2023/09/27 00:17:17 [DEBUG] obtaining hash for file /usr/local/lib/libpact_ffi.dylib
2023/09/27 00:17:17 [DEBUG] writing config {map[libpact_ffi:{libpact_ffi 0.4.5 5971854e21bf8b41db8b821a77caa268}]}
2023/09/27 00:17:17 [DEBUG] writing yaml config to file libraries:
  libpact_ffi:
    libname: libpact_ffi
    version: 0.4.5
    hash: 5971854e21bf8b41db8b821a77caa268

2023/09/27 00:17:17 [INFO] setting install_name on library libpact_ffi for osx
2023/09/27 00:17:17 [DEBUG] output from command []
2023/09/27 00:17:17 [INFO] package libpact_ffi found
2023/09/27 00:17:17 [INFO] checking version 0.4.5 of libpact_ffi against semver constraint >= 0.4.0, < 1.0.0
2023/09/27 00:17:17 [DEBUG] 0.4.5 satisfies constraints 0.4.5 >= 0.4.0, < 1.0.0
2023/09/27 00:17:17 [INFO] package libpact_ffi is correctly installed
2023/09/27 00:17:17 [DEBUG] skip checking ffi version() call because FFI not loaded. This is expected when running the 'pact-go' command

[YOU54F]

thx for the thorough report, we’ve had the same on Slack.

is this with ventura 13.6? i wonder if they’ve changed dyld load paths.

it’s late now but will investigate tomorrow. i have vms running amd64 and arm64 so will pull down the ipsw’s and test this out

[YOU54F]

Been able to replicate that now on my work laptop

❯ DYLD_LIBRARY_PATH=/usr/local/lib go test main/consumer_test.go
ok  	command-line-arguments	0.054s

❯ go test main/consumer_test.go
Alias tip: got main/consumer_test.go
dyld[58542]: Library not loaded: libpact_ffi.dylib
  Referenced from: <9E73780A-35B2-3074-8A0D-96B50CDAD065> /private/var/folders/2j/z1vzhzfj3g75d7cwc039m8280000gs/T/go-build759910136/b001/main.test
  Reason: tried: 'libpact_ffi.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpact_ffi.dylib' (no such file), 'libpact_ffi.dylib' (no such file), '/Users/yousaf.nabi/dev/you54f/go-pact-test/go-1/main/libpact_ffi.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/yousaf.nabi/dev/you54f/go-pact-test/go-1/main/libpact_ffi.dylib' (no such file), '/Users/yousaf.nabi/dev/you54f/go-pact-test/go-1/main/libpact_ffi.dylib' (no such file)
signal: abort trap
FAIL	command-line-arguments	0.017s
FAIL

[🔴 ERROR] ❯ go version
Alias tip: gove
go version go1.21.1 darwin/arm64

❯ sw_vers
ProductName:		macOS
ProductVersion:		13.6
BuildVersion:		22G120

❯ otool -L /usr/local/lib/libpact_ffi.dylib
/usr/local/lib/libpact_ffi.dylib:
	libpact_ffi.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1953.255.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.60.24)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

[YOU54F]

One more thing that I'm still confused about is how it was supposed to work when custom --libDir value was passed to pact-go install - how would that custom library be found without modifying DYLD_* variables or providing a full path in install_name_tool? E.g. in the Makefile of pact-go itself. Is /tmp searched by linker and loader by default?

It appears to only be possible either by modifying adding to the DYLD_LIBRARY_PATH or downloading it to the current working dir

🕙12:37:59 ❯ pact-go install --libDir main
2023/09/29 12:38:05 [INFO] set lib dir target to main
2023/09/29 12:38:05 [INFO] package libpact_ffi not found
2023/09/29 12:38:05 [INFO] downloading library from https://github.com/pact-foundation/pact-reference/releases/download/libpact_ffi-v0.4.5/libpact_ffi-osx-aarch64-apple-darwin.dylib.gz to main/libpact_ffi.dylib
&{}
2023/09/29 12:38:07 [INFO] setting install_name on library libpact_ffi for osx
2023/09/29 12:38:07 [INFO] package libpact_ffi found
2023/09/29 12:38:07 [INFO] checking version 0.4.5 of libpact_ffi against semver constraint >= 0.4.0, < 1.0.0
2023/09/29 12:38:07 [INFO] package libpact_ffi is correctly installed

🕙12:38:07 ❯ go test main/consumer_test.go
Alias tip: got main/consumer_test.go
ok  	command-line-arguments	0.077s

Some useful posts around DYLD*

https://hynek.me/articles/macos-dyld-env/
https://www.fullstaq.com/knowledge-hub/blogs/an-alternative-to-macos-dyld-library-path

The latter is used by Traveling-ruby, using add_rpath

It is also used in pact-js-core

https://github.com/pact-foundation/pact-js-core/blob/a00bddaad35d678e54d6afae04209a2ef0021867/binding.gyp#L142

It would probably make sense for it to be invoked when we are using a custom libDir, so that we can lookup.

I note you are running go1.20 on an intel Mac. We've had the same report on a 13.5.1 apple silicon machine.

https://pact-foundation.slack.com/archives/C9UTHTFFB/p1695927031221869

[StevieBoyWonder]

I've also encountered this issue on

Go: 1.21.1 MacOSX : 14 Apple Silicon

[stan-is-hate]

is this with ventura 13.6

Yeah, this is with 13.6 for me.

I think I saw a line in Apple XCode latest release notes that they rewrote the linker, could it be the root cause?
I've checked with folks running older OSs, and for them DYLD_FALLBACK_LIBRARY_PATH is also unset, so it seems there used to be a default value hardcoded in dyld.

Interesting that for you dyld_library_path is propagated correctly to the go test here:

DYLD_LIBRARY_PATH=/usr/local/lib go test main/consumer_test.go

cause for me (albeit with dyld_fallback_library_path) I had to run go test -exec with a custom script to set the value.

Also, here's the doc I found the most helpful around dyld and linking:
https://matthew-brett.github.io/docosx/mac_runtime_link.html

[stan-is-hate]

On @rpath:

Every macOS executable can embed a list of library search paths, or “rpaths”. Whenever macOS encounters a dependency path that references “@rpath”, macOS will search for that dependency in the embedded list of paths.

Worth testing if rpath is resolved correctly with go.
Is there any downside to setting install_name to a full path for libpact_ffi.dylib, e.g. install_name_tool -id /usr/local/lib/libpact_ffi.dylib /usr/local/lib/libpact_ffi.dylib ?

[YOU54F]

Interesting that for you dyld_library_path is propagated correctly to the go test here:

DYLD_LIBRARY_PATH=/usr/local/lib go test main/consumer_test.go

cause for me (albeit with dyld_fallback_library_path) I had to run go test -exec with a custom script to set the value.

So that may be because I have sip disabled on this particular machine

csrutil status
System Integrity Protection status: disabled.

and I was testing out Docker containers on mac, and needed to install macfuse. Interestingly that works fine, and is also located in /usr/local/lib

Thanks for the additional links Stan! If you wanted to try a go at PR with a couple of different attempts to sort this, I'd be up for testing it! I can test with VM's with SIP enabled and without.

Is there any downside to setting install_name to a full path for libpact_ffi.dylib, e.g. install_name_tool -id /usr/local/lib/libpact_ffi.dylib /usr/local/lib/libpact_ffi.dylib

I would say no as we aren't expecting it to be packaged up as portable, and if the libDir option was set at install time, we could set the full path to wherever the user is requesting to install

[stan-is-hate]

I'll try to send some PRs next week, thanks for all the info Yousaf!

So that may be because I have sip disabled on this particular machine

Yes that would explain it.

For macfuse, can you run otool -D <library> maybe? I wonder if maybe they have the correct install path embedded already by some install script? Or it could be that the binary is in the same folder?

[YOU54F]

bindfs bins in /usr/local/bin and libs in /usr/local/lib

it's used by some golang apps, that have been built and in homebrew lib dirs.

it's got the full path there

[stan-is-hate]

so the full path is why it works out of the box, cool

[stan-is-hate]

I'm wondering if we should report this to apple, but I'm not really sure how their bug reporting tool works

[YOU54F]

You could do, seems like the way it to do it through apple developer website and log a ticket, from what I've read in the apple forums.

Guide from Eskimo - who is super helpful on the forums - https://developer.apple.com/forums/thread/712889 (works for Apple)

FYI I've just been testing this locally, I get the same error running the tests (can't find lib), and updating the install_name_tool invocation as you suggested

and everything is working as expected - as per bindfs, we have our full path to wherever the user specifies the lib to be installed

otool -D /tmp/libpact_ffi.dylib
/tmp/libpact_ffi.dylib:
/tmp/libpact_ffi.dylib

I think this might be a sensible way to go. I'll test it out against older versions of MacOS over the weekend

Thanks for investigating dude

[stan-is-hate]

Thanks! I'll try to come up with a concise report to apple next week.

[mefellows]

First of all, thank you for the very detailed bug report and investigation - this is super helpful and I really appreciate the time you took to report it.

One more thing that I'm still confused about is how it was supposed to work when custom --libDir value was passed to pact-go install - how would that custom library be found without modifying DYLD_* variables or providing a full path in install_name_tool? E.g. in the Makefile of pact-go itself. Is /tmp searched by linker and loader by default?

Yes, that's one option, another is to specify the other pre-configured search path (/tmp). This can be seen here (there was also this option, but that caused unnecessary warnings: #187).

Is there any downside to setting install_name to a full path for libpact_ffi.dylib, e.g. install_name_tool -id /usr/local/lib/libpact_ffi.dylib /usr/local/lib/libpact_ffi.dylib ?

Possibly, I don't know. It might mean it can only be installed into /usr/local/lib (because the full path is hard coded). I know at the time, without using install_name_tool it wouldn't work at all.

On install_name and @rpath etc. From your linked article:

One advantage of @rpath is that you can put several different search paths into the the library or executable @rpath. For example, you could set the @rpath so that the library or executable looks for its libraries in a relative path and also an absolute system path.

This is what lib.go is aiming to do. If we can improve this, I'm all ears. I can't quite remember the details now, but I remember it you couldn't do this like "check the ~/.pact/libs` directory" because it couldn't determine that at link / compilation time. It's possibly it is doable though, and I just didn't quite know how to make it all work at the time.

TL;DR - I would definitely appreciate a PR. I think if we encode the full path that corresponds to what is passed to --libDir, that would probably resolve the issue.

[YOU54F]

So it seems I came across this a few months when trying to get this running for MacOS in CI

Run here https://github.com/YOU54F/pact-go/actions/runs/5544271940/job/15016645245

Tried some changes to support testing x platform, managed to recreate the issue on MacOS and resolve with a full path in install name tool, but have a failing test on arm64

master...ci/test_x_plat

[stan-is-hate]

Created #350 - a simple fix really, with using absolute path for install_name. I can't think of downsides, but since I'm very new to building and linking native libs, might be missing some finer details? doesn't look like it tho.

I don't have an arm64 machine handy to test the fix, will ask the colleagues.

[stan-is-hate]

Searching apple forums and it seems it's been reported multiple times already, e.g. here:
https://developer.apple.com/forums/thread/737920

[mefellows]

I don't have an arm64 machine handy to test the fix, will ask the colleagues.

I have an arm64 and it works on my machine. I'm just sorting through some other minor admin and will get a release out today.

[mefellows]

v2.0.2 released with the fix, thanks so much for the great bug report and for submitting a PR 🙏 . Please double check it works for you, in the off chance something went bork in the release.

[hborham]

We confirmed this works for us as well. Thanks @stan-confluent!

[stan-is-hate]

Thanks Matt for quick review and release! I'll let you know if something doesn't work.

[mefellows]

Thanks all, great teamwork 👏