td-agent 4.4.2 for macOS x86_64 lacks libyaml

[ashie]

It's originally reported by @dferreiralopessc at Slack.

Although fluentd 1.15 requires libyaml mandatory but td-agent for macOS x86_64 lacks it:

sudo /opt/td-agent/usr/sbin/td-agent
Traceback (most recent call last):
	24: from /opt/td-agent/usr/sbin/td-agent:15:in `<main>'
	23: from /opt/td-agent/usr/sbin/td-agent:15:in `load'
	22: from /opt/td-agent/bin/fluentd:23:in `<top (required)>'
	21: from /opt/td-agent/bin/fluentd:23:in `load'
	20: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
	19: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	18: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	17: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:19:in `<top (required)>'
	16: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	15: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	14: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:21:in `<top (required)>'
	13: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	12: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	11: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/config.rb:20:in `<top (required)>'
	10: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	 9: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	 8: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/config/yaml_parser.rb:17:in `<top (required)>'
	 7: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	 6: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	 5: from /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/config/yaml_parser/loader.rb:17:in `<top (required)>'
	 4: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	 3: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
	 2: from /opt/td-agent/lib/ruby/2.7.0/psych.rb:13:in `<top (required)>'
	 1: from /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
/opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': dlopen(/opt/td-agent/lib/ruby/2.7.0/x86_64-darwin20/psych.bundle, 0x0009): Library not loaded: /usr/local/opt/libyaml/lib/libyaml-0.2.dylib (LoadError)
  Referenced from: <35E2A478-8DE6-37F5-B116-40F99453B946> /opt/td-agent/lib/ruby/2.7.0/x86_64-darwin20/psych.bundle
  Reason: tried: '/usr/local/opt/libyaml/lib/libyaml-0.2.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/libyaml/lib/libyaml-0.2.dylib' (no such file), '/usr/local/opt/libyaml/lib/libyaml-0.2.dylib' (no such file), '/usr/local/lib/libyaml-0.2.dylib' (no such file), '/usr/lib/libyaml-0.2.dylib' (no such file, not in dyld cache) - /opt/td-agent/lib/ruby/2.7.0/x86_64-darwin20/psych.bundle

On my environment, it uses Homebrew's one:

iMac5k-211:td-agent aho$ otool -L /opt/td-agent/lib/ruby/2.7.0/x86_64-darwin20/psych.bundle 
/opt/td-agent/lib/ruby/2.7.0/x86_64-darwin20/psych.bundle:
	/opt/td-agent/lib/libruby.2.7.dylib (compatibility version 2.7.0, current version 2.7.6)
	/usr/local/opt/libyaml/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

td-agent for macOS arm64 doesn't have this issue since it uses built-in libyaml:

aho@M1-Mac-mini td-agent % otool -L /opt/td-agent/lib/ruby/2.7.0/arm64-darwin21/psych.bundle
/opt/td-agent/lib/ruby/2.7.0/arm64-darwin21/psych.bundle:
	/opt/td-agent/lib/libruby.2.7.dylib (compatibility version 2.7.0, current version 2.7.6)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

We should rebuild ruby with built-in libyaml on x86_64 too.

[ashie]

In addition, Fluentd should require psych lazily only when it's really needed.

[ashie]

Bundled libyaml has been removed since Ruby 3.2.
From the next version of td-agent we'll use Ruby 3.2 so we should add libyaml by ourself.

[ashie]

libffi is also required for fiddle on Ruby 3.2.

[dferreiralopessc]

hi @ashie, do we have an ETA for the new td-agent release? many thanks!

[ashie]

I've fixed it in v4.5.0 for v4 series.
For v5 (work in progress), it's not yet fixed.

[dferreiralopessc]

thanks @ashie!