DyNet could not be loaded! on Apple Silicon

[moritzdueck]

Hi everyone!

When trying to follow the walkthrough, I got the following error:

[error] java.lang.RuntimeException: DyNet could not be loaded!
[error]         at edu.cmu.dynet.internal.dynet_swigJNI.<clinit>(dynet_swigJNI.java:79)
[error]         at edu.cmu.dynet.internal.DynetParams.<init>(DynetParams.java:87)
[error]         at edu.cmu.dynet.Initialize$.initialize(Initialize.scala:14)
[error]         at org.clulab.fatdynet.utils.Initializer$.$anonfun$initialize$1(Initializer.scala:41)
[error]         at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
[error]         at org.clulab.fatdynet.utils.DebugSynchronizer.during(Synchronizer.scala:52)
[error]         at org.clulab.fatdynet.utils.DebugSynchronizer.doSynchronized(Synchronizer.scala:96)
[error]         at org.clulab.fatdynet.utils.DebugSynchronizer.withoutComputationGraph(Synchronizer.scala:122)
[error]         at org.clulab.fatdynet.utils.Synchronizer$.withoutComputationGraph(Synchronizer.scala:181)
[error]         at org.clulab.fatdynet.utils.Initializer$.initialize(Initializer.scala:37)
[error]         at org.clulab.dynet.Utils$.initializeDyNet(Utils.scala:58)
[error]         at ai.lum.odinson.extra.utils.ProcessorsUtils$.getProcessor(ProcessorsUtils.scala:38)
[error]         at ai.lum.odinson.extra.AnnotateText$.delayedEndpoint$ai$lum$odinson$extra$AnnotateText$1(AnnotateText.scala:49)
[error]         at ai.lum.odinson.extra.AnnotateText$delayedInit$body.apply(AnnotateText.scala:17)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]         at scala.collection.immutable.List.foreach(List.scala:392)
[error]         at scala.App.main(App.scala:80)
[error]         at scala.App.main$(App.scala:78)
[error]         at ai.lum.odinson.extra.AnnotateText$.main(AnnotateText.scala:17)
[error]         at ai.lum.odinson.extra.AnnotateText.main(AnnotateText.scala)
[error] Exception in thread "main" java.lang.UnsatisfiedLinkError: 'long edu.cmu.dynet.internal.dynet_swigJNI.new_DynetParams()'
[error]         at edu.cmu.dynet.internal.dynet_swigJNI.new_DynetParams(Native Method)
[error]         at edu.cmu.dynet.internal.DynetParams.<init>(DynetParams.java:87)
[error]         at edu.cmu.dynet.Initialize$.initialize(Initialize.scala:14)
[error]         at org.clulab.fatdynet.utils.Initializer$.$anonfun$initialize$1(Initializer.scala:41)
[error]         at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
[error]         at org.clulab.fatdynet.utils.DebugSynchronizer.during(Synchronizer.scala:52)
[error]         at org.clulab.fatdynet.utils.DebugSynchronizer.doSynchronized(Synchronizer.scala:96)
[error]         at org.clulab.fatdynet.utils.DebugSynchronizer.withoutComputationGraph(Synchronizer.scala:122)
[error]         at org.clulab.fatdynet.utils.Synchronizer$.withoutComputationGraph(Synchronizer.scala:181)
[error]         at org.clulab.fatdynet.utils.Initializer$.initialize(Initializer.scala:37)
[error]         at org.clulab.dynet.Utils$.initializeDyNet(Utils.scala:58)
[error]         at ai.lum.odinson.extra.utils.ProcessorsUtils$.getProcessor(ProcessorsUtils.scala:38)
[error]         at ai.lum.odinson.extra.AnnotateText$.delayedEndpoint$ai$lum$odinson$extra$AnnotateText$1(AnnotateText.scala:49)
[error]         at ai.lum.odinson.extra.AnnotateText$delayedInit$body.apply(AnnotateText.scala:17)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]         at scala.collection.immutable.List.foreach(List.scala:392)
[error]         at scala.App.main(App.scala:80)
[error]         at scala.App.main$(App.scala:78)
[error]         at ai.lum.odinson.extra.AnnotateText$.main(AnnotateText.scala:17)
[error]         at ai.lum.odinson.extra.AnnotateText.main(AnnotateText.scala)
[error] Nonzero exit code returned from runner: 1

I tried both, executing the code locally with the cloned repository and using the docker image as mentioned.

The way I was able to make it work was to specify the platform for Docker as:

docker run \
  --name="odinson-extras" \
  -it \
  --rm \
  --platform linux/amd64 \
  -e "HOME=/app" \
  -e "JAVA_OPTS=-Dodinson.extra.processorType=CluProcessor" \
  -v "path/to/data/odinson/data:/app/data/odinson" \
  --entrypoint "bin/annotate-text" \
  "lumai/odinson-extras:latest"

Do you know of any way I can run the code on a newer Mac?

Thanks for the help!

Cheers,
Moritz

[kwalcock]

The current version of odinson appears to depend on processors 8.4.6 which in turn depends on fatdynet 0.3.2. Hardware support for Apple silicon was not added to fatdynet until 0.4.4 which is used in processors 8.4.9 and above. So, one could try to update the procVersion in extra/build.sbt to 8.4.9 or higher (up to 8.5.4).

However, older versions should run OK on a Mac with Apple silicon using Rosetta, which might have been automatically installed at some time. I'm not sure of the current situation there and one would want to check the local computer. It may be important to use an Intel version of the JVM in order to be able to access fatdynet compiled for Intel. So, there's a chance that a JVM change along with Rosetta can fix the problem.

[moritzdueck]

Thank you so much for the quick and super helpful answer 🙏 updating processors to 8.5.4 did the trick!