-
-
Notifications
You must be signed in to change notification settings - Fork 359
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use GraalVM Native Image for Mill Client (1500USD Bounty) #4007
Comments
@lihaoyi Any tips on how to resolve the following?
IIUC, the value is read from |
@lihaoyi What do you have in mind for this point? I have a mill-native-image plugin, used by Scala CLI in particular, that can be used for that, and could be added to the contrib folder of Mill if needed |
@ajaychandran I am assuming you're using #4009 because without that PR I can't build a native-image at all. w.r.t. the MILL_CLASSPATH. Currently, the mill client and mill server use the same assembly jar with a bash prelude launcher, which is why we can do that. If we use a native client, we'll need to find another way to package them together. Maybe we can use the native image as the prelude launcher for the assembly and have it unzip or unpack itself somehow before execution so it can reference the jar file? I don't actually know what the answer here is, but we can probably come up with something |
@alexarchambault I was thinking to inline native-image support into In general we cannot use third-party plugins in our example test suite without causing circular dependency and bootstrapping issues, so if we want to begin using native image (which I do) we need it to be in-sourced into the Mill repo |
@ajaychandran seems like if we bundle the graal launcher and the rest of the jars in a zip, like we do now for the bash launcher, we can access the executable path via https://stackoverflow.com/a/77736422/871202 |
Since binary headers are at the beginning of files and zip headers (jar is a zip) are at the end, one hack you can use ( cosmopolitan libc does that) is to concatenate the binary and the jar which makes the file both a native binary and a jar. If you run the binary directly you are running the client, if you run it with //> using platform scalaNative
@main
def run =
println("hello world from scala native!")
generates @main
def run =
println("hello world from java!")
generates Then you concatenate them:
And you get two applications in the same file. Running the Scala Native application
Running the Scala JVM application
|
@lolgab Thank you for the wonderful tip! Initial results look promising: $ time ./mill --version
Mill Build Tool version 0.12.2-50-b1f6be
Java version: 24, vendor: Oracle Corporation, runtime: ~/.sdkman/candidates/java/24.ea.22-graal
Default locale: en_US, platform encoding: UTF-8
OS name: "Linux", version: 6.8.0-49-generic, arch: amd64
real 0m0.116s
user 0m0.204s
sys 0m0.084s
$ time ./native --version
Mill Build Tool version 0.12.2-55-2482ce-DIRTY700a3522
Java version: 24, vendor: Oracle Corporation, runtime: ~/.sdkman/candidates/java/24.ea.22-graal
Default locale: en_US, platform encoding: UTF-8
OS name: "Linux", version: 6.8.0-49-generic, arch: amd64
real 0m0.023s
user 0m0.002s
sys 0m0.016s
$ time java -jar native --version
Mill Build Tool version 0.12.2-55-2482ce-DIRTY700a3522
Java version: 24, vendor: Oracle Corporation, runtime: ~/.sdkman/candidates/java/24.ea.22-graal
Default locale: en_US, platform encoding: UTF-8
OS name: "Linux", version: 6.8.0-49-generic, arch: amd64
real 0m0.100s
user 0m0.201s
sys 0m0.069s |
Should these be added under |
Why? GraalVM seems to be well supported by ordinary setup-java: https://github.com/actions/setup-java/?tab=readme-ov-file#supported-distributions |
The new Configuring JVM versions functionality should be able to pull in the desired Graal versions we want without needing a global install I think. In my cursory experiments I was able to do so https://mill-build.org/mill/fundamentals/configuring-jvm-versions.html |
…4085) Fixes #657 We re-use `MillBackgroundWrapper` and extend it to optionally spawn subprocess instead of calling the in-process main method. There's a bit of overhead in the JVM wrapper, but we need some kind of process wrapper to manage the mutex and termination even when the Mill process terminates, and eventually when #4007 lands we can use that to provide lighter-weight wrappers. We add a shutdown hook and copy the termination grace-period logic from OS-Lib to try and gently kill the wrapped process when necessary Integrated this into `PythonModule#runBackground` and added a unit test to verify the asynchronous launch, background existence (by checking that a file lock is taken) and termination on deletion. We can integrate this into `TypescriptModule` as well but punting that for later The subprocess APIs are a mess but leaving that to fix in 0.13.0 #3772
From the maintainer Li Haoyi: I'm putting a 1500USD bounty on this issue, payable by bank transfer on a merged PR implementing this.
Mill's client is currently a Java/JVM application. This means it suffers from the normal JVM slow startup (>100ms) and warmup times, and it prevents us from moving more logic into the client otherwise it will become even slower.
We should explore using GraalVM to generate cross-platform native image launchers for the Mill client, and integrate that into our
./mill
script. This will allow us to make the client smarter:-i must be passed as first argument
restrictionFor this bounty, the success criteria of this issue is twofold:
An example in the Mill documentation of the recommended way to use Graal native image to compile a native binary from a
JavaModule
,ScalaModule
, orKotlinModule
Have a Mill generate a native image version of the
runner.client
jar on different operating systems and CPU architectures (linux,mac,windows X arm,intel) that is able to replace the current JVM client launcher to launch and manage the Mill backend server process, using the same Java code thatrunner.client
is implemented with today.Reference https://blogs.oracle.com/developers/post/building-cross-platform-native-images-with-graalvm
The text was updated successfully, but these errors were encountered: