Native extension roll-out procedure

[marc0der]

The rewrite of SDKMAN has been a long time coming, and I've put a lot of thought into how to roll it out. Above all, I want the transition to be smooth and effortless for all our users. I don't even want them to realise that they are using Rust, but just that they slowly see improvements in their day-to-day experience in using our tool. With that approach in mind, here are some thoughts on making this a reality.

I've never believed in big-bang releases, as they invariably seem to go wrong. I've seen many projects sink because of this approach. Instead, I hope to see us transition to native gradually, with Bash functions and Rust binaries living side-by-side as we gradually strangle out the Bash functions. The proposed approach is pretty simple, and a lot of the work has already been happening over the past months.

Bash wrapper

The idea is always to continue to have a light bash wrapper (this is required for exporting environment variables such as PATH in the user's shell. I foresee this wrapper getting lighter over time until it barely does anything. The wrapper will be comprised of a scaled-down version of our bootstrap script sdkman-init.sh, and a loaded main function that will pick either a bash helper or a native binary for each command. This allows us to cut over functionality on a per-command basis.

Walking skeleton

I love the approach of getting a thin slice of functionality working upfront. This slice (or walking skeleton) is what I've been working on up to this point. My idea was to pick some non-critical functionality, in this case, the help subcommand, and get that working end-to-end in the stable channel. In retrospect, I could have done something even more straightforward by cutting over the version subcommand, but hey ho! 😆

I also rolled out some changes to our main bash function that will select our native binary above a bash function if it is present.

Overlays

A prerequisite was that SDKMAN itself would become an SDK which would simplify the install.sh and update.sh scripts in our sdkman-hooks service. It allows us to unzip the SDK archive and move its contents to SDKMAN_DIR. Further, I wanted our new native archive to follow the same approach. It should simply be a second SDK archive we pull down on install/selfupdate that overlays on top of our existing SDKMAN_DIR structure. The only difference in this new archive is that it is platform-specific.

libexec folder

In UNIX parlance, the libexec directory is used for internal binaries that are not meant to be used directly by users. All our native subcommand binaries will be placed here to be invoked by our main bash function. As we continue to rewrite our subcommands, they will gradually fill up this directory. Meanwhile, we will be deleting the old bash helpers from the src folder once we feel that the native counterpart is stable.

That's the basic idea! All that still needs doing is work on the install.sh and update.sh hooks to pull down the appropriate binaries and explode them into the users' SDMAN_DIR. I hope to have this working pretty soon.

[helpermethod]

Sounds great!

[aalmiray]

I guess we'll have to expand compiling binaries to other platforms, don't we? Such as amd64/arm64 for the 3 major, perhaps libc/musl for Linux. Thus 8 targets perhaps?

[marc0der]

amd64/arm64 is already supported for Linux and Mac, but yes, we can support all the others too.

[aalmiray]

Yup, noticed the 5 targets. Missing win-arm64 to cover 3 x 2. I suggested musl to go along with Alpine and Docker images based on alpine.

[marc0der]

Just a note here that we don't identify libc/musl and windows arm64 in the sdkman-init.sh yet. We'll need to infer what their platform identifiers are for these binaries to be of use.

[hgeraldino]

It looks like you have everything figured out :)

Should we do a bit of planning and add tasks to the project board? There are a few issues on sdkman-cli that I'd like to address as well (plus, there's still the checksum backfill discussion pending)

[marc0der]

Totally agree, let's start engaging with that board 😅

[marc0der]

It is more than a year later, so I'd like to give an update on the progress. The rollout has gone pretty smoothly up to this point, and many of the simpler commands have been ported. Most people use them without knowing that the rollout has happened, so that is a victory in itself. You can test this for yourself by typing sdk version. If you see the following in your shell, you have invoked one of our new Rust commands!

SDKMAN!
script: 5.18.2
native: 0.4.6

However, last year, I stopped developing the Rust components temporarily because I had reached a crossroads. We could keep using the old APIs or do the right thing and spin up a new simplified JSON REST API to serve the new helpers. I've gone for the latter. This does bring about some other problems, like keeping datastores in sync and ensuring that the vendor APIs stay operational. Most of these hurdles have now been crossed, and soon, we should be able to resume work on the Rust commands.

The new Rust components will rely on a local database, making SDKMAN much less chatty. It will work like apt or git, where you will pull down the state locally before doing some work. In other words, you would need to do something like:

# pull down all the latest info about java
$ sdk update java

# install the version of java you want
$ sdk install java 21.0.2 amzn

Once the backend is established, we should hopefully start seeing some more movement in the rollout of these new native components.