feat(wasm): allow using a custom loader

[tiennou]

Rollup Plugin Name: rollup-wasm

This PR contains:

• bugfix
• feature
• refactor
• documentation
• other

Are tests included?

• yes
• no

Breaking Changes?

• yes
• no

Description

This deprecates the targetEnv option and replaces it with loader which performs the same thing while also accepting an actual function.

When that happens, that function's body will be inserted in lieu of the default one, giving complete control over the way WASM is loaded into the runtime.

I need it because I'm trying to load code in a sandboxed Node 10 environment, which doesn't have process nor Buffer available, and that felt like the cleanest way to do that without giving you a bunch of hardcoded code that's meaningless outside of it.

[tiennou]

My initial attempt at this was to give a function name from the bundle instead of something that would only live in the config. But I couldn't find a way to get the mangling done by Rollup to apply to it from inside the plugin; that's why I settled on having a real function.

In other words, if I had set loader: 'myCustomLoader', that would translate to something like this in the final bundle:

function wasm (imports) { return myCustomLoader(0, null, 'AGFzbQEAAAABBwFgAn9/AX8DAgEABQMBAAAHEAIDYWRkAAAGbWVtb3J5AgAKCQEHACAAIAFqCwAkEHNvdXJjZU1hcHBpbmdVUkwSLi9yZWxlYXNlLndhc20ubWFw', imports) }

/* somewhere else */

function myCustomLoader$1(sync, path, src, imports) { /* … */ }

and that $1 prefix would have been a problem.

[shellscape]

Thanks for the PR. I would love to move this forward, but I just don't know enough about WASM to approve or not. Please tag some of the past committers to this plugin to see if they can offer some feedback.

[tiennou]

Okay, rebased because someone else expressed interest (this is for the Screeps MMO game BTW), so I'm gonna tag people I find in the history that touched the wasm package in a meaningful way not too long ago.

@sky0014 @nickbabcock @featherbear @bminixhofer Would one of you have feedback on the changes I made?

[nickbabcock]

Yeah, give me a couple days to try this out. Thanks for the ping.

[nickbabcock]

I agree that introducing loader and deprecating targetEnv is the right choice, as custom functions may contain code unrelated to the environment and would have been an awkward fit for this functionality inside a field called targetEnv. targetEnv, in retrospect, appears short-sighted.

QA tests:

• PR remains backwards compatible with projects that use targetEnv
• Projects can switch to the loader from targetEnv with a find and replace.
• I was able to write a loader function and confirm it worked

I'm approving this PR, but have some reservations. I'm happy to review again if changes are made.

[nickbabcock]

"builtin" might be a better word than default, as I can see it being useful to call out auto is the default but other options exist.

Btw, the readme probably should be updated in this PR too, right?

[tiennou]

builtin indeed sounds better. I'll switch to that. Will go over the readme once the changes are okay-ed

[nickbabcock]

This return type should be something like Promise<WebAssembly.Module | WebAssembly.Instance> right?

[tiennou]

It's been a while, so I forgot a little about how it works, but as I could see, the loader function's return value isn't used anywhere? The pre-change loading code was this:

`import { _loadWasmModule } from ${JSON.stringify(HELPERS_ID)};
export default function(imports){return _loadWasmModule(${+isSync}, ${publicFilepath}, ${src}, imports)}`

but as far as I can see, that return could just as well not be here, since I don't see how you could make use of it and there's no higher-level code expecting anything out of it…
So at that point, if you want the loader to do clever things with the compiled module/instance, just provide a custom loader that does so since there's not much we could do over that that wouldn't lock in details about what's expected of the loader.

[nickbabcock]

I don't see how you could make use of it and there's no higher-level code expecting anything out of it

The code importing the wasm module expects the return value.

If the return is removed from the following example, downstream libraries will fail

{
  loader: async (sync, filepath, src, imports) => {
    const fs = require("fs/promises");
    const path = require("path");
    const wasmBuffer = await fs.readFile(path.resolve(__dirname, filepath));
    return WebAssembly.compile(wasmBuffer);
  },
}

While it is possible to write a wasm library with a loader that assigns to a global variable instead of use the return value, I don't think that would be recommended.

[nickbabcock]

Is this a type signature that we (as in rollup) are willing to commit to exposing? Previously it was internal and I'm assuming not much thought was given. I'm flagging this in case there is a concern about future compatibility issues that arise from the need to update, reorder, or remove parameters that could impact all parameters -- not just the ones that are changed.

[tiennou]

Possibly? I don't feel confident on that part. It might be possible to hide this type externally, but IMO I'd prefer to be sure we agree on the way forward w.r.t the loader as a function/name issue first.

[nickbabcock]

Is there a way to support arrow functions? The following config will fail

{
  loader: async (sync, filepath, src, imports) => {
    const fs = require("fs/promises");
    const path = require("path");
    const wasmBuffer = await fs.readFile(path.resolve(__dirname, filepath));
    return WebAssembly.compile(wasmBuffer);
  },
}

with the error:

[!] RollupError: wasmHelpers.js (1:33): Expected ';', '}' or <eof>
wasmHelpers.js (1:33)

[nickbabcock]

I'm worried about the ergonomics of the loader function. Are there any other rollup plugins with similar functionality?

Functions must be self contained, which means no top-level imports

const fs = require("fs");
const path = require("path");

// ...

{
  loader: function myloader(sync, filepath, src, imports) {
    // BAD
    fs.readFileSync(filepath);
    // ...

    // GOOD
    // const fs = require("fs");
    // fs.readFileSync(/* ... */)
  }
}

All code has to be inlined (can't reference user defined function)

function add(a, b) { return a + b; }
{
  loader: function myloader(sync, filepath, src, imports) {
    add(1, 2); // ERROR
    // ...
  }
}

Perhaps including examples of these edge cases is enough to mitigate the problems, but I see it as a potential sore spot.

[tiennou]

I agree with that. The way I wanted it to work initially was to have that loader function just be the name of a function from the compiled bundle, not a free-standing function as it is. But as explained in the PR's cover, and given I'm a complete noob at rollup's internals, I found no way to properly link that function's name to its actual implementation once rolled up, meaning 1) the name wouldn't get namespaced properly and 2) the actual function, having no discernible callers, would be tree-shaken out.

Given the choice, I'd go back to doing that — have the custom loader option output a chunk with a call to the in-bundle function of your choice, since that'd mean you could use whichever language (JS or TS) you wanted to implement it.

[shellscape]

Since I am not a wasm developer and @nickbabcock is the only active reviewer in our repository so far, I'm going to go off of his signal on whether or not to merge this.

[nickbabcock]

Yeah I'd wait to merge this until either the caveats are addressed or documented as there's a few rough edges for users to be aware of that aren't intuitive.

[shellscape]

Closing as abandoned.