Web support (theoretical) #10
Comments
Some collected thoughts1.WASM/WASI with a web WASI polyfill seems like the best option here so far because it is closer to the web platform and will probably have more maintenance over time. And maybe, if other platforms support WASM/WASI properly in the future, I could just ship one
2.Making a new Rust target from the learning technologies libc
3.
ConclusionI am really hoping option 1 turns out, and it should in theory. It will, however, be a bust if one of the depended-on crates relies on something in |
|
Few immediate blockers with option 1:
|
|
Also looks like WASI might get threading soon-ish™️, and Dart will also get Native Assets some months down the line. Might just be worth waiting on these for web support, instead of trying to work around them. |
|
Interesting documentation. It was very helpful.
Are you referring to this article Beyond Dart 3? Will this make porting native code into browsers easier? Just curious, have you considered going with Isar's approach https://github.com/isar/isar/tree/main/packages? I.e, using web browsers indexedDB for the time being? |
No, I’m referring to dart-lang/sdk#50565 If Flutter provides an easy way for plugins to use .wasm files with Native Assets, then I plan to go with that approach. The flutter approach would also need to support a browser WASI polyfill; otherwise, I would need to handle it all manually.
mimir wraps around milli, which is the search engine library that powers meilisearch. milli uses LMDB as a key value store under the hood, and I could try to make a version of milli that uses a different key value store, but I’d rather not as it’d be pretty difficult to do correctly and would be time consuming. Most FRB libraries can probably get away with simply compiling to WASI and using a WASI polyfill for web support (which would use something like IndexedDB under the hood in the polyfill), but I can’t do that directly in mimir (at the moment) because of its unique position (due to its dependency on LMDB). |
Slight UpdateLMDB
Edit: I didn't actually get LMDB to compile (thought I did based on Terminal output, but it was still in the process of compiling), but I got close. Need to tweak a few LMDB compilation flags. Edit 2: LMDB is failing to compile due to missing POSIX file lock & signal support in WASI libc. Since file locking was recently added to WASI (WebAssembly/wasi-filesystem#2), I'm guessing it'll be added to WASI libc sometime in the near future. However, the issue of some missing POSIX signal functions is still present. LMDB Compilation Errorspage_sizeNow the only blocking issue, as far as I can see, is page_size failing to compile to WASI. See Elzair/page_size_rs#3
|
|
Seeing WebAssembly/WASI#166, I'm inclined to think that the best way forward would actually be trying an alternative backend in heed that is WASM/WASI friendly. Trying to see if mdbx works. Update: compiling mdbx fails because compilation cannot find assert.h in wasi-libc. Not sure what the fix for this is, considering I can see an assert.h in |
|
For picking up where I left off: investigate mdbx-sys’ build.rs. I have a feeling it might be doing something wrong, causing assert.h to not be found. |
|
mdbx is out of the picture because the team over at meili would rather go for a full Rust-only approach (rather than a different C library); specifically, either sanakirja or redb. Sanakirja is a bit more complex and has some limitations (like limited key/value size), and fails spectacularly when trying to compile to WASI due to some unmaintained dependencies. redb looks more promising (despite being in beta still), so I plan to spend some time to see how WASI compatible it is; see cberner/redb#507 |
|
Based on the 3 options I originally gave above:
Now it is just a matter of first making sure that milli's deps can actually run on WASI before going through the trouble of writing a shim based on LMDB's APIs that actually uses IndexedDB. |
|
See meilisearch/heed#162. For future self, can probably use https://github.com/devashishdxt/rexie as a reference implementation on top of web-sys. |
|
Instead of indexedDB, will try out redb with WASI as that’d be preferred. If wasmer js is problematic, can also try out https://github.com/bjorn3/browser_wasi_shim |
|
This may be very helpful for future implementation: https://wasmer.io/posts/announcing-wasix |
|
Hey @GregoryConrad Great work on mimir, looks very promising, esp. the relevance part. We've been on Isar since its start, so maybe a few insights might help here:
Hope this helps. |
|
Hi @mdmm13 👋 About your comments:
|
|
+1 for web |
Web support for mimir is currently not possible due to the lack of a publicly-available libc implementation for WASM made with JS & browser APIs. This concept of a browser-based libc sounds weird, because it is, but it isn't actually too bad; keep reading to see why.A libc implementation via web APIs is theoretically possible (and has been done in some capacity!), as seen with https://webvm.io. WebVM was created with CheerpX and is written about in more detail here.Considering an entire VM runs natively in the browser already using an in-progress libc implementation, and that VM itself isn't terribly slow, I see promise for the possibility of web support for this library (and many others) in the not-so distant future. (Again, this all relies on an open-source libc implementation that can be used by Rust.)libcs for the web are already available to differing degrees of completeness. Now it is just a matter of being able to use them.The text was updated successfully, but these errors were encountered: