You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
this may be able to cut server disk use by around half a gigabyte and could provide for better results (every word would have an embedding, the keywords system could die, potentially less dataset patching needed for words like "DEEEEEEEAF"
fasttext.cc has info on how to build fasttext to a wasm binary, and compress fasttext down to just a few megabytes.
Upsides:
server storage requirements dramatically reduced
every word gets an embedding
slightly improved privacy traits
possibly maybe could remove embeddings from search index data, generate them locally on index parse. index could be as simple as a result index number and a list of keywords and hashtags
By ditching vector-library, server disk data size becomes mainly defined by locally hosted videos in video cache, potentially makes it possible to run find-sign off services like free tier glitch for smaller indexes, or for larger indexes if hotlinking videos or rehosting them elsewhere (some cloud video provider maybe?)
Downsides
initial load to interactive goes from < 1mb to a few megabytes to load the fasttext model
devices which do not support wasm maybe an issue (defer to server execution? improve the old cgi-bin fallback?)
unknown impact on older devices? will ancient android phones cope with running fasttext locally?
even ditching embedding cache in search index, likely to still be larger and more resource intensive to load and run.
The text was updated successfully, but these errors were encountered:
this may be able to cut server disk use by around half a gigabyte and could provide for better results (every word would have an embedding, the keywords system could die, potentially less dataset patching needed for words like "DEEEEEEEAF"
fasttext.cc has info on how to build fasttext to a wasm binary, and compress fasttext down to just a few megabytes.
Upsides:
Downsides
The text was updated successfully, but these errors were encountered: