-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add worker api #36
base: master
Are you sure you want to change the base?
Add worker api #36
Changes from all commits
b0c14d5
a1977c1
f65f15c
a39649e
411d8aa
6dae67e
26e8ec6
1dd13a6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
import * as path from 'path'; | ||
|
||
export default { | ||
mode: 'production', | ||
module: { | ||
rules: [ | ||
{ | ||
test: /(\.ts$|\.js$)/, | ||
exclude: /node_modules/, | ||
use: ['ts-loader'], | ||
}, | ||
{ | ||
test: /(\.worker\.ts$|\.worker\.js$)/, | ||
exclude: /node_modules/, | ||
use: [ | ||
{ | ||
loader: 'worker-loader', | ||
options: { fallback: false, inline: true }, | ||
}, | ||
'ts-loader', | ||
], | ||
}, | ||
], | ||
}, | ||
resolve: { | ||
extensions: ['.ts', '.js'], | ||
}, | ||
entry: { | ||
lib: path.resolve(__dirname, '../worker/worker-lib.ts'), | ||
}, | ||
output: { | ||
filename: 'umap-worker-js.js', | ||
libraryTarget: 'commonjs2', | ||
path: path.resolve(__dirname, '../lib'), | ||
}, | ||
optimization: { minimize: true }, | ||
}; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
{ | ||
"extends": "../tsconfig.json", | ||
"include": ["./*.ts"] | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
import { UMAP } from '../src/lib'; | ||
|
||
async function messageHandler({ data: { id, method, args } }) { | ||
console.log( | ||
`UMAPWorker: received instruction to execute 'umap.${method}' with ${args.length} args` | ||
); | ||
switch (method) { | ||
case 'fit': { | ||
const umap = new UMAP(); | ||
const embedding = await umap.fit(args[0]); | ||
// @ts-ignore | ||
postMessage({ id, method, results: { embedding } }); | ||
break; | ||
} | ||
default: { | ||
// @ts-ignore | ||
postMessage({ | ||
id, | ||
error: `Unknown method requested of UMAPWorker: 'umap.${method}'`, | ||
}); | ||
} | ||
} | ||
} | ||
|
||
onmessage = messageHandler; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
import { v4 as uuid } from 'uuid'; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not at all wedded to pulling in this dep to keep track of promises, just part of the proof-of-concept nature of what's currently in this PR. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Totally, I'd prefer to keep this lib as dependency-free as possible (mainly for google-internal compatibility reasons) - I usually just use a start-at-one-and-increment id generating system for stuff like this. |
||
|
||
// @ts-ignore | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Tried many variations on worker-loader's advice for using it with TypeScript, but it never bundled correctly (didn't fail out during build, but the bundle would error in my example in the browser). Only ignoring all advice and using the config I've used for other projects with this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is this issue open on the worker-loader, though that problem isn't exactly what I was seeing. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmmm.... sounds awful and I'm not at all opposed to just circumventing TS if it winds up causing too much trouble. |
||
import UMAPWorker from './umap.worker'; | ||
|
||
const requests = {}; | ||
|
||
function onError(id, error) { | ||
console.error(error); | ||
try { | ||
requests[id].reject(error.message); | ||
} finally { | ||
delete requests[id]; | ||
} | ||
} | ||
|
||
function onMessage({ data: { error, id, method, results } }) { | ||
if (error) { | ||
onError(id, new Error(error)); | ||
} | ||
switch (method) { | ||
case 'fit': { | ||
const { embedding } = results; | ||
if (requests[id]) { | ||
requests[id].resolve(embedding); | ||
} else { | ||
requests[id].reject( | ||
`Could not find existing UMAPWorker request with id ${id}` | ||
); | ||
} | ||
delete requests[id]; | ||
break; | ||
} | ||
default: { | ||
throw new Error( | ||
`Unknown method performed by UMAPWorker: 'umap.${method}'` | ||
); | ||
} | ||
} | ||
} | ||
|
||
const worker = new UMAPWorker(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Creating a worker instance should be in the constructor of the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
worker.onerror = onError; | ||
worker.onmessage = onMessage; | ||
|
||
class UMAPWorkerWrapper { | ||
fit(data) { | ||
const id = uuid(); | ||
worker.postMessage({ id, method: 'fit', args: [data] }); | ||
|
||
const promise = new Promise((resolve, reject) => { | ||
requests[id] = { resolve, reject }; | ||
}); | ||
|
||
return promise; | ||
} | ||
} | ||
|
||
export { UMAPWorkerWrapper as UMAP }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TypeScript is incorrectly seeing the
postMessage
s here aswindow.postMessage
s instead ofWorker.postMessage
s and thus wanting atargetOrigin
2nd argument. This is almost certainly related to the@ts-ignore
required to used the worker-loader on the module (see comment below). Getting the Worker to bundle via Webpack with TypeScript was the hardest part of this. I've bundled Workers into libs many times with the worker-loader and not had any problems in plain JS but Googling around confirms that this is indeed a common issue. (I'm also not sure how cross-browser friendly this bundling approach will be, either, since I'm not sure it works without{ fallback: false, inline: true }
and that may exclude some browsers...)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I had to deal with this once before and I wound up opting just to redefine a type for postMessage on the window object.... kind of equally hacky but at least you get type hints on that method. As far as the browser compatibility goes I think we can cross that bridge when we get there.