Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker: Update version listings #611

Merged
merged 22 commits into from
Feb 23, 2024
Merged

Conversation

josephjclark
Copy link
Collaborator

@josephjclark josephjclark commented Feb 23, 2024

  • Only print version numbers on workflow start
  • Allow an adaptor to report multiple versions

It's a bit too hard to do reliable, responsible adaptor version logging on step start. I looked at it for a while and I can't think of a good way.

The runtime WILL log the version of all imported modules (ie, the adaptor), so there is currently a record of exactly what version a step used. It's just a bit buried.

At least we have a fairly clear, accurate, high level list of all eeh versions in the workflow now. It's a bit more reliable and a bit less spammy than before.

Closes #574

Output

here's how this looks in lightning:

  • Notice that two versions are printed up at the top
  • If you look carefully down the log, you'll see that the runtime DOES list the actual versions loaded within each step
Memory limit: 500mb
Timeout: 300s
Versions:
    ▸ node.js                  18.12.1
    ▸ worker                   1.0.0
    ▸ @openfn/language-http    4.0.0, 6.0.0
Executing 178220f3-738f-42a2-9aeb-eccfb048a916
Starting step New job
[linker] loading module @openfn/language-http
[linker] Loading module @openfn/language-http from /tmp/openfn/worker/repo/node_modules/@openfn/language-http_6.0.0/dist/index.cjs
Resolved adaptor @openfn/language-http to version 6.0.0
Executing expression (2 operations)
Timeout set to 300000ms
Starting operation 1
hello
Operation 1 complete in 0ms
Starting operation 2
GET https://jsonplaceholder.typicode.com/todos/1 - 200 in 100ms
Operation 2 complete in 101ms
Expression complete!
Removed configuration from final state
Removed response from final state
Completed step New job in 309ms
Final memory usage: [step 19mb] [system 100mb]
Edge function (state) {
return Boolean(!state?.errors?.["b2b0b19e-917e-4177-8f9e-dd9aab7bc18d"] ?? true)
} returned true; 2d19cf04-0313-4f27-90ad-9843654ab08b will be executed next
Starting step postgres
[linker] loading module @openfn/language-http
[linker] Loading module @openfn/language-http from /tmp/openfn/worker/repo/node_modules/@openfn/language-http_4.0.0/lib/index.js
Resolved adaptor @openfn/language-http to version 4.0.0
Executing expression (1 operations)
Timeout set to 300000ms
Starting operation 1
GET request succeeded with 200 ✓
Operation 1 complete in 79ms
Expression complete!
Removed configuration from final state
Removed response from final state
Completed step postgres in 316ms
Final memory usage: [step 34mb] [system 124mb]
Run complete with status: success

Basically instead of looking up the adaptor version from the global list of options, we calculate the version for each step and pass that through to execution
This lets the runtime use the repo to install the correct versions of adaptors. It might run a little slower though
We only include versions on workflow start and allow multiple adaptor versions to be printed
@josephjclark josephjclark changed the base branch from main to workflow-multiple-adaptors February 23, 2024 15:09
@josephjclark josephjclark changed the title Update version printing Worker: Update version listings Feb 23, 2024
@josephjclark josephjclark marked this pull request as ready for review February 23, 2024 17:35
Copy link
Member

@taylordowns2000 taylordowns2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i like the output

Base automatically changed from workflow-multiple-adaptors to release/next February 23, 2024 17:44
@josephjclark josephjclark merged commit 58e0d11 into release/next Feb 23, 2024
5 checks passed
@josephjclark josephjclark deleted the better-version-printing branch February 23, 2024 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Split up Run Versions and Step Versions (worker)
2 participants