Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new: Implement new pipeline for running tasks/actions. #1050

Merged
merged 15 commits into from
Sep 14, 2023

Conversation

milesj
Copy link
Collaborator

@milesj milesj commented Sep 12, 2023

Our current pipeline works but there are a few issues with it, like signals not working properly, and synchronization between tasks being rudimentary.

So this PR implements a new pipeline that solves the following issues:

  • Run jobs in isolation, or in a batch.
  • Limits running jobs/concurrency (using a semaphore).
  • Use channels to send results to the parent pipeline.
  • Monitor signals (mainly ctrl+c) and cancel pending/running jobs.
  • Monitors running progress to track long-running jobs.
  • Abort running jobs if a sibling job fails.
  • Jobs can be configured with a timeout (this is new).

@milesj milesj merged commit cf89d8c into develop-1.14 Sep 14, 2023
25 checks passed
@milesj milesj deleted the 1.14-pipeline branch September 14, 2023 01:03
milesj added a commit that referenced this pull request Sep 18, 2023
* Start on pipeline.

* Hook up channels.

* Hook up steps and results.

* Add events.

* Rework job.

* Polish step/job.

* Polish.

* Add job action.

* Rework self.

* Start on tests.

* Polish.

* Remove assert.

* Add bail.

* Polish.

* Polish.
milesj added a commit that referenced this pull request Sep 19, 2023
* Start on pipeline.

* Hook up channels.

* Hook up steps and results.

* Add events.

* Rework job.

* Polish step/job.

* Polish.

* Add job action.

* Rework self.

* Start on tests.

* Polish.

* Remove assert.

* Add bail.

* Polish.

* Polish.
milesj added a commit that referenced this pull request Sep 25, 2023
* Start on pipeline.

* Hook up channels.

* Hook up steps and results.

* Add events.

* Rework job.

* Polish step/job.

* Polish.

* Add job action.

* Rework self.

* Start on tests.

* Polish.

* Remove assert.

* Add bail.

* Polish.

* Polish.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant