Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bulk-disseminate Actions will fail if more than 256 submission candidates are found #22

Open
rhigman opened this issue Feb 20, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@rhigman
Copy link
Member

rhigman commented Feb 20, 2024

"A matrix will generate a maximum of 256 jobs per workflow run"

If the run-script stage of any bulk-disseminate Action finds more than 256 works which need to be disseminated based on the Action's inclusion criteria, the run will fail and none of the works will be disseminated. This is because the list of relevant work IDs is passed to a GitHub Actions matrix to create one submission job per work.

Additionally, due to the fact that the list of work IDs found by run-script is not output anywhere (e.g. in logs or artifacts), it's not possible to make up for the failure by disseminating the required works in a more manual fashion.

At present scale, this will rarely be an issue; in most cases, it would require the total number of works produced in a month by all publishers opted-in to a specific dissemination platform to exceed 256 (the current average is closer to 10). However, this did occur in a run of cr-bulk-disseminate, after a bulk database update caused ~6000 works to show up as having been updatedWithRelations within a single dissemination period.

(The update itself wasn't actually one which warranted mass resubmission of metadata anyway; we should consider how to handle similar future updates without unnecessarily overloading distribution platforms.)

Failure report:

image

@rhigman rhigman added the bug Something isn't working label Feb 20, 2024
@rhigman
Copy link
Member Author

rhigman commented Mar 7, 2024

Could investigate removing matrix and instead using GitHub API to trigger a new workflow run for each work (instead of a new job within a single run): use cron workflow to do run-script check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant