Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bump-lockfile is failing? #1579

Closed
cgwalters opened this issue Sep 21, 2023 · 3 comments
Closed

bump-lockfile is failing? #1579

cgwalters opened this issue Sep 21, 2023 · 3 comments
Labels
releng Related to Fedora Release Engineering team/input

Comments

@cgwalters
Copy link
Member

cgwalters commented Sep 21, 2023 •
edited
Loading

https://bodhi.fedoraproject.org/updates/FEDORA-2023-13b41d2e37 shipped a few days ago, but isn't in testing-devel yet.

Are we blocked on e.g. https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/bump-lockfile/754/ ? Something going wrong recently with ppc64le virt?
@cgwalters cgwalters added kind/bug releng Related to Fedora Release Engineering team/input and removed kind/bug labels Sep 21, 2023
@dustymabe
Copy link
Member

The ppc64le jobs runs in bump-lockfile started taking a lot longer recently.. There were a few possible culprits that we were taking aim at:

One thing we noted was the flakyness of tests on ppc64le were rising and I did put in a fix in coreos/coreos-assembler#3628 to try to make a single flake not delay the build as much as it might otherwise..

So far we don't have any answers on why things started taking much longer. We increased the timeout by 30m and it's still consistently timing out.

@cgwalters cgwalters mentioned this issue Sep 21, 2023
Merged
dustymabe added a commit to dustymabe/fedora-coreos-pipeline that referenced this issue Sep 22, 2023
@dustymabe
jobs/bump-lockfile: limit to single run at a time
4a5f591 
Right now when the bump-lockfiles job runs it kicks off two runs of
bump-lockfile (one for each stream) at the exact same time. This might
be what is causing our timeouts to be getting hit lately (primarily
because the ppc64le runs are taking longer) [1].

Let's limit the bump-lockfile job to a singular run at any given time.

[1] coreos/fedora-coreos-tracker#1579
@dustymabe dustymabe mentioned this issue Sep 22, 2023
Merged
@dustymabe
Copy link
Member

Another thing that happened recently is that we enabled the next-devel stream so now when bump-lockfiles runs (it's just a job that spawns the bump-lockfile job specifying what stream to run against) it now kicks off two bump-lockfile jobs: one for testing-devel and one for next-devel. These run simultaneously, which could be the reason why the ppc64le job is taking longer now.

Maybe this can help: coreos/fedora-coreos-pipeline#913

jlebon pushed a commit to coreos/fedora-coreos-pipeline that referenced this issue Sep 22, 2023
@dustymabe @jlebon
jobs/bump-lockfile: limit to single run at a time
f412e64 
Right now when the bump-lockfiles job runs it kicks off two runs of
bump-lockfile (one for each stream) at the exact same time. This might
be what is causing our timeouts to be getting hit lately (primarily
because the ppc64le runs are taking longer) [1].

Let's limit the bump-lockfile job to a singular run at any given time.

[1] coreos/fedora-coreos-tracker#1579
@dustymabe
Copy link
Member

Since coreos/fedora-coreos-pipeline#913 and we've had a few runs succeed that have been going serially I'll close this out and re-open it if we run into more problems.

aaradhak pushed a commit to aaradhak/fedora-coreos-pipeline that referenced this issue May 14, 2024
@dustymabe @aaradhak
jobs/bump-lockfile: limit to single run at a time
28cf0b9 
Right now when the bump-lockfiles job runs it kicks off two runs of
bump-lockfile (one for each stream) at the exact same time. This might
be what is causing our timeouts to be getting hit lately (primarily
because the ppc64le runs are taking longer) [1].

Let's limit the bump-lockfile job to a singular run at any given time.

[1] coreos/fedora-coreos-tracker#1579
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
releng Related to Fedora Release Engineering team/input
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cgwalters @dustymabe