Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Vendoring: timeout in root remote rawhide int tests #24463

Closed
edsantiago opened this issue Nov 4, 2024 · 10 comments
Closed

Vendoring: timeout in root remote rawhide int tests #24463

edsantiago opened this issue Nov 4, 2024 · 10 comments
Assignees
@giuseppe @mtrmac
Labels
flakes Flakes from Continuous Integration

Comments

@edsantiago
Copy link
Member

Three people independently vendoring in c-xxx, all three are seeing test timeouts. E.g.,

[+0753s] • [FAILED] [182.110 seconds]
         Podman something
         ...
           [FAILED] Timed out after 90.001s.
           command timed out after 90s: [/var/tmp/go/src/github.com/containers/podman/bin/podman-remote --remote --url unix:///run/podman/podman-2aef993ce0cc07023a7509e353c561760fff70fb0da7aa29d08680f8d0bb03ba.sock run -q --seccomp-policy default quay.io/libpod/testimage:20241011 ls]
           Expected process to exit.  It did not.
           In [It] at: /var/tmp/go/src/github.com[/containers/podman/test/utils/utils.go:370](https://github.com/containers/podman/blob/92b856722ed235d3c2f52b99444898531bcae00c/test/utils/utils.go#L370) @ 11/04/24 09:49:13.325
         
           Full Stack Trace
             github.com/containers/podman/v5/test/utils.(*PodmanSession).WaitWithTimeout(0xc000686a80, 0x5a)
            .....

Am filing this as a central point for gathering logs and data.

x x x x x x
int(6) remote(6) rawhide(6) root(6) host(6) sqlite(6)
@edsantiago edsantiago added the flakes Flakes from Continuous Integration label Nov 4, 2024
@Luap99
Copy link
Member

Luap99 commented Nov 5, 2024 

[+0950s]   SIGABRT: abort
[+0950s]   PC=0x4827e1 m=0 sigcode=0
[+0950s] 
[+0950s]   goroutine 25 gp=0xc0003876c0 m=0 mp=0x2e73360 [syscall, 3 minutes]:
[+0950s]   runtime.notetsleepg(0x2e9a820, 0xffffffffffffffff)
[+0950s]   	/usr/lib/golang/src/runtime/lock_futex.go:246 +0x29 fp=0xc00038e7a0 sp=0xc00038e778 pc=0x412689
[+0950s]   os/signal.signal_recv()
[+0950s]   	/usr/lib/golang/src/runtime/sigqueue.go:152 +0x29 fp=0xc00038e7c0 sp=0xc00038e7a0 pc=0x47a4c9
[+0950s]   os/signal.loop()
[+0950s]   	/usr/lib/golang/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00038e7e0 sp=0xc00038e7c0 pc=0x56d8b3
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00038e7e8 sp=0xc00038e7e0 pc=0x480a01
[+0950s]   created by os/signal.Notify.func1.1 in goroutine 1
[+0950s]   	/usr/lib/golang/src/os/signal/signal.go:151 +0x1f
[+0950s] 
[+0950s]   goroutine 1 gp=0xc0000081c0 m=nil [chan receive, 3 minutes]:
[+0950s]   runtime.gopark(0x17b686b?, 0x7faa73bee198?, 0x18?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc0006857c8 sp=0xc0006857a8 pc=0x47842e
[+0950s]   runtime.chanrecv(0xc0006435e0, 0xc0000798a0, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:639 +0x41c fp=0xc000685840 sp=0xc0006857c8 pc=0x40cabc
[+0950s]   runtime.chanrecv1(0x5f?, 0x1?)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:489 +0x12 fp=0xc000685868 sp=0xc000685840 pc=0x40c672
[+0950s]   github.com/containers/podman/v5/pkg/api/server.(*APIServer).Serve(0xc0001938c0)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/server.go:235 +0x168 fp=0xc0006858c0 sp=0xc000685868 pc=0x17b6bc8
[+0950s]   github.com/containers/podman/v5/cmd/podman/system.restService(0xc00061e800, 0x2e705a0, {{0x0, 0x0}, {0x0, 0x0}, 0x0, {0x7ffcb02ffaa0, 0x5f}})
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/cmd/podman/system/service_abi.go:138 +0xbd6 fp=0xc000685ad0 sp=0xc0006858c0 pc=0x17c4af6
[+0950s]   github.com/containers/podman/v5/cmd/podman/system.service(0x2da0f40, {0xc000626990?, 0x1?, 0x3?})
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/cmd/podman/system/service.go:102 +0x25e fp=0xc000685bc0 sp=0xc000685ad0 pc=0x17c3ade
[+0950s]   github.com/spf13/cobra.(*Command).execute(0x2da0f40, {0xc0001541f8, 0x3, 0x4})
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:985 +0xaaa fp=0xc000685d48 sp=0xc000685bc0 pc=0x90d02a
[+0950s]   github.com/spf13/cobra.(*Command).ExecuteC(0x2d8a500)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff fp=0xc000685e20 sp=0xc000685d48 pc=0x90d8ff
[+0950s]   github.com/spf13/cobra.(*Command).Execute(...)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:1041
[+0950s]   github.com/spf13/cobra.(*Command).ExecuteContext(...)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:1034
[+0950s]   main.Execute()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/cmd/podman/root.go:116 +0xb4 fp=0xc000685ea8 sp=0xc000685e20 pc=0x17d8374
[+0950s]   main.main()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/cmd/podman/main.go:61 +0x4b2 fp=0xc000685f50 sp=0xc000685ea8 pc=0x17d7ad2
[+0950s]   runtime.main()
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:272 +0x28b fp=0xc000685fe0 sp=0xc000685f50 pc=0x4413eb
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000685fe8 sp=0xc000685fe0 pc=0x480a01
[+0950s] 
[+0950s]   goroutine 2 gp=0xc000008c40 m=nil [force gc (idle), 3 minutes]:
[+0950s]   runtime.gopark(0xe473876b87?, 0x0?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000094fa8 sp=0xc000094f88 pc=0x47842e
[+0950s]   runtime.goparkunlock(...)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:430
[+0950s]   runtime.forcegchelper()
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:337 +0xb3 fp=0xc000094fe0 sp=0xc000094fa8 pc=0x441733
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000094fe8 sp=0xc000094fe0 pc=0x480a01
[+0950s]   created by runtime.init.7 in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:325 +0x1a
[+0950s] 
[+0950s]   goroutine 3 gp=0xc000009180 m=nil [GC sweep wait]:
[+0950s]   runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000095780 sp=0xc000095760 pc=0x47842e
[+0950s]   runtime.goparkunlock(...)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:430
[+0950s]   runtime.bgsweep(0xc00004a180)
[+0950s]   	/usr/lib/golang/src/runtime/mgcsweep.go:317 +0xdf fp=0xc0000957c8 sp=0xc000095780 pc=0x42947f
[+0950s]   runtime.gcenable.gowrap1()
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:203 +0x25 fp=0xc0000957e0 sp=0xc0000957c8 pc=0x41db65
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000957e8 sp=0xc0000957e0 pc=0x480a01
[+0950s]   created by runtime.gcenable in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:203 +0x66
[+0950s] 
[+0950s]   goroutine 4 gp=0xc000009340 m=nil [GC scavenge wait]:
[+0950s]   runtime.gopark(0x10000?, 0x2049310?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000095f78 sp=0xc000095f58 pc=0x47842e
[+0950s]   runtime.goparkunlock(...)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:430
[+0950s]   runtime.(*scavengerState).park(0x2e6d8e0)
[+0950s]   	/usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000095fa8 sp=0xc000095f78 pc=0x426e69
[+0950s]   runtime.bgscavenge(0xc00004a180)
[+0950s]   	/usr/lib/golang/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000095fc8 sp=0xc000095fa8 pc=0x4273f9
[+0950s]   runtime.gcenable.gowrap2()
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:204 +0x25 fp=0xc000095fe0 sp=0xc000095fc8 pc=0x41db05
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x480a01
[+0950s]   created by runtime.gcenable in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:204 +0xa5
[+0950s] 
[+0950s]   goroutine 18 gp=0xc000104700 m=nil [finalizer wait]:
[+0950s]   runtime.gopark(0x0?, 0x1e18da0?, 0x40?, 0x80?, 0x2000000020?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000094620 sp=0xc000094600 pc=0x47842e
[+0950s]   runtime.runfinq()
[+0950s]   	/usr/lib/golang/src/runtime/mfinal.go:193 +0x107 fp=0xc0000947e0 sp=0xc000094620 pc=0x41cbe7
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000947e8 sp=0xc0000947e0 pc=0x480a01
[+0950s]   created by runtime.createfing in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/mfinal.go:163 +0x3d
[+0950s] 
[+0950s]   goroutine 19 gp=0xc0002056c0 m=nil [chan receive, 3 minutes]:
[+0950s]   runtime.gopark(0xc000090760?, 0x61cc85?, 0x50?, 0xe1?, 0x208a440?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000090718 sp=0xc0000906f8 pc=0x47842e
[+0950s]   runtime.chanrecv(0xc000112a10, 0x0, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:639 +0x41c fp=0xc000090790 sp=0xc000090718 pc=0x40cabc
[+0950s]   runtime.chanrecv1(0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:489 +0x12 fp=0xc0000907b8 sp=0xc000090790 pc=0x40c672
[+0950s]   runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1732
[+0950s]   runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1735 +0x2f fp=0xc0000907e0 sp=0xc0000907b8 pc=0x420b6f
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x480a01
[+0950s]   created by unique.runtime_registerUniqueMapCleanup in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1730 +0x96
[+0950s] 
[+0950s]   goroutine 5 gp=0xc000009c00 m=nil [GC worker (idle), 3 minutes]:
[+0950s]   runtime.gopark(0xc87fd2fc3f?, 0x0?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000096738 sp=0xc000096718 pc=0x47842e
[+0950s]   runtime.gcBgMarkWorker(0xc0000601c0)
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000967c8 sp=0xc000096738 pc=0x41fe89
[+0950s]   runtime.gcBgMarkStartWorkers.gowrap1()
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1279 +0x25 fp=0xc0000967e0 sp=0xc0000967c8 pc=0x41fd65
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000967e8 sp=0xc0000967e0 pc=0x480a01
[+0950s]   created by runtime.gcBgMarkStartWorkers in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1279 +0x105
[+0950s] 
[+0950s]   goroutine 34 gp=0xc000386000 m=nil [GC worker (idle), 3 minutes]:
[+0950s]   runtime.gopark(0xc87f9efb08?, 0x1?, 0x93?, 0x3?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc00038c738 sp=0xc00038c718 pc=0x47842e
[+0950s]   runtime.gcBgMarkWorker(0xc0000601c0)
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1363 +0xe9 fp=0xc00038c7c8 sp=0xc00038c738 pc=0x41fe89
[+0950s]   runtime.gcBgMarkStartWorkers.gowrap1()
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1279 +0x25 fp=0xc00038c7e0 sp=0xc00038c7c8 pc=0x41fd65
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00038c7e8 sp=0xc00038c7e0 pc=0x480a01
[+0950s]   created by runtime.gcBgMarkStartWorkers in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1279 +0x105
[+0950s] 
[+0950s]   goroutine 20 gp=0xc000205a40 m=nil [GC worker (idle), 3 minutes]:
[+0950s]   runtime.gopark(0xc87fc2f9b6?, 0x0?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000090f38 sp=0xc000090f18 pc=0x47842e
[+0950s]   runtime.gcBgMarkWorker(0xc0000601c0)
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1363 +0xe9 fp=0xc000090fc8 sp=0xc000090f38 pc=0x41fe89
[+0950s]   runtime.gcBgMarkStartWorkers.gowrap1()
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1279 +0x25 fp=0xc000090fe0 sp=0xc000090fc8 pc=0x41fd65
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x480a01
[+0950s]   created by runtime.gcBgMarkStartWorkers in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1279 +0x105
[+0950s] 
[+0950s]   goroutine 21 gp=0xc000205c00 m=nil [GC worker (idle)]:
[+0950s]   runtime.gopark(0xe4741353c0?, 0x0?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000091738 sp=0xc000091718 pc=0x47842e
[+0950s]   runtime.gcBgMarkWorker(0xc0000601c0)
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000917c8 sp=0xc000091738 pc=0x41fe89
[+0950s]   runtime.gcBgMarkStartWorkers.gowrap1()
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1279 +0x25 fp=0xc0000917e0 sp=0xc0000917c8 pc=0x41fd65
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x480a01
[+0950s]   created by runtime.gcBgMarkStartWorkers in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/mgc.go:1279 +0x105
[+0950s] 
[+0950s]   goroutine 6 gp=0xc000386a80 m=nil [select, 3 minutes]:
[+0950s]   runtime.gopark(0xc000091f88?, 0x2?, 0xb8?, 0x75?, 0xc000091f84?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000091e28 sp=0xc000091e08 pc=0x47842e
[+0950s]   runtime.selectgo(0xc000091f88, 0xc000091f80, 0x0?, 0x0, 0x0?, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/select.go:335 +0x7a5 fp=0xc000091f50 sp=0xc000091e28 pc=0x4538e5
[+0950s]   database/sql.(*DB).connectionOpener(0xc000629790, {0x2069370, 0xc0004f4280})
[+0950s]   	/usr/lib/golang/src/database/sql/sql.go:1253 +0x87 fp=0xc000091fb8 sp=0xc000091f50 pc=0xc7ac27
[+0950s]   database/sql.OpenDB.gowrap1()
[+0950s]   	/usr/lib/golang/src/database/sql/sql.go:833 +0x28 fp=0xc000091fe0 sp=0xc000091fb8 pc=0xc78f48
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x480a01
[+0950s]   created by database/sql.OpenDB in goroutine 1
[+0950s]   	/usr/lib/golang/src/database/sql/sql.go:833 +0x130
[+0950s] 
[+0950s]   goroutine 23 gp=0xc000386c40 m=nil [chan receive, 3 minutes]:
[+0950s]   runtime.gopark(0xc000152120?, 0x0?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000092708 sp=0xc0000926e8 pc=0x47842e
[+0950s]   runtime.chanrecv(0xc0004f8000, 0xc0000927c8, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:639 +0x41c fp=0xc000092780 sp=0xc000092708 pc=0x40cabc
[+0950s]   runtime.chanrecv2(0xc0000927d0?, 0x4498fb?)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:494 +0x12 fp=0xc0000927a8 sp=0xc000092780 pc=0x40c692
[+0950s]   github.com/containers/podman/v5/libpod.(*Runtime).startWorker.func1()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/runtime_worker.go:11 +0x6c fp=0xc0000927e0 sp=0xc0000927a8 pc=0x152130c
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000927e8 sp=0xc0000927e0 pc=0x480a01
[+0950s]   created by github.com/containers/podman/v5/libpod.(*Runtime).startWorker in goroutine 1
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/runtime_worker.go:10 +0x4f
[+0950s] 
[+0950s]   goroutine 16 gp=0xc000386e00 m=nil [semacquire, 3 minutes, locked to thread]:
[+0950s]   runtime.gopark(0x2?, 0x48d135?, 0x0?, 0x20?, 0x151?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000842eb0 sp=0xc000842e90 pc=0x47842e
[+0950s]   runtime.goparkunlock(...)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:430
[+0950s]   runtime.semacquire1(0xc000481e48, 0x0, 0x1, 0x0, 0x12)
[+0950s]   	/usr/lib/golang/src/runtime/sema.go:178 +0x225 fp=0xc000842f18 sp=0xc000842eb0 pc=0x454985
[+0950s]   sync.runtime_Semacquire(0xc000a4cb70?)
[+0950s]   	/usr/lib/golang/src/runtime/sema.go:71 +0x25 fp=0xc000842f50 sp=0xc000842f18 pc=0x479985
[+0950s]   sync.(*WaitGroup).Wait(0xc00032a5a0?)
[+0950s]   	/usr/lib/golang/src/sync/waitgroup.go:118 +0x48 fp=0xc000842f78 sp=0xc000842f50 pc=0x48f5e8
[+0950s]   github.com/containers/podman/v5/libpod.(*Container).prepare(0xc000482a00)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/container_internal_linux.go:126 +0x298 fp=0xc000843100 sp=0xc000842f78 pc=0x14ae0b8
[+0950s]   github.com/containers/podman/v5/libpod.(*Container).prepareToStart(0xc000482a00, {0x2069338, 0xc0009359e0}, 0x0?)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/container_internal.go:850 +0x145 fp=0xc0008431d0 sp=0xc000843100 pc=0x1484b25
[+0950s]   github.com/containers/podman/v5/libpod.(*Container).Start(0xc000482a00, {0x2069338, 0xc0009359e0}, 0x1?)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/container_api.go:107 +0x105 fp=0xc000843258 sp=0xc0008431d0 pc=0x1461a05
[+0950s]   github.com/containers/podman/v5/pkg/api/handlers/compat.StartContainer({0x2067700, 0xc0006ee060}, 0xc0006a6280)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/handlers/compat/containers_start.go:39 +0x1fc fp=0xc000843300 sp=0xc000843258 pc=0x173023c
[+0950s]   github.com/containers/podman/v5/pkg/api/server.(*APIServer).apiWrapper(0xc0001938c0, 0x1e1a9a0, {0x2067700, 0xc0006ee060}, 0xc0006a6280, 0x0)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/handler_api.go:66 +0x9ca fp=0xc0008435f0 sp=0xc000843300 pc=0x179fcea
[+0950s]   github.com/containers/podman/v5/pkg/api/server.(*APIServer).registerContainersHandlers.(*APIServer).APIHandler.func53({0x2067700?, 0xc0006ee060?}, 0x1d1f3a7?)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/handler_api.go:26 +0x37 fp=0xc000843630 sp=0xc0008435f0 pc=0x17a4fd7
[+0950s]   net/http.HandlerFunc.ServeHTTP(0xc000935860?, {0x2067700?, 0xc0006ee060?}, 0xc00090ce30?)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:2220 +0x29 fp=0xc000843658 sp=0xc000843630 pc=0x77a3a9
[+0950s]   github.com/containers/podman/v5/pkg/api/server.newServer.referenceIDHandler.func7.1({0x2067700, 0xc0006ee060}, 0xc0006a6280)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/handler_rid.go:40 +0x14f fp=0xc0008436d0 sp=0xc000843658 pc=0x17b80ef
[+0950s]   net/http.HandlerFunc.ServeHTTP(0x2065650?, {0x2067700?, 0xc0006ee060?}, 0x0?)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:2220 +0x29 fp=0xc0008436f8 sp=0xc0008436d0 pc=0x77a3a9
[+0950s]   github.com/gorilla/handlers.loggingHandler.ServeHTTP({{0x2056e60, 0x2e98da0}, {0x20587a0, 0xc000906b40}, 0x1e1acd0}, {0x2065650, 0xc0006fa1c0}, 0xc0006a6280)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/gorilla/handlers/logging.go:47 +0xef fp=0xc000843948 sp=0xc0008436f8 pc=0x171faef
[+0950s]   github.com/gorilla/handlers.(*loggingHandler).ServeHTTP(0x130?, {0x2065650?, 0xc0006fa1c0?}, 0xc000bbfa30?)
[+0950s]   	<autogenerated>:1 +0x7e fp=0xc0008439c0 sp=0xc000843948 pc=0x172131e
[+0950s]   github.com/containers/podman/v5/pkg/api/server.newServer.panicHandler.func6.1({0x2065650?, 0xc0006fa1c0?}, 0xc0009359e0?)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/handler_panic.go:31 +0x63 fp=0xc000843a18 sp=0xc0008439c0 pc=0x17b7ce3
[+0950s]   net/http.HandlerFunc.ServeHTTP(0xc0006a6140?, {0x2065650?, 0xc0006fa1c0?}, 0x503369?)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:2220 +0x29 fp=0xc000843a40 sp=0xc000843a18 pc=0x77a3a9
[+0950s]   github.com/gorilla/mux.(*Router).ServeHTTP(0xc0001786c0, {0x2065650, 0xc0006fa1c0}, 0xc0006a6000)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2 fp=0xc000843b68 sp=0xc000843a40 pc=0x848702
[+0950s]   net/http.serverHandler.ServeHTTP({0xc0006311a0?}, {0x2065650?, 0xc0006fa1c0?}, 0x6?)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:3210 +0x8e fp=0xc000843b98 sp=0xc000843b68 pc=0x7998ae
[+0950s]   net/http.(*conn).serve(0xc0003a3320, {0x2069338, 0xc0006310e0})
[+0950s]   	/usr/lib/golang/src/net/http/server.go:2092 +0x5d0 fp=0xc000843fb8 sp=0xc000843b98 pc=0x778d50
[+0950s]   net/http.(*Server).Serve.gowrap3()
[+0950s]   	/usr/lib/golang/src/net/http/server.go:3360 +0x28 fp=0xc000843fe0 sp=0xc000843fb8 pc=0x77e1c8
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000843fe8 sp=0xc000843fe0 pc=0x480a01
[+0950s]   created by net/http.(*Server).Serve in goroutine 14
[+0950s]   	/usr/lib/golang/src/net/http/server.go:3360 +0x485
[+0950s] 
[+0950s]   goroutine 35 gp=0xc000387180 m=nil [chan receive, 3 minutes]:
[+0950s]   runtime.gopark(0xc0000c3140?, 0xc86085?, 0xd8?, 0x25?, 0xc000093ee8?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000093e40 sp=0xc000093e20 pc=0x47842e
[+0950s]   runtime.chanrecv(0xc000258380, 0x0, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:639 +0x41c fp=0xc000093eb8 sp=0xc000093e40 pc=0x40cabc
[+0950s]   runtime.chanrecv1(0xc000093ec8?, 0xc000093f88?)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:489 +0x12 fp=0xc000093ee0 sp=0xc000093eb8 pc=0x40c672
[+0950s]   github.com/containers/common/pkg/servicereaper.reaper(0xc000258380)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/containers/common/pkg/servicereaper/service.go:41 +0x33 fp=0xc000093fc8 sp=0xc000093ee0 pc=0x1287ad3
[+0950s]   github.com/containers/common/pkg/servicereaper.Start.gowrap1()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/containers/common/pkg/servicereaper/service.go:35 +0x25 fp=0xc000093fe0 sp=0xc000093fc8 pc=0x1287a65
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x480a01
[+0950s]   created by github.com/containers/common/pkg/servicereaper.Start in goroutine 1
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/containers/common/pkg/servicereaper/service.go:35 +0x96
[+0950s] 
[+0950s]   goroutine 11 gp=0xc000387340 m=nil [select]:
[+0950s]   runtime.gopark(0xc0000a7f98?, 0x2?, 0x10?, 0x86?, 0xc0000a7e94?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc0000a7c78 sp=0xc0000a7c58 pc=0x47842e
[+0950s]   runtime.selectgo(0xc0000a7f98, 0xc0000a7e90, 0x0?, 0x0, 0x2?, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/select.go:335 +0x7a5 fp=0xc0000a7da0 sp=0xc0000a7c78 pc=0x4538e5
[+0950s]   github.com/containers/podman/v5/libpod.(*Runtime).libimageEvents.func2()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/runtime.go:746 +0xda fp=0xc0000a7fe0 sp=0xc0000a7da0 pc=0x15043da
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a7fe8 sp=0xc0000a7fe0 pc=0x480a01
[+0950s]   created by github.com/containers/podman/v5/libpod.(*Runtime).libimageEvents in goroutine 1
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/runtime.go:719 +0xfc
[+0950s] 
[+0950s]   goroutine 24 gp=0xc000387500 m=nil [select, 3 minutes, locked to thread]:
[+0950s]   runtime.gopark(0xc00038dfa8?, 0x2?, 0x0?, 0x75?, 0xc00038df94?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc00038de30 sp=0xc00038de10 pc=0x47842e
[+0950s]   runtime.selectgo(0xc00038dfa8, 0xc00038df90, 0x0?, 0x0, 0x0?, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/select.go:335 +0x7a5 fp=0xc00038df58 sp=0xc00038de30 pc=0x4538e5
[+0950s]   runtime.ensureSigM.func1()
[+0950s]   	/usr/lib/golang/src/runtime/signal_unix.go:1060 +0x19f fp=0xc00038dfe0 sp=0xc00038df58 pc=0x46f39f
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00038dfe8 sp=0xc00038dfe0 pc=0x480a01
[+0950s]   created by runtime.ensureSigM in goroutine 1
[+0950s]   	/usr/lib/golang/src/runtime/signal_unix.go:1043 +0xc8
[+0950s] 
[+0950s]   goroutine 26 gp=0xc000387880 m=nil [select, 3 minutes]:
[+0950s]   runtime.gopark(0xc00038ef90?, 0x2?, 0x0?, 0x0?, 0xc00038ee9c?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc00038ed38 sp=0xc00038ed18 pc=0x47842e
[+0950s]   runtime.selectgo(0xc00038ef90, 0xc00038ee98, 0x0?, 0x0, 0x0?, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/select.go:335 +0x7a5 fp=0xc00038ee60 sp=0xc00038ed38 pc=0x4538e5
[+0950s]   github.com/containers/podman/v5/libpod/shutdown.Start.func1()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/shutdown/handler.go:56 +0x87 fp=0xc00038efe0 sp=0xc00038ee60 pc=0x1332687
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00038efe8 sp=0xc00038efe0 pc=0x480a01
[+0950s]   created by github.com/containers/podman/v5/libpod/shutdown.Start in goroutine 1
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/shutdown/handler.go:55 +0xe5
[+0950s] 
[+0950s]   goroutine 12 gp=0xc0002048c0 m=nil [chan receive, 3 minutes]:
[+0950s]   runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc0003896d8 sp=0xc0003896b8 pc=0x47842e
[+0950s]   runtime.chanrecv(0xc0004e8150, 0x0, 0x1)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:639 +0x41c fp=0xc000389750 sp=0xc0003896d8 pc=0x40cabc
[+0950s]   runtime.chanrecv1(0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/chan.go:489 +0x12 fp=0xc000389778 sp=0xc000389750 pc=0x40c672
[+0950s]   github.com/containers/podman/v5/pkg/domain/infra.StartWatcher.func1()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:318 +0x56 fp=0xc0003897e0 sp=0xc000389778 pc=0x15ecdb6
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0003897e8 sp=0xc0003897e0 pc=0x480a01
[+0950s]   created by github.com/containers/podman/v5/pkg/domain/infra.StartWatcher in goroutine 1
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:314 +0xa7
[+0950s] 
[+0950s]   goroutine 41 gp=0xc0005f5500 m=11 mp=0xc0003ea008 [syscall, 3 minutes, locked to thread]:
[+0950s]   runtime.cgocall(0x18d02b0, 0xc0007da3e8)
[+0950s]   	/usr/lib/golang/src/runtime/cgocall.go:167 +0x4b fp=0xc0007da3c0 sp=0xc0007da388 pc=0x4718cb
[+0950s]   github.com/containers/podman/v5/libpod/lock/shm._Cfunc_lock_semaphore(0x7faa7305b000, 0x50)
[+0950s]   	_cgo_gotypes.go:242 +0x4b fp=0xc0007da3e8 sp=0xc0007da3c0 pc=0x1237ccb
[+0950s]   github.com/containers/podman/v5/libpod/lock/shm.(*SHMLocks).LockSemaphore(0xc000334100, 0x50)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/lock/shm/shm_lock.go:227 +0x4e fp=0xc0007da460 sp=0xc0007da3e8 pc=0x1238dae
[+0950s]   github.com/containers/podman/v5/libpod/lock.(*SHMLock).Lock(0xc0007da4b8?)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/lock/shm_lock_manager_linux.go:127 +0x1f fp=0xc0007da480 sp=0xc0007da460 pc=0x1239d9f
[+0950s]   github.com/containers/podman/v5/libpod.(*Container).Batch(0xc0003fe1e0, 0xc0007da8f8)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/container_api.go:887 +0x3f fp=0xc0007da4e0 sp=0xc0007da480 pc=0x146793f
[+0950s]   github.com/containers/podman/v5/pkg/ps.ListContainerBatch(_, _, {0x1, 0xc000a52bd0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x1, ...})
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/ps/ps.go:158 +0x30c fp=0xc0007daa10 sp=0xc0007da4e0 pc=0x1587d2c
[+0950s]   github.com/containers/podman/v5/pkg/ps.GetContainerLists(0xc0003868c0, {0x1, 0xc000a52bd0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x1, 0x0, ...})
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/ps/ps.go:77 +0x509 fp=0xc0007db180 sp=0xc0007daa10 pc=0x15872a9
[+0950s]   github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).ContainerList(...)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/domain/infra/abi/containers.go:1058
[+0950s]   github.com/containers/podman/v5/pkg/api/handlers/libpod.ListContainers({0x2067700, 0xc0007a97a0}, 0xc0004cc640)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/handlers/libpod/containers.go:117 +0x478 fp=0xc0007db300 sp=0xc0007db180 pc=0x1751498
[+0950s]   github.com/containers/podman/v5/pkg/api/server.(*APIServer).apiWrapper(0xc0001938c0, 0x1e1ab08, {0x2067700, 0xc0007a97a0}, 0xc0004cc640, 0x0)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/handler_api.go:66 +0x9ca fp=0xc0007db5f0 sp=0xc0007db300 pc=0x179fcea
[+0950s]   github.com/containers/podman/v5/pkg/api/server.(*APIServer).registerContainersHandlers.(*APIServer).APIHandler.func42({0x2067700?, 0xc0007a97a0?}, 0x1d1f3a7?)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/handler_api.go:26 +0x37 fp=0xc0007db630 sp=0xc0007db5f0 pc=0x17a5557
[+0950s]   net/http.HandlerFunc.ServeHTTP(0xc000a52600?, {0x2067700?, 0xc0007a97a0?}, 0xc000226110?)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:2220 +0x29 fp=0xc0007db658 sp=0xc0007db630 pc=0x77a3a9
[+0950s]   github.com/containers/podman/v5/pkg/api/server.newServer.referenceIDHandler.func7.1({0x2067700, 0xc0007a97a0}, 0xc0004cc640)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/handler_rid.go:40 +0x14f fp=0xc0007db6d0 sp=0xc0007db658 pc=0x17b80ef
[+0950s]   net/http.HandlerFunc.ServeHTTP(0x2065650?, {0x2067700?, 0xc0007a97a0?}, 0x0?)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:2220 +0x29 fp=0xc0007db6f8 sp=0xc0007db6d0 pc=0x77a3a9
[+0950s]   github.com/gorilla/handlers.loggingHandler.ServeHTTP({{0x2056e60, 0x2e98da0}, {0x20587a0, 0xc0008d4b40}, 0x1e1acd0}, {0x2065650, 0xc0006a8000}, 0xc0004cc640)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/gorilla/handlers/logging.go:47 +0xef fp=0xc0007db948 sp=0xc0007db6f8 pc=0x171faef
[+0950s]   github.com/gorilla/handlers.(*loggingHandler).ServeHTTP(0x130?, {0x2065650?, 0xc0006a8000?}, 0xc0000a7a30?)
[+0950s]   	<autogenerated>:1 +0x7e fp=0xc0007db9c0 sp=0xc0007db948 pc=0x172131e
[+0950s]   github.com/containers/podman/v5/pkg/api/server.newServer.panicHandler.func6.1({0x2065650?, 0xc0006a8000?}, 0xc000a52a50?)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/handler_panic.go:31 +0x63 fp=0xc0007dba18 sp=0xc0007db9c0 pc=0x17b7ce3
[+0950s]   net/http.HandlerFunc.ServeHTTP(0xc0004cc280?, {0x2065650?, 0xc0006a8000?}, 0x503369?)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:2220 +0x29 fp=0xc0007dba40 sp=0xc0007dba18 pc=0x77a3a9
[+0950s]   github.com/gorilla/mux.(*Router).ServeHTTP(0xc0001786c0, {0x2065650, 0xc0006a8000}, 0xc0004cc000)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2 fp=0xc0007dbb68 sp=0xc0007dba40 pc=0x848702
[+0950s]   net/http.serverHandler.ServeHTTP({0xc000a52540?}, {0x2065650?, 0xc0006a8000?}, 0x6?)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:3210 +0x8e fp=0xc0007dbb98 sp=0xc0007dbb68 pc=0x7998ae
[+0950s]   net/http.(*conn).serve(0xc00072a240, {0x2069338, 0xc000a52450})
[+0950s]   	/usr/lib/golang/src/net/http/server.go:2092 +0x5d0 fp=0xc0007dbfb8 sp=0xc0007dbb98 pc=0x778d50
[+0950s]   net/http.(*Server).Serve.gowrap3()
[+0950s]   	/usr/lib/golang/src/net/http/server.go:3360 +0x28 fp=0xc0007dbfe0 sp=0xc0007dbfb8 pc=0x77e1c8
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0007dbfe8 sp=0xc0007dbfe0 pc=0x480a01
[+0950s]   created by net/http.(*Server).Serve in goroutine 14
[+0950s]   	/usr/lib/golang/src/net/http/server.go:3360 +0x485
[+0950s] 
[+0950s]   goroutine 14 gp=0xc0005f56c0 m=nil [IO wait, 3 minutes]:
[+0950s]   runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc0000a8bf0 sp=0xc0000a8bd0 pc=0x47842e
[+0950s]   runtime.netpollblock(0x1a4d840?, 0x409ec6?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000a8c28 sp=0xc0000a8bf0 pc=0x439d97
[+0950s]   internal/poll.runtime_pollWait(0x7faa73b87ec8, 0x72)
[+0950s]   	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0000a8c48 sp=0xc0000a8c28 pc=0x4772e5
[+0950s]   internal/poll.(*pollDesc).wait(0xc0000dc000?, 0xe?, 0x0)
[+0950s]   	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000a8c70 sp=0xc0000a8c48 pc=0x502f67
[+0950s]   internal/poll.(*pollDesc).waitRead(...)
[+0950s]   	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
[+0950s]   internal/poll.(*FD).Accept(0xc0000dc000)
[+0950s]   	/usr/lib/golang/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc0000a8d18 sp=0xc0000a8c70 pc=0x508335
[+0950s]   net.(*netFD).accept(0xc0000dc000)
[+0950s]   	/usr/lib/golang/src/net/fd_unix.go:172 +0x29 fp=0xc0000a8dd0 sp=0xc0000a8d18 pc=0x63f449
[+0950s]   net.(*UnixListener).accept(0xc0000a8e30?)
[+0950s]   	/usr/lib/golang/src/net/unixsock_posix.go:172 +0x16 fp=0xc0000a8df8 sp=0xc0000a8dd0 pc=0x65ffd6
[+0950s]   net.(*UnixListener).Accept(0xc0006502a0)
[+0950s]   	/usr/lib/golang/src/net/unixsock.go:260 +0x30 fp=0xc0000a8e28 sp=0xc0000a8df8 pc=0x65e490
[+0950s]   net/http.(*onceCloseListener).Accept(0xc00072a240?)
[+0950s]   	<autogenerated>:1 +0x24 fp=0xc0000a8e40 sp=0xc0000a8e28 pc=0x7a5ec4
[+0950s]   net/http.(*Server).Serve(0xc0001938c0, {0x2065290, 0xc0006502a0})
[+0950s]   	/usr/lib/golang/src/net/http/server.go:3330 +0x30c fp=0xc0000a8f70 sp=0xc0000a8e40 pc=0x77ddcc
[+0950s]   github.com/containers/podman/v5/pkg/api/server.(*APIServer).Serve.func3()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/server.go:227 +0x32 fp=0xc0000a8fe0 sp=0xc0000a8f70 pc=0x17b6c32
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a8fe8 sp=0xc0000a8fe0 pc=0x480a01
[+0950s]   created by github.com/containers/podman/v5/pkg/api/server.(*APIServer).Serve in goroutine 1
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/pkg/api/server/server.go:226 +0x153
[+0950s] 
[+0950s]   goroutine 42 gp=0xc0006aafc0 m=nil [IO wait, 3 minutes]:
[+0950s]   runtime.gopark(0x0?, 0x0?, 0xc0?, 0xaf?, 0xb?)
[+0950s]   	/usr/lib/golang/src/runtime/proc.go:424 +0xce fp=0xc000096da8 sp=0xc000096d88 pc=0x47842e
[+0950s]   runtime.netpollblock(0x4e1f98?, 0x409ec6?, 0x0?)
[+0950s]   	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000096de0 sp=0xc000096da8 pc=0x439d97
[+0950s]   internal/poll.runtime_pollWait(0x7faa73b87cb8, 0x72)
[+0950s]   	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000096e00 sp=0xc000096de0 pc=0x4772e5
[+0950s]   internal/poll.(*pollDesc).wait(0xc00046c100?, 0xc000a52551?, 0x0)
[+0950s]   	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000096e28 sp=0xc000096e00 pc=0x502f67
[+0950s]   internal/poll.(*pollDesc).waitRead(...)
[+0950s]   	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
[+0950s]   internal/poll.(*FD).Read(0xc00046c100, {0xc000a52551, 0x1, 0x1})
[+0950s]   	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc000096ec0 sp=0xc000096e28 pc=0x50425a
[+0950s]   net.(*netFD).Read(0xc00046c100, {0xc000a52551?, 0x128ef58?, 0x1d37a85?})
[+0950s]   	/usr/lib/golang/src/net/fd_posix.go:55 +0x25 fp=0xc000096f08 sp=0xc000096ec0 pc=0x63d485
[+0950s]   net.(*conn).Read(0xc000a620c8, {0xc000a52551?, 0x0?, 0x0?})
[+0950s]   	/usr/lib/golang/src/net/net.go:189 +0x45 fp=0xc000096f50 sp=0xc000096f08 pc=0x64f225
[+0950s]   net.(*UnixConn).Read(0x128ef80?, {0xc000a52551?, 0xc83db8?, 0xc0004820a0?})
[+0950s]   	<autogenerated>:1 +0x25 fp=0xc000096f80 sp=0xc000096f50 pc=0x667ae5
[+0950s]   net/http.(*connReader).backgroundRead(0xc000a52540)
[+0950s]   	/usr/lib/golang/src/net/http/server.go:690 +0x37 fp=0xc000096fc8 sp=0xc000096f80 pc=0x7736d7
[+0950s]   net/http.(*connReader).startBackgroundRead.gowrap2()
[+0950s]   	/usr/lib/golang/src/net/http/server.go:686 +0x25 fp=0xc000096fe0 sp=0xc000096fc8 pc=0x773605
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000096fe8 sp=0xc000096fe0 pc=0x480a01
[+0950s]   created by net/http.(*connReader).startBackgroundRead in goroutine 41
[+0950s]   	/usr/lib/golang/src/net/http/server.go:686 +0xb6
[+0950s] 
[+0950s]   goroutine 29 gp=0xc0006aa1c0 m=10 mp=0xc00063fc08 [syscall, 3 minutes]:
[+0950s]   syscall.Syscall(0x102, 0x8, 0xc00023859c, 0x1ed)
[+0950s]   	/usr/lib/golang/src/syscall/syscall_linux.go:73 +0x25 fp=0xc0009af988 sp=0xc0009af928 pc=0x4e3fe5
[+0950s]   syscall.Syscall(0x102, 0x8, 0xc00023859c, 0x1ed)
[+0950s]   	<autogenerated>:1 +0x2e fp=0xc0009af9b8 sp=0xc0009af988 pc=0x4e4d8e
[+0950s]   golang.org/x/sys/unix.Mkdirat(0x8, {0x1d0c77a?, 0x0?}, 0x1ed)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1255 +0x85 fp=0xc0009afa18 sp=0xc0009af9b8 pc=0x578ba5
[+0950s]   github.com/containers/podman/v5/libpod.(*Container).mountStorage(0xc000482a00)
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/container_internal.go:1767 +0xe72 fp=0xc0009aff20 sp=0xc0009afa18 pc=0x148a052
[+0950s]   github.com/containers/podman/v5/libpod.(*Container).prepare.func2()
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/container_internal_linux.go:110 +0x85 fp=0xc0009affe0 sp=0xc0009aff20 pc=0x14ae685
[+0950s]   runtime.goexit({})
[+0950s]   	/usr/lib/golang/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0009affe8 sp=0xc0009affe0 pc=0x480a01
[+0950s]   created by github.com/containers/podman/v5/libpod.(*Container).prepare in goroutine 16
[+0950s]   	/var/tmp/go/src/github.com/containers/podman/libpod/container_internal_linux.go:108 +0x28b
[+0950s] 
[+0950s]   rax    0xca
[+0950s]   rbx    0x0
[+0950s]   rcx    0x4827e3
[+0950s]   rdx    0x0
[+0950s]   rdi    0x2e9a820
[+0950s]   rsi    0x80
[+0950s]   rbp    0xc00038e728
[+0950s]   rsp    0xc00038e6e0
[+0950s]   r8     0x0
[+0950s]   r9     0x0
[+0950s]   r10    0x0
[+0950s]   r11    0x286
[+0950s]   r12    0x0
[+0950s]   r13    0x8
[+0950s]   r14    0xc0003876c0
[+0950s]   r15    0xc000520480
[+0950s]   rip    0x4827e1
[+0950s]   rflags 0x286
[+0950s]   cs     0x33
[+0950s]   fs     0x0
[+0950s]   gs     0x0

https://cirrus-ci.com/task/4973455794241536

Using 80fc34e I got us a server stack trace which I think is more interesting then the client one we get by default.

goroutine 29 hangs on unix.Mkdirat(dirfd, "etc", 0755) for 3 minutes?! That seems very odd in the stack trace.

I see some other hang on the container lock in contianer list but that is expected as the one container is stucked is holding a container lock so no listing will work.

@Luap99
Copy link
Member

Luap99 commented Nov 5, 2024

Oh and for those who don't know rawhide is using composefs to test so my guess would be it has to do something with that.
--storage-driver overlay --storage-opt overlay.use_composefs=true --pull-option=enable_partial_images=true --pull-option=convert_images=true

@mtrmac
Copy link
Collaborator

mtrmac commented Nov 5, 2024

Is it possible that we are stuck in kernel-land?

@Luap99
Copy link
Member

Luap99 commented Nov 5, 2024 •
edited
Loading

yes, that is what the stack trace would make me believe. mkdirat() should never take 3 mins I would say.

And on that note I regret not taking a quick look in the journal before because this looks bad:

Nov 04 09:53:46 cirrus-task-6327338403627008 audit: BPF prog-id=1016 op=LOAD
Nov 04 09:53:46 cirrus-task-6327338403627008 kernel: erofs: (device erofs): mounted with root inode @ nid 36.
Nov 04 09:53:46 cirrus-task-6327338403627008 kernel: erofs: (device erofs): mounted with root inode @ nid 36.
Nov 04 09:53:46 cirrus-task-6327338403627008 kernel: erofs: (device erofs): mounted with root inode @ nid 36.
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: BUG: unable to handle page fault for address: 000000000002a200
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: #PF: supervisor read access in kernel mode
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: #PF: error_code(0x0000) - not-present page
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: PGD 800000010b326067 P4D 800000010b326067 PUD 10b327067 PMD 0 
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: Oops: Oops: 0000 [#4] PREEMPT SMP PTI
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: CPU: 3 UID: 0 PID: 128738 Comm: podman Tainted: G      D           -------  ---  6.12.0-0.rc2.20241010gitd3d1556696c1.27.fc42.x86_64 #1
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: Tainted: [D]=DIE
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RIP: 0010:workingset_refault+0x86/0x3a0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: Code: 89 e8 48 89 ef 48 c1 e8 02 48 c1 ef 0c 25 ff 03 00 00 0f b7 ff 4c 8b 34 c5 60 ab 6b 9b e8 62 a0 0b 00 0f 1f 44 00 00 48 85 c0 <49> 63 96 00 a2 02 00 48 0f 44 05 f3 5b 66 02 4c 8b ac d0 78 09 00
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RSP: 0018:ffffb62dc755f658 EFLAGS: 00010246
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RAX: 0000000000000000 RBX: 000fffffc0000001 RCX: 0000000000000006
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff8c600207a9c0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RBP: 00000000000896ab R08: 0000000000000001 R09: ffffb62dc755f638
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000001
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffffe96b82b37bc0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: FS:  00007f34acc006c0(0000) GS:ffff8c603bd80000(0000) knlGS:0000000000000000
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: CR2: 000000000002a200 CR3: 0000000109cf4006 CR4: 00000000001726f0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: DR0: 00007f17311bfa10 DR1: 0000000000000000 DR2: 0000000000000000
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: Call Trace:
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  <TASK>
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? __die_body.cold+0x19/0x27
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? page_fault_oops+0x15a/0x2f0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? search_module_extables+0x19/0x60
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? search_bpf_extables+0x5f/0x80
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? exc_page_fault+0x7e/0x180
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? asm_exc_page_fault+0x26/0x30
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? workingset_refault+0x86/0x3a0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? workingset_refault+0x7e/0x3a0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  filemap_add_folio+0xb0/0xe0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  do_read_cache_folio+0x113/0x1b0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  erofs_bread+0x95/0x210 [erofs]
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  erofs_iget+0x132/0x930 [erofs]
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  erofs_lookup+0xe9/0x100 [erofs]
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  __lookup_slow+0x86/0x130
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  lookup_one_unlocked+0x93/0xb0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ovl_lookup_single+0x7b/0x380 [overlay]
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ovl_lookup_layer+0x12d/0x140 [overlay]
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ovl_lookup+0x72e/0xd80 [overlay]
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  lookup_one_qstr_excl+0x6f/0xa0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  filename_create+0xc6/0x1a0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  do_mkdirat+0x61/0x180
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  __x64_sys_mkdirat+0x4a/0x70
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  do_syscall_64+0x82/0x160
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? __legitimize_path+0x2a/0x60
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? try_to_unlazy+0x5a/0xc0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? path_get+0x15/0x30
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? do_dentry_open+0x57/0x4a0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? mntput_no_expire+0x4a/0x260
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? path_openat+0x99a/0x12e0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? ovl_getattr+0xaf/0x450 [overlay]
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? do_filp_open+0xc4/0x170
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? syscall_exit_to_user_mode_prepare+0x112/0x170
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? do_syscall_64+0x8e/0x160
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? syscall_exit_to_user_mode_prepare+0x112/0x170
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? do_syscall_64+0x8e/0x160
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? do_syscall_64+0x8e/0x160
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? clear_bhb_loop+0x25/0x80
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? clear_bhb_loop+0x25/0x80
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  ? clear_bhb_loop+0x25/0x80
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RIP: 0033:0x485fae
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: Code: 24 28 44 8b 44 24 2c e9 70 ff ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RSP: 002b:000000c00118b8d8 EFLAGS: 00000202 ORIG_RAX: 0000000000000102
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 0000000000485fae
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RDX: 00000000000001ed RSI: 000000c00040895c RDI: 0000000000000008
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RBP: 000000c00118b918 R08: 0000000000000000 R09: 0000000000000000
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 000000c000408950
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: R13: 0000000000000000 R14: 000000c00093d500 R15: 0003ffffffffffdf
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel:  </TASK>
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: Modules linked in: ipvlan macvlan dummy nf_conntrack_netlink xt_mark ip6table_filter iptable_filter brd nft_nat tun nft_ct nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables veth bridge erofs netfs overlay 8021q garp mrp stp llc sunrpc intel_rapl_msr intel_rapl_common binfmt_misc kvm_intel kvm vfat fat rapl virtio_net i2c_piix4 net_failover failover i2c_smbus pvpanic_mmio pvpanic fuse ip6_tables ip_tables loop nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci zram crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 virtio_scsi serio_raw nvme nvme_core nvme_auth
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: CR2: 000000000002a200
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: ---[ end trace 0000000000000000 ]---
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RIP: 0010:workingset_refault+0x86/0x3a0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: Code: 89 e8 48 89 ef 48 c1 e8 02 48 c1 ef 0c 25 ff 03 00 00 0f b7 ff 4c 8b 34 c5 60 ab 6b 9b e8 62 a0 0b 00 0f 1f 44 00 00 48 85 c0 <49> 63 96 00 a2 02 00 48 0f 44 05 f3 5b 66 02 4c 8b ac d0 78 09 00
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RSP: 0018:ffffb62dc2343908 EFLAGS: 00010286
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RAX: ffff8c5f23af0000 RBX: 000fffffc0000001 RCX: 0000000000000000
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8c5f23a5a9c0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: RBP: 0000000000051521 R08: 0000000000000001 R09: ffffb62dc23438e8
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000001
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffffe96b80b33340
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: FS:  00007f34acc006c0(0000) GS:ffff8c603bd80000(0000) knlGS:0000000000000000
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: CR2: 000000000002a200 CR3: 0000000109cf4006 CR4: 00000000001726f0
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: DR0: 00007f17311bfa10 DR1: 0000000000000000 DR2: 0000000000000000
Nov 04 09:53:47 cirrus-task-6327338403627008 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600

Sounds like we are hitting a bug in erofs in the kernel... I will check the other hang logs to see if we have the same trace there

@Luap99
Copy link
Member

Luap99 commented Nov 5, 2024

Confirmed we are hitting a kernel bug, I see the same trace in all linked logs.
Though one should note we are using kernel 6.12.0-0.rc2.20241010gitd3d1556696c1.27.fc42.x86_64 so not exactly a production ready kernel. We should try to get this run with newer CI images/kernels I will try to spin up with newer builds to see if they work or not.
This would explain why we were unable to reproduce locally as well

@Luap99
Copy link
Member

Luap99 commented Nov 5, 2024

Good news, it passed in #24447 https://cirrus-ci.com/task/5246645728706560
I will repush once again to confirm it is not a flake.

These images have other known issues so we cannot just merge them unfortunately. Hopefully tomorrow we might can build new images that actually can pass CI so we can put this behind us.
But I guess at least we do not have to spend time actually finding the root cause here

@mtrmac
Copy link
Collaborator

mtrmac commented Nov 5, 2024

To confirm another way, I have added 2879646 + eb52100 to #24462 . That seems to be the only obvious composefs-kernel-related part, let’s see if that makes a difference.

@mtrmac mtrmac mentioned this issue Nov 5, 2024
Closed
@Luap99
Copy link
Member

Luap99 commented Nov 5, 2024

Ok passed again in my PR, https://cirrus-ci.com/task/5752469513306112
kernel in the images is 6.12.0-0.rc5.20241101git6c52d4da1c74.48.fc42.x86_64

I think we should be able to get new images into CI by next Tuesday so I don't think we need to block the vendor dance on it

@mtrmac
Copy link
Collaborator

mtrmac commented Nov 5, 2024

To confirm another way, I have added 2879646 + eb52100 to #24462 . That seems to be the only obvious composefs-kernel-related part, let’s see if that makes a difference.

I’m still seeing a panic there, so that guess didn’t pan out.

@Luap99 Luap99 mentioned this issue Nov 6, 2024
Closed
@Luap99
Copy link
Member

Luap99 commented Nov 7, 2024

#24447 passed CI and is merged so this can be closed

@Luap99 Luap99 closed this as completed Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@giuseppe giuseppe

@mtrmac mtrmac

Labels
flakes Flakes from Continuous Integration
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@giuseppe @edsantiago @mtrmac @Luap99