20-files-present-and-referenced: Check for files in *.obscpio + bonus fix

[Vogtinator]

.obscpio files are extracted before build and the contents can be used as
sources. This is not the only use case for obscpio though: most of the time
they carry the (large) source code, which is converted to a tarball by a
buildtime service. We can't easily tell those cases apart, so just don't
bother.

It would be possible to avoid false negatives here by checking against the cpio listing as fallback, but that might take very long for larger .obscpio archives containing source code.

[AdamMajer]

I'm not sure it would take a very long time, if the cpio contents are listed once and the list is stored. But it's additional feature though.

For context, 100MB cpio,

time cpio -ti < node_modules.obscpio

[.... snip ....]
yargs-parser-20.2.9.tgz
yocto-queue-0.1.0.tgz
218928 blocks

real    0m0.064s
user    0m0.007s
sys     0m0.057s

[AdamMajer]

I guess the skip should be temporary until cpio scanning is added.

[Vogtinator]

I'm not sure it would take a very long time, if the cpio contents are listed once and the list is stored. But it's additional feature though.

It's about 0.4s here for libqt5-qtbase. Might be possible to do this lazily only when first needed, then there's only the overhead of possibly scanning through >50k filenames. I'll have a try, maybe it's simple enough.

[AdamMajer]

You can also use something like systemsmanagement:cockpit/cockpit as a test case. It also includes quite a large number of sources in cpio, so if search is not scaling, it will show up there.

[Vogtinator]

You can also use something like systemsmanagement:cockpit/cockpit as a test case. It also includes quite a large number of sources in cpio, so if search is not scaling, it will show up there.

Took ~2.3s. For some reason cockpit-tukit has a larger node_modules (142MiB vs. 107MiB).

[AdamMajer]

I tried it locally, and there is basically no difference (in runtimes) between the version that reports errors by not scanning cpio and the current version that actually scans those files lists. I guess all the grep is not that heavy!

Runtime for me is also ~2.0s

LGTM! Thanks!

[dirkmueller]

not happy with unpacking all the obscpio archives, but don't yet see a better way