Bad CSAF ingestion performance

[ctron]

I noticed a massive drop in performance when running the CSAF import process. I don't have numbers yet, but I'll investigate.

In the SQL logs I noticed at of a "select then insert" patters again.

[ctron]

I tried to go back a bit in the git history to see if that's a recent change. Doesn't look like that's the case.

I started to work with the test data file cve-2023-33201.json. Which takes around 470ms to ingest. It looks like the majority of time goes into ingesting the vulnerabilities. Parsing, inserting, and committing the main entry takes around 70ms.

[ctron]

The majority of time (~390ms) goes into ingest_product_statuses.

[ctron]

I changed the title as I cannot provide any numbers. I just have the feeling it got much slower since some time ago.

[JimFuller-RedHat]

@ctron is this still an issue ?

[ctron]

Hm, good question. Could it be better. Probably, but we tried to optimize things already.

Still, importing that initial dataset still takes ~12 hours.

[helio-frota]

Generating flamegraph with cargo run --bin xtask --release generate-dump for ImporterConfiguration::Csaf(CsafImporter { only, generated this after some 2 or 3 minutes:

at lest has some relation with previous comments

Adding a zipped svg because when we click the preview svg in github comment, for some reason we can't navigate on flamegraph.. better to download unzip and click to open in the browser
d.zip

[helio-frota]

@ctron can we change the title , suggestion:

try to prevent asyn-runtime from stealing cpu cycles that slow down csaf ingestion performance (up to an acceptable level) ?

I was not able to "tokio::spawn_blocking" this due some weirdo lifetime things I'm not good yet.

But I noticed that it also helps a bit with tokio trying to poll causing more blocking situation

[ctron]

Let's close this for now. I'd expect the database to be the bottleneck (or how we use the DB). But let's focus on more specific issues.

[JimFuller-RedHat]

might be related/superseded by #999

[helio-frota]

good I'll try to find a way to cover the lines that starts with the upload part..
but in advance I have to say that I saw no deadlocks during some other tests like calling a function 1 million times inside test-binary-function.. probably some extra things happens before , in the case of issue you shared this is started by api endpoint .. def something to investigate

[helio-frota]

but in advance I have to say that I saw no deadlocks during some other tests like calling a function 1 million times inside test-binary-function..

probably some extra things happens before

is the parallel behavior #1009