[AppSec] Head request is delayed due to other writes on DB

[LaurenceJJones]

Confirmed database types issue:

• Sqlite

other databases should not be affected as they use a connection pool since Sqlite by design is a single writer and if WAL is enabled multiple readers.

When using AppSec component you may sometimes see a context.deadline exceeded error due to the database writer being held by a community blocklist update. By default the header request has a default 200ms timeout to be responsive as possible, but since the writer is held it cannot get a response fast enough.

CrowdSec Logs:

time="2024-10-15T08:23:49Z" level=info msg="Starting community-blocklist update"
time="2024-10-15T08:23:54Z" level=info msg="capi/community-blocklist : 0 explicit deletions"
time="2024-10-15T08:23:55Z" level=error msg="Error performing request: Head \"http://127.0.0.1:8080/v1/decisions/stream\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
time="2024-10-15T08:23:55Z" level=error msg="Unauthorized request from '@' (real IP = <redacted_wan_ip>)" type=appsec
time="2024-10-15T08:23:58Z" level=info msg="crowdsecurity/community-blocklist : added 49800 entries, deleted 49669 entries (alert:127938)"
time="2024-10-15T08:23:58Z" level=info msg="lists:free_proxies : added 5655 entries, deleted 5655 entries (alert:127939)"
time="2024-10-15T08:23:59Z" level=info msg="lists:tor-exit-nodes : added 2192 entries, deleted 2192 entries (alert:127940)"
time="2024-10-15T08:23:59Z" level=info msg="lists:crowdsec_public_scanners : added 9618 entries, deleted 9618 entries (alert:127941)"

LAPI Response timer:

time="2024-10-15T08:23:58Z" level=info msg="127.0.0.1 - [Tue, 15 Oct 2024 08:23:58 UTC] \"HEAD /v1/decisions/stream HTTP/1.1 200 2.334997401s \"Go-http-client/1.1\" \""

[github-actions]

@LaurenceJJones: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@LaurenceJJones: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.