How one should manage the retention policy with Quickwit?

[zbalkan]

Hi all,

I wonder how one should manage the retention period of the data written into Quickwit. Assume that we are focusing on PCI DSS compliance, and we must retain logs "at least 12 months, with at least the most recent three months immediately available for analysis.". We can call it hot and cold state, then removal. Should we solve it by the application using it? Or is there a best practice around it?

Should we export data from Quickwit to another instance or to text files for cold storage/archiving?

One more question. If we use immutable splits, how can we delete data for retention?

[trinity-1686a]

Hi,

Quickwit doesn't have a concept of hot and cold storage. The storage used by default by Quickwit is already what many software would consider to be cold storage. In this situation, I would likely use Quickwit's retention, and depending on how often the "cold" data is queried, possibly move it automatically to lower cost storage tiers such as AWS standard-ia and glacier-ir or GCS coldline.

Should we export data from Quickwit to another instance or to text files for cold storage/archiving?

Exporting data from one Quickwit to another is unlikely to give you any gain. You could export everything to text files after a while, which you can compress and store elsewhere. This is leaving the realm of what Quickwit can do for you, but assuming queries are unlikely to ever happen, this would be the most compact storage you can do. How to query such data would be up to you though.

One more question. If we use immutable splits, how can we delete data for retention?

a split gets deleted when all of the documents it contains are ready to be deleted. When you specify a retention of 12 month, some documents will be kept for 12 month plus a few hours/days while other documents in the same splits aren't ready yet.
For the sake of completeness, it's also possible to delete specific documents, though this is a fairly expensive operation as we need to create a new split containing all but the data that needs to be deleted. This is meant for compliance with GDPR and alike, not for retention.