Remove expired snapshots #29

in0rdr · 2024-09-09T07:01:54Z

In AWS S3 you can use lifecycle rules to remove expired objects.

However, I was wondering how this works in other S3 compatible storage servers, where the lifecycle rules are not implemented, such as Exoscale (see limitations) or cloudscale?

I think there needs to be some process that regularly iterates all the objects and decides which ones to prune, no?

Let's discuss..

Bump vault version to 1.16.3, the latest community release: https://github.com/hashicorp/vault/blob/main/CHANGELOG.md#1163

This adds the variable S3_EXPIRE_DAYS. The idea of this feature is to allow the script to prune expired snapshot files on the S3 compatible remote storage. Files are considered expired once they exceed the threshold defined by S3_EXPIRE_DAYS. This feature is usefull for S3 compatible storage where there exist no lifecycle rules to clean up the storage of expired or old files, such as: * cloudscale object storage * Exoscale simple object storage (SOS) It is recommended to also configure a "Governance" lock on the files, to ensure no files are deleted by accident before the defined S3_EXPIRE_DAYS threshold.

kubernetes/Dockerfile

kubernetes/vault-snapshot.sh

kubernetes/README.md

kubernetes/vault-snapshot.sh

in0rdr · 2024-09-12T14:41:37Z

date: invalid date '1 days ago'
sh: out of range

in0rdr · 2024-09-12T14:51:05Z

also tried with -1 day format, same error in my test env

in0rdr · 2024-09-12T15:15:21Z

Seems like this busybox date thingy has some different logic. This works, for example, extract 1 day:

date --date @$((`date +%s` - 86400*1)) +%s

The date manipulation did not work in my tests with busybox on OpenShift. This should work even in the busybox environments. It simply subtracts seconds.

in0rdr · 2024-09-12T15:31:27Z

Ok, new logic seems to work fine. For example, it deletes all snapshots except for the ones from today with S3_EXPIRE_DAYS=1:

$ oc logs test-vw276 
upload: '/vault-snapshots/vault_2024-09-12-1526.snapshot' -> 's3://vault-snapshots/vault_2024-09-12-1526.snapshot' (23246 bytes in 0.0 seconds, 1484.17 KB/s) [1 of 1]
delete: 's3://vault-snapshots/s3://vault-snapshots/vault_2024-09-09-1348.snapshot'
delete: 's3://vault-snapshots/s3://vault-snapshots/vault_2024-09-09-1354.snapshot'
...
delete: 's3://vault-snapshots/s3://vault-snapshots/vault_2024-09-11-0827.snapshot'

However, when i do mc ls, I can see that none of the snapshot are actually deleted 😆

So, I think I'm back to my issue with "what API commands s3cmd is actually doing"..

in0rdr · 2024-09-12T15:46:32Z

Had a bug in my script. Was evident from the output ('s3://vault-snapshots/s3://vault-snapshots/vault_2024-09-09-1354.snapshot'). Now that this is fixed, it actually deletes the files 🎉

in0rdr · 2024-09-12T15:59:30Z

@tongpu the scripting is on the next level now. It was actually tested on an OpenShift cluster. I still have some confusion regarding what a Governance lock does exactly (I could not notice any difference in the deletion behavior, the files are "archived" not deleted), but we can also take this discussion offline. Thank you for another round of review 🙏

kubernetes/vault-snapshot.sh

in0rdr · 2024-09-12T22:07:10Z

I hope the auto-tagging workflow does not explode 🤞

in0rdr added 2 commits September 7, 2024 16:58

feat: bump vault version to 1.16.3

7c2a348

Bump vault version to 1.16.3, the latest community release: https://github.com/hashicorp/vault/blob/main/CHANGELOG.md#1163

in0rdr requested a review from eyenx September 9, 2024 07:01