Container based implementation of backup. Features:
- rsync to destination system
- can backup IMAP accounts
- Plugin system allows adding other types of data
- History Mode preserves deleted files or old versions optimizing space by using hard links
Destination can be anything that is reachable via rsync/SSH or a local disk mounted in the container. Copying via internet is supported.
History mode safes the backup to a subdirectory YYYY/mm/dd based on the backup date (today, unless --histdate is used). In order to save space, starting from the second backup we are hard-linking to unchanged filed of previous backup. With --histkeep also unsuccessful backups are reused in order to avoid double-downloading.
backup supports currently the following sources:
- IMAP
- File
- rclone
Start a container and pass the parameters (see below) for one action to this container.
This way of usage is recommended for testing and small number data storages.
- Create your own custom script
- This script can contain multiple backup actions. see Example
- mount this script e.g. on /backup into container
- give ". " as parameter to the container
This way of usage is recommended if you have a number of backup actions and other complex setups.
- "backup" (fix)
- Type (currently only "imap", "file" or "rclone2file" supported)
- Source
- Destination
- global options
- '--'
- local options (specific to "type", see relevant section)
see also Example.
- --runonsrc Prefer to run on the source side
- --runondst Prefer to run on destination side
- --local Run locally (=in the container) only
- --srcsecret Secret to access source system (password/ssh-key, depending on type)
- --dstsecret Secret to access destination system (password/ssh-key, depending on type)
- --hist Backup in History Mode (see separate section)
- --histdate Backup in History Mode, using instead of today. Not intended for production use, but mainly for testing
- --histraw <YYYY/mm/dd> Backup in History Mode, using fixed subdirectory as target. Only intended for internal use
- --histkeep Reuse backup data of unsuccessful backups. This is especially useful for rclone backups from cloud providers like Microsoft OneDrive.
- DEBUG: 0 or 1, defaults to 0
- MAIL_TO: EMail recipients Recipient(s) of Alerting emails. If set, it also activates the alerting feature.
- MAIL_FROM: Email Sender sender of the Alerting emails
- MAIL_URL: Email Server where to deliver the Alerting emails like 'smtp[s]://user:[email protected]:port'
- MAIL_HOSTNAME: hostname reported in SMTP HELO optionally give the hostname that we report to SMTP server in HELO command. Usefule if the SMTP server requires a FQDN. Defaults to hostname.
Backups data from an IMAP Server using offlineimap. This is done in a delta approach, so only modified Emails are downloaded whereas deleted EMails are deleted on the file directory
- Source - Logon user used to connect to the IMAP Server. Typically it contains the Email including the domain.
- Destination - Directory where to store the downloaded EMails. Optionally prefixed by Server name followed by a colon.
- Source Secret is the password to logon to the server [mandatory]
- Destination Secret is the password to connect to the target server [optional]
- IMAP Server URL (e.g. imap.mydomain.xxx:143) [mandatory]
Backups data from any server reachable via rsync and/or SSH. It is based on rsync, so only modified files are copied (delta-approach). No more existing files are deleted in the target.
- Source - Directoy of Files to be backed up. Optionally prefixed by Server name followed by a colon
- Destination - Directory where to store the downloaded EMails. Optionally prefixed by Server name followed by a colon.
- Source Secret is the password to logon to the server [optional] mandatory if the Source is remote
- Destination Secret is the password to connect to the target server [optional] mandatory if the Destination is remote
Backups data from various Clouds, leveraging rclone.
- Source - Name of the cloud in rclone config
- Destination - Directory where to store the download. Optionally prefixed by Server name followed by a colon.
- Source Secret - rclone.conf file to be used
Backups data from local to various Clouds, leveraging rclone.
- Source - Name of the cloud in rclone config
- Destination - Directory where to store the download. Optionally prefixed by Server name followed by a colon.
- Source Secret - rclone.conf file to be used
Each remote system needs a Secret for Authentication and authorisation. Typical form of Secrets are public keys or passwords. If needed, Secrets can be defined with the --srcsecret or --dstsecret parameter. Starting from version 0.2.1 no Defaults are applied.
Rebasing a history backup in order to save space can be done. You need to start rebasing on the first backup that introduced the files to be based on other data and then rebase each of the existing backups up to the latest. a sample command could be:
rsync -v -aHX --delete --progress --link-dest=../../../../../user_home/2021/11/20 --link-dest=../20 20/ 20.rsync-rebase.tmp
Features planned in the future
- can backup MySQL databases
- can backup Samba configuration
- Find a way to improve sync to cloud when files / directories have been moved. It would be fantastic if we can reuse files that are already present in the backup. see https://lincolnloop.com/blog/detecting-file-moves-renames-rsync/
If you want to develop and test on macOS, you need:
- brew install offlineimap jq bash rsync rclone
- activated local SSH server
- trust your own SSH key (e.g. ~/.ssh/id_rsa.pub must be in authorized_keys)
- tested with docker desktop for Mac
To run tests in test subdirectory you need to have some test data available. ATTENTION: Test data will be DELETED on every run, so make sure you don´t have any data on it!
-
IMAP Test Account
Setup an IMAP Test Account and set Environment variables $MAIL_ADR, $MAIL_SRV and $MAIL_PW accordingly
-
rclone Test Cloud
Setup a Test-Cloud (I use Microsoft OneDrive that is for free). Create an rclone.cfg and pass its filename in $RCLONE_CONF and the Cloud config to be used in $RCLONE_NAME. You may wan to use something like
rclone --config ./rclonf.conf configto create the file. -
alert test settings
Set variables MAIL_ADR MAIL_PW MAIL_SRV MAIL_URL, see test/test for details. Tests will use this vars to write alert logs to and check if they are there. IMAP Test Account can be reused. be aware that ALL Emails in this account will be deleted on every test run.
start with running test/test to verify your environment
GitHub Actions are setup in this repository to test against a dummy OneDrive
(<[email protected]>) and Mailbox (`[email protected]`).
Dependabot triggered pull requests (e.g. to update actions or Git subrepos)
automatically trigger a GitHub Action. However, dependatbot triggered actions
cannot access the "normal" action secrets in GitHub. In order to overcome this
limitations, relevant secrets have been copied into the special dependabot
area of secrets:
- GH_SECRETS_ADMIN_TOKEN hold the value of the token named
dependabot_ghaction_nafet227backupThis is needed to let the action update RCLONE config after re-authentication. - MAIL_PW holds the mail password of the dummy accounts
[email protected]. This account has limited send-rate to reduce the risk of being used as spam sender in case anyone steals the secret. - RCLONE_CONF holds the rclone config of the dummy OneDrive
[email protected]. It is updated after every run, since authentication needs to be re-evaluated according to oauth technology.
Copyright (C) 2017-2021 Stefan Schallenberg License: GPLv3, see LICENSE
Leveraging on