0.0.8 release

CHANGES.md

@@ -1,8 +1,21 @@
 # Changes
 
+## 0.0.8 (2022-01-21)
+
+- FEATURE: `zfs-auto-snapshot` can be told to ignore backup datasets on the target side, see #3.
+- FEATURE: `samba` can optionally be told to NOT share/expose backup datasets on the target side, see #4.
+- FEATURE: `ssh`-port on source and target becomes configurable, see #22.
+- FEATURE: New configuration fields for `source` and `target` each: `processing`. They can carry shell commands for pre- and post-processing of data before and after it is transferred via ssh. This enables the use of e.g. `lzma` or `bzip2` as a custom transfer compression beyond the compression capabilities of `ssh` itself. See #23.
+- FEATURE: `abgleich clean` can also remove snapshots on `target` but only if they are not part of the current overlap with `source`. The behavior can be controlled via the new `keep_backlog` configuration option, see #24 and #25.
+- FEATURE: Configuration module contains default values for parameters, making it much easier to write lightweight configuration files, see #28. The configuration parser now also provides much more useful output.
+- FEATURE: `abgleich tree` and `abgleich compare` highlight ignored datasets.
+- FEATURE: Significantly more flexible shell command wrapper and, as a result, cleaned up transaction handling.
+- FEATURE: Python 3.9 and 3.10 compatibility.
+- FIX: Many cleanups in code base, enabling future developments.
+
 ## 0.0.7 (2020-08-05)
 
-- FIX: `tree` now property checks if source or target is up, depending on what a user wants to see, see #20.
+- FIX: `tree` now properly checks if source or target is up, depending on what a user wants to see, see #20.
 - FIX: All `abgleich` commands can properly initialize (instead of crashing) if the target tree is empty, see #19.
 - FIX: `tree` shows message if there is no tree instead of crashing, see #18.
 

README.md

@@ -2,7 +2,7 @@
 
 ## SYNOPSIS
 
-`abgleich` is a simple ZFS sync tool. It displays source and target ZFS zpool, dataset and snapshot trees. It creates meaningful snapshots only if datasets have actually been changed. It compares a source zpool tree to a target, backup zpool tree. It pushes backups from a source to a target. It cleanes up older snapshots on the source side if they are present on the target side. It runs on a command line and produces nice, user-friendly, human-readable, colorized output. It also includes a GUI.
+[`abgleich`](https://dict.leo.org/englisch-deutsch/abgleich?side=right) is a simple ZFS sync tool. It displays source and target ZFS zpool, dataset and snapshot trees. It creates meaningful snapshots only if datasets have actually been changed. It compares a source zpool tree to a target, backup zpool tree. It pushes backups from a source to a target. It cleanes up older snapshots on the source side if they are present on the target side. It runs on a command line and produces nice, user-friendly, human-readable, colorized output. It also includes a GUI.
 
 ## CLI EXAMPLE
 
@@ -59,13 +59,16 @@ source:
     prefix:
     host: localhost
     user:
+    port:
 target:
     zpool: tank_hdd
     prefix: BACKUP_SOMEMACHINE
     host: bigdata
     user: zfsadmin
+    port:
 include_root: yes
 keep_snapshots: 2
+keep_backlog: True
 always_changed: no
 written_threshold: 1048576
 check_diff: yes
@@ -77,9 +80,37 @@ ignore:
 ssh:
     compression: no
     cipher: [email protected]
+compatibility:
+    target_samba_noshare: yes
+    target_autosnapshot_ignore: yes
 ```
 
-The prefix can be empty on either side. If a `host` is set to `localhost`, the `user` field can be left empty. Both source and target can be remote hosts or localhost at the same time. `include_root` indicates whether `{zpool}{/{prefix}}` should be  included in all operations. `keep_snapshots` is an integer and must be greater or equal to `1`. It specifies the number of snapshots that are kept per dataset on the source side when a cleanup operation is triggered. `suffix` contains the name suffix for new snapshots. Setting `always_changed` to `yes` causes `abgleich` to beliefe that all datasets have always changed since the last snapshot, completely ignoring what ZFS actually reports. No diff will be produced & checked for values of `written` lower than `written_threshold`. Checking diffs can be completely deactivated by setting `check_diff` to `no`. `digits` specifies how many digits are used for a decimal number describing the n-th snapshot per dataset per day as part of the name of new snapshots. `ignore` lists stuff underneath the `prefix` which will be ignored by this tool, i.e. no snapshots, backups or cleanups. `ssh` allows to fine-tune the speed of backups. In fast local networks, it is best to set `compression` to `no` because the compression is usually slowing down the transfer. However, for low-bandwidth transmissions, it makes sense to set it to `yes`. For significantly better speed in fast local networks, make sure that both the source and the target system support a common cipher, which is accelerated by [AES-NI](https://en.wikipedia.org/wiki/AES_instruction_set) on both ends.
+`zpool` defines the name of the zpools on source and target sides. The `prefix` value defines a "path" to a dataset underneath the `zpool`, so the name of the zpool itself is not part of the `prefix`. The `prefix` can be empty on either side. Prefixes can differ between source and target side. `host` specifies a value used by `ssh`. It does not have to be an actual host name. It can also be an alias from ssh's configuration. If a `host` is set to `localhost`, `ssh` wont be used and the `user` field can be left empty or omitted. Both source and target can be remote hosts or `localhost` at the same time. The `port` parameter specifies a custom `ssh` port. It can be left empty or omitted. `ssh` will then use its defaults or configuration to determine the correct port.
+
+`include_root` indicates whether `{zpool}{/{prefix}}` should be included in all operations. `keep_snapshots` is an integer and must be greater or equal to `1`. It specifies the number of snapshots that are kept per dataset on the source side when a cleanup operation is triggered. `keep_backlog` is either an integer or a boolean. It specifies if (or how many) snapshots are kept on the target side if the target side is cleaned. Snapshots that are part of the overlap with the source side are never considered for removal. `suffix` contains the name suffix for new snapshots.
+
+Whether or not snapshots are generated is based on the following sequence of checks:
+
+- Dataset is ignored: NO
+- Dataset has no snapshot: YES
+- If the `always_changed` configuration option is set to `yes`: YES
+- If the `tagging` configuration option underneath `compatibility` is set to yes and the last snapshot of the dataset has not been tagged by `abgleich` as a backup: YES
+- `written` property of dataset equals `0`: NO
+- Dataset is a volume: YES
+- If the `written_threshold` configuration is set and the `written` property of dataset is larger than the value of `written_threshold`: YES
+- If the `check_diff` configuration option is set to `no`: YES
+- If `zfs diff` produces any output relative to the last snapshot: YES
+- Otherwise: NO
+
+Setting `always_changed` to `yes` causes `abgleich` to beliefe that all datasets have always changed since the last snapshot, completely ignoring what ZFS actually reports. No diff will be produced & checked for values of `written` lower than `written_threshold`. Checking diffs can be completely deactivated by setting `check_diff` to `no`.
+
+`digits` specifies how many digits are used for a decimal number describing the n-th snapshot per dataset per day as part of the name of new snapshots. `ignore` lists stuff underneath the `prefix` which will be ignored by this tool, i.e. no snapshots, backups or cleanups.
+
+`ssh` allows to fine-tune the speed of backups. In fast local networks, it is best to set `compression` to `no` because the compression is usually slowing down the transfer. However, for low-bandwidth transmissions, it makes sense to set it to `yes`. For significantly better speed in fast local networks, make sure that both the source and the target system support a common cipher, which is accelerated by [AES-NI](https://en.wikipedia.org/wiki/AES_instruction_set) on both ends. The `ssh` port can be specified per side via the `port` configuration option, i.e. for source and/or target.
+
+Custom pre- and post-processing can be applied after `send` and before `receive` per side via shell commands specified in the `processing` configuration option (underneath `source` and `target`). This can be useful for a custom transfer compression based on e.g. `lzma` or `bzip2`.
+
+`compatibility` adds options for making `abgleich` more compatible with other tools. If `target_samba_noshare` is active, the `sharesmb` property will - as part of backup operations - be set to `off` for `{zpool}{/{prefix}}` on the target side, preventing sharing/exposing backup datasets by accident. If `target_autosnapshot_ignore` is active, the `com.sun:auto-snapshot` property will - similarly as part of backup operations - be set to `false` for `{zpool}{/{prefix}}` on the target side, telling `zfs-auto-snapshot` to ignore the dataset.
 
 ## USAGE
 
@@ -101,9 +132,9 @@ Compare source ZFS tree with target ZFS tree. See what is missing where.
 
 Send (new) datasets and new snapshots from source to target.
 
-### `abgleich cleanup config.yaml`
+### `abgleich cleanup config.yaml [source|target]`
 
-Cleanup older local snapshots on source side if they are present on both sides. Of those snapshots present on both sides, keep at least `keep_snapshots` number of snapshots on source side.
+Cleanup older local snapshots on source side if they are present on both sides. Of those snapshots present on both sides, keep at least `keep_snapshots` number of snapshots on source side. Or: Cleanup older snapshots on target side. Beyond the overlap with source, keep at least `keep_backlog` snapshots. If `keep_backlog` is `False`, all snapshots older than the overlap will be removed. If `keep_backlog` is `True`, no snapshots will be removed. If `abgleich clean` runs against the target side, an extra warning will be displayed and must be confirmed by the user before any dangerous actions are attempted.
 
 ### `abgleich wizard config.yaml`
 

setup.py

@@ -8,7 +8,7 @@
 
     setup.py: Used for package distribution
 
-    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License
@@ -42,7 +42,7 @@
 
 # List all versions of Python which are supported
 python_minor_min = 6
-python_minor_max = 8
+python_minor_max = 10
 confirmed_python_versions = [
     "Programming Language :: Python :: 3.{MINOR:d}".format(MINOR=minor)
     for minor in range(python_minor_min, python_minor_max + 1)
@@ -73,8 +73,16 @@ def get_version(code):
 
 # Requirements
 extras_require = {
-    "dev": ["black", "python-language-server[all]", "setuptools", "twine", "wheel",],
-    "gui": ["pyqt5",],
+    "dev": [
+        "black",
+        "python-language-server[all]",
+        "setuptools",
+        "twine",
+        "wheel",
+    ],
+    "gui": [
+        "pyqt5",
+    ],
 }
 extras_require["all"] = list(
     {rq for target in extras_require.keys() for rq in extras_require[target]}
@@ -95,15 +103,27 @@ def get_version(code):
     download_url="https://github.com/pleiszenburg/abgleich/archive/v%s.tar.gz"
     % __version__,
     license="LGPLv2",
-    keywords=["zfs", "ssh",],
+    keywords=[
+        "zfs",
+        "ssh",
+    ],
     scripts=[],
     include_package_data=True,
     python_requires=">=3.{MINOR:d}".format(MINOR=python_minor_min),
     setup_requires=[],
-    install_requires=["click", "tabulate", "pyyaml", "typeguard",],
+    install_requires=[
+        "click",
+        "tabulate",
+        "pyyaml",
+        "typeguard",
+    ],
     extras_require=extras_require,
     zip_safe=False,
-    entry_points={"console_scripts": ["abgleich = abgleich.cli:cli",],},
+    entry_points={
+        "console_scripts": [
+            "abgleich = abgleich.cli:cli",
+        ],
+    },
     classifiers=[
         "Development Status :: 5 - Production/Stable",
         "Environment :: Console",

src/abgleich/__init__.py

@@ -8,7 +8,7 @@
 
 	src/abgleich/__init__.py: Package root
 
-	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+	Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License
@@ -28,4 +28,4 @@
 # META
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
-__version__ = "0.0.7"
+__version__ = "0.0.8"

src/abgleich/cli/__init__.py

@@ -8,7 +8,7 @@
 
 	src/abgleich/cli/__init__.py: CLI package root
 
-	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+	Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License
@@ -26,7 +26,7 @@
 
 
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-# IMPORT
+# EXPORT
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 from ._main_ import cli

src/abgleich/cli/_main_.py

@@ -8,7 +8,7 @@
 
     src/abgleich/cli/_main_.py: CLI auto-detection
 
-    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License

src/abgleich/cli/backup.py

@@ -8,7 +8,7 @@
 
     src/abgleich/cli/backup.py: backup command entry point
 
-    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License

src/abgleich/cli/cleanup.py

@@ -8,7 +8,7 @@
 
     src/abgleich/cli/cleanup.py: cleanup command entry point
 
-    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License
@@ -47,20 +47,37 @@
 
 @click.command(short_help="cleanup older snapshots")
 @click.argument("configfile", type=click.File("r", encoding="utf-8"))
-def cleanup(configfile):
+@click.argument("side", default="source", type=str)
+def cleanup(configfile, side):
 
     config = Config.from_fd(configfile)
 
-    for side in ("source", "target"):
+    assert side in ("source", "target")
+
+    cleanup_side = side
+    control_side = "target" if cleanup_side == "source" else "source"
+
+    if cleanup_side == "target":
+        click.confirm(
+            t(
+                "DANGER ZONE: You are about to clean the TARGET. Do you want to continue?"
+            ),
+            abort=True,
+        )
+        if config["keep_backlog"].value == True:
+            print(t("nothing to do"))
+            return
+
+    for side in (cleanup_side, control_side):
         if not is_host_up(side, config):
             print(f'{t("host is not up"):s}: {side:s}')
             sys.exit(1)
 
-    source_zpool = Zpool.from_config("source", config=config)
-    target_zpool = Zpool.from_config("target", config=config)
-    available_before = Zpool.available("source", config=config)
+    cleanup_zpool = Zpool.from_config(cleanup_side, config=config)
+    control_zpool = Zpool.from_config(control_side, config=config)
+    available_before = Zpool.available(cleanup_side, config=config)
 
-    transactions = source_zpool.get_cleanup_transactions(target_zpool)
+    transactions = cleanup_zpool.get_cleanup_transactions(control_zpool)
 
     if len(transactions) == 0:
         print(t("nothing to do"))
@@ -74,7 +91,10 @@ def cleanup(configfile):
     WAIT = 10
     print(f"waiting {WAIT:d} seconds ...")
     time.sleep(WAIT)
-    available_after = Zpool.available("source", config=config)
+    available_after = Zpool.available(cleanup_side, config=config)
     print(
-        f"{humanize_size(available_after, add_color = True):s} available, {humanize_size(available_after - available_before, add_color = True):s} freed"
+        (
+            f"{humanize_size(available_after, add_color = True):s} available, "
+            f"{humanize_size(available_after - available_before, add_color = True):s} freed"
+        )
     )

src/abgleich/cli/compare.py

@@ -8,7 +8,7 @@
 
     src/abgleich/cli/compare.py: compare command entry point
 
-    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License

src/abgleich/cli/snap.py

@@ -8,7 +8,7 @@
 
     src/abgleich/cli/snap.py: snap command entry point
 
-    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License

src/abgleich/cli/tree.py

@@ -8,7 +8,7 @@
 
     src/abgleich/cli/tree.py: tree command entry point
 
-    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License

src/abgleich/cli/wizard.py

@@ -8,7 +8,7 @@
 
     src/abgleich/cli/wizard.py: wizard command entry point
 
-    Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+    Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License

src/abgleich/core/__init__.py

@@ -8,7 +8,7 @@
 
 	src/abgleich/core/__init__.py: Core package root
 
-	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
+	Copyright (C) 2019-2022 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License