diff --git a/CHANGES.md b/CHANGES.md index 1ca06e6..c80e866 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,8 +1,21 @@ # Changes +## 0.0.8 (2022-01-21) + +- FEATURE: `zfs-auto-snapshot` can be told to ignore backup datasets on the target side, see #3. +- FEATURE: `samba` can optionally be told to NOT share/expose backup datasets on the target side, see #4. +- FEATURE: `ssh`-port on source and target becomes configurable, see #22. +- FEATURE: New configuration fields for `source` and `target` each: `processing`. They can carry shell commands for pre- and post-processing of data before and after it is transferred via ssh. This enables the use of e.g. `lzma` or `bzip2` as a custom transfer compression beyond the compression capabilities of `ssh` itself. See #23. +- FEATURE: `abgleich clean` can also remove snapshots on `target` but only if they are not part of the current overlap with `source`. The behavior can be controlled via the new `keep_backlog` configuration option, see #24 and #25. +- FEATURE: Configuration module contains default values for parameters, making it much easier to write lightweight configuration files, see #28. The configuration parser now also provides much more useful output. +- FEATURE: `abgleich tree` and `abgleich compare` highlight ignored datasets. +- FEATURE: Significantly more flexible shell command wrapper and, as a result, cleaned up transaction handling. +- FEATURE: Python 3.9 and 3.10 compatibility. +- FIX: Many cleanups in code base, enabling future developments. + ## 0.0.7 (2020-08-05) -- FIX: `tree` now property checks if source or target is up, depending on what a user wants to see, see #20. +- FIX: `tree` now properly checks if source or target is up, depending on what a user wants to see, see #20. - FIX: All `abgleich` commands can properly initialize (instead of crashing) if the target tree is empty, see #19. - FIX: `tree` shows message if there is no tree instead of crashing, see #18. diff --git a/README.md b/README.md index c8f1a1c..fbcf468 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ## SYNOPSIS -`abgleich` is a simple ZFS sync tool. It displays source and target ZFS zpool, dataset and snapshot trees. It creates meaningful snapshots only if datasets have actually been changed. It compares a source zpool tree to a target, backup zpool tree. It pushes backups from a source to a target. It cleanes up older snapshots on the source side if they are present on the target side. It runs on a command line and produces nice, user-friendly, human-readable, colorized output. It also includes a GUI. +[`abgleich`](https://dict.leo.org/englisch-deutsch/abgleich?side=right) is a simple ZFS sync tool. It displays source and target ZFS zpool, dataset and snapshot trees. It creates meaningful snapshots only if datasets have actually been changed. It compares a source zpool tree to a target, backup zpool tree. It pushes backups from a source to a target. It cleanes up older snapshots on the source side if they are present on the target side. It runs on a command line and produces nice, user-friendly, human-readable, colorized output. It also includes a GUI. ## CLI EXAMPLE @@ -59,13 +59,16 @@ source: prefix: host: localhost user: + port: target: zpool: tank_hdd prefix: BACKUP_SOMEMACHINE host: bigdata user: zfsadmin + port: include_root: yes keep_snapshots: 2 +keep_backlog: True always_changed: no written_threshold: 1048576 check_diff: yes @@ -77,9 +80,37 @@ ignore: ssh: compression: no cipher: aes256-gcm@openssh.com +compatibility: + target_samba_noshare: yes + target_autosnapshot_ignore: yes ``` -The prefix can be empty on either side. If a `host` is set to `localhost`, the `user` field can be left empty. Both source and target can be remote hosts or localhost at the same time. `include_root` indicates whether `{zpool}{/{prefix}}` should be included in all operations. `keep_snapshots` is an integer and must be greater or equal to `1`. It specifies the number of snapshots that are kept per dataset on the source side when a cleanup operation is triggered. `suffix` contains the name suffix for new snapshots. Setting `always_changed` to `yes` causes `abgleich` to beliefe that all datasets have always changed since the last snapshot, completely ignoring what ZFS actually reports. No diff will be produced & checked for values of `written` lower than `written_threshold`. Checking diffs can be completely deactivated by setting `check_diff` to `no`. `digits` specifies how many digits are used for a decimal number describing the n-th snapshot per dataset per day as part of the name of new snapshots. `ignore` lists stuff underneath the `prefix` which will be ignored by this tool, i.e. no snapshots, backups or cleanups. `ssh` allows to fine-tune the speed of backups. In fast local networks, it is best to set `compression` to `no` because the compression is usually slowing down the transfer. However, for low-bandwidth transmissions, it makes sense to set it to `yes`. For significantly better speed in fast local networks, make sure that both the source and the target system support a common cipher, which is accelerated by [AES-NI](https://en.wikipedia.org/wiki/AES_instruction_set) on both ends. +`zpool` defines the name of the zpools on source and target sides. The `prefix` value defines a "path" to a dataset underneath the `zpool`, so the name of the zpool itself is not part of the `prefix`. The `prefix` can be empty on either side. Prefixes can differ between source and target side. `host` specifies a value used by `ssh`. It does not have to be an actual host name. It can also be an alias from ssh's configuration. If a `host` is set to `localhost`, `ssh` wont be used and the `user` field can be left empty or omitted. Both source and target can be remote hosts or `localhost` at the same time. The `port` parameter specifies a custom `ssh` port. It can be left empty or omitted. `ssh` will then use its defaults or configuration to determine the correct port. + +`include_root` indicates whether `{zpool}{/{prefix}}` should be included in all operations. `keep_snapshots` is an integer and must be greater or equal to `1`. It specifies the number of snapshots that are kept per dataset on the source side when a cleanup operation is triggered. `keep_backlog` is either an integer or a boolean. It specifies if (or how many) snapshots are kept on the target side if the target side is cleaned. Snapshots that are part of the overlap with the source side are never considered for removal. `suffix` contains the name suffix for new snapshots. + +Whether or not snapshots are generated is based on the following sequence of checks: + +- Dataset is ignored: NO +- Dataset has no snapshot: YES +- If the `always_changed` configuration option is set to `yes`: YES +- If the `tagging` configuration option underneath `compatibility` is set to yes and the last snapshot of the dataset has not been tagged by `abgleich` as a backup: YES +- `written` property of dataset equals `0`: NO +- Dataset is a volume: YES +- If the `written_threshold` configuration is set and the `written` property of dataset is larger than the value of `written_threshold`: YES +- If the `check_diff` configuration option is set to `no`: YES +- If `zfs diff` produces any output relative to the last snapshot: YES +- Otherwise: NO + +Setting `always_changed` to `yes` causes `abgleich` to beliefe that all datasets have always changed since the last snapshot, completely ignoring what ZFS actually reports. No diff will be produced & checked for values of `written` lower than `written_threshold`. Checking diffs can be completely deactivated by setting `check_diff` to `no`. + +`digits` specifies how many digits are used for a decimal number describing the n-th snapshot per dataset per day as part of the name of new snapshots. `ignore` lists stuff underneath the `prefix` which will be ignored by this tool, i.e. no snapshots, backups or cleanups. + +`ssh` allows to fine-tune the speed of backups. In fast local networks, it is best to set `compression` to `no` because the compression is usually slowing down the transfer. However, for low-bandwidth transmissions, it makes sense to set it to `yes`. For significantly better speed in fast local networks, make sure that both the source and the target system support a common cipher, which is accelerated by [AES-NI](https://en.wikipedia.org/wiki/AES_instruction_set) on both ends. The `ssh` port can be specified per side via the `port` configuration option, i.e. for source and/or target. + +Custom pre- and post-processing can be applied after `send` and before `receive` per side via shell commands specified in the `processing` configuration option (underneath `source` and `target`). This can be useful for a custom transfer compression based on e.g. `lzma` or `bzip2`. + +`compatibility` adds options for making `abgleich` more compatible with other tools. If `target_samba_noshare` is active, the `sharesmb` property will - as part of backup operations - be set to `off` for `{zpool}{/{prefix}}` on the target side, preventing sharing/exposing backup datasets by accident. If `target_autosnapshot_ignore` is active, the `com.sun:auto-snapshot` property will - similarly as part of backup operations - be set to `false` for `{zpool}{/{prefix}}` on the target side, telling `zfs-auto-snapshot` to ignore the dataset. ## USAGE @@ -101,9 +132,9 @@ Compare source ZFS tree with target ZFS tree. See what is missing where. Send (new) datasets and new snapshots from source to target. -### `abgleich cleanup config.yaml` +### `abgleich cleanup config.yaml [source|target]` -Cleanup older local snapshots on source side if they are present on both sides. Of those snapshots present on both sides, keep at least `keep_snapshots` number of snapshots on source side. +Cleanup older local snapshots on source side if they are present on both sides. Of those snapshots present on both sides, keep at least `keep_snapshots` number of snapshots on source side. Or: Cleanup older snapshots on target side. Beyond the overlap with source, keep at least `keep_backlog` snapshots. If `keep_backlog` is `False`, all snapshots older than the overlap will be removed. If `keep_backlog` is `True`, no snapshots will be removed. If `abgleich clean` runs against the target side, an extra warning will be displayed and must be confirmed by the user before any dangerous actions are attempted. ### `abgleich wizard config.yaml` diff --git a/setup.py b/setup.py index c56a060..013557c 100644 --- a/setup.py +++ b/setup.py @@ -8,7 +8,7 @@ setup.py: Used for package distribution - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -42,7 +42,7 @@ # List all versions of Python which are supported python_minor_min = 6 -python_minor_max = 8 +python_minor_max = 10 confirmed_python_versions = [ "Programming Language :: Python :: 3.{MINOR:d}".format(MINOR=minor) for minor in range(python_minor_min, python_minor_max + 1) @@ -73,8 +73,16 @@ def get_version(code): # Requirements extras_require = { - "dev": ["black", "python-language-server[all]", "setuptools", "twine", "wheel",], - "gui": ["pyqt5",], + "dev": [ + "black", + "python-language-server[all]", + "setuptools", + "twine", + "wheel", + ], + "gui": [ + "pyqt5", + ], } extras_require["all"] = list( {rq for target in extras_require.keys() for rq in extras_require[target]} @@ -95,15 +103,27 @@ def get_version(code): download_url="https://github.com/pleiszenburg/abgleich/archive/v%s.tar.gz" % __version__, license="LGPLv2", - keywords=["zfs", "ssh",], + keywords=[ + "zfs", + "ssh", + ], scripts=[], include_package_data=True, python_requires=">=3.{MINOR:d}".format(MINOR=python_minor_min), setup_requires=[], - install_requires=["click", "tabulate", "pyyaml", "typeguard",], + install_requires=[ + "click", + "tabulate", + "pyyaml", + "typeguard", + ], extras_require=extras_require, zip_safe=False, - entry_points={"console_scripts": ["abgleich = abgleich.cli:cli",],}, + entry_points={ + "console_scripts": [ + "abgleich = abgleich.cli:cli", + ], + }, classifiers=[ "Development Status :: 5 - Production/Stable", "Environment :: Console", diff --git a/src/abgleich/__init__.py b/src/abgleich/__init__.py index adae71e..2e3442d 100644 --- a/src/abgleich/__init__.py +++ b/src/abgleich/__init__.py @@ -8,7 +8,7 @@ src/abgleich/__init__.py: Package root - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,4 +28,4 @@ # META # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -__version__ = "0.0.7" +__version__ = "0.0.8" diff --git a/src/abgleich/cli/__init__.py b/src/abgleich/cli/__init__.py index f96b87e..148d5b8 100644 --- a/src/abgleich/cli/__init__.py +++ b/src/abgleich/cli/__init__.py @@ -8,7 +8,7 @@ src/abgleich/cli/__init__.py: CLI package root - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -26,7 +26,7 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -# IMPORT +# EXPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ from ._main_ import cli diff --git a/src/abgleich/cli/_main_.py b/src/abgleich/cli/_main_.py index 4512286..a9aa004 100644 --- a/src/abgleich/cli/_main_.py +++ b/src/abgleich/cli/_main_.py @@ -8,7 +8,7 @@ src/abgleich/cli/_main_.py: CLI auto-detection - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/cli/backup.py b/src/abgleich/cli/backup.py index 33c3faf..d2c028a 100644 --- a/src/abgleich/cli/backup.py +++ b/src/abgleich/cli/backup.py @@ -8,7 +8,7 @@ src/abgleich/cli/backup.py: backup command entry point - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/cli/cleanup.py b/src/abgleich/cli/cleanup.py index 68803fd..dc97bdb 100644 --- a/src/abgleich/cli/cleanup.py +++ b/src/abgleich/cli/cleanup.py @@ -8,7 +8,7 @@ src/abgleich/cli/cleanup.py: cleanup command entry point - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -47,20 +47,37 @@ @click.command(short_help="cleanup older snapshots") @click.argument("configfile", type=click.File("r", encoding="utf-8")) -def cleanup(configfile): +@click.argument("side", default="source", type=str) +def cleanup(configfile, side): config = Config.from_fd(configfile) - for side in ("source", "target"): + assert side in ("source", "target") + + cleanup_side = side + control_side = "target" if cleanup_side == "source" else "source" + + if cleanup_side == "target": + click.confirm( + t( + "DANGER ZONE: You are about to clean the TARGET. Do you want to continue?" + ), + abort=True, + ) + if config["keep_backlog"].value == True: + print(t("nothing to do")) + return + + for side in (cleanup_side, control_side): if not is_host_up(side, config): print(f'{t("host is not up"):s}: {side:s}') sys.exit(1) - source_zpool = Zpool.from_config("source", config=config) - target_zpool = Zpool.from_config("target", config=config) - available_before = Zpool.available("source", config=config) + cleanup_zpool = Zpool.from_config(cleanup_side, config=config) + control_zpool = Zpool.from_config(control_side, config=config) + available_before = Zpool.available(cleanup_side, config=config) - transactions = source_zpool.get_cleanup_transactions(target_zpool) + transactions = cleanup_zpool.get_cleanup_transactions(control_zpool) if len(transactions) == 0: print(t("nothing to do")) @@ -74,7 +91,10 @@ def cleanup(configfile): WAIT = 10 print(f"waiting {WAIT:d} seconds ...") time.sleep(WAIT) - available_after = Zpool.available("source", config=config) + available_after = Zpool.available(cleanup_side, config=config) print( - f"{humanize_size(available_after, add_color = True):s} available, {humanize_size(available_after - available_before, add_color = True):s} freed" + ( + f"{humanize_size(available_after, add_color = True):s} available, " + f"{humanize_size(available_after - available_before, add_color = True):s} freed" + ) ) diff --git a/src/abgleich/cli/compare.py b/src/abgleich/cli/compare.py index d74588c..fb5b969 100644 --- a/src/abgleich/cli/compare.py +++ b/src/abgleich/cli/compare.py @@ -8,7 +8,7 @@ src/abgleich/cli/compare.py: compare command entry point - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/cli/snap.py b/src/abgleich/cli/snap.py index b228341..09b00f2 100644 --- a/src/abgleich/cli/snap.py +++ b/src/abgleich/cli/snap.py @@ -8,7 +8,7 @@ src/abgleich/cli/snap.py: snap command entry point - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/cli/tree.py b/src/abgleich/cli/tree.py index 7348b59..ea2d897 100644 --- a/src/abgleich/cli/tree.py +++ b/src/abgleich/cli/tree.py @@ -8,7 +8,7 @@ src/abgleich/cli/tree.py: tree command entry point - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/cli/wizard.py b/src/abgleich/cli/wizard.py index 616a344..c28884b 100644 --- a/src/abgleich/cli/wizard.py +++ b/src/abgleich/cli/wizard.py @@ -8,7 +8,7 @@ src/abgleich/cli/wizard.py: wizard command entry point - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/core/__init__.py b/src/abgleich/core/__init__.py index edf8425..b9189ed 100644 --- a/src/abgleich/core/__init__.py +++ b/src/abgleich/core/__init__.py @@ -8,7 +8,7 @@ src/abgleich/core/__init__.py: Core package root - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/core/abc.py b/src/abgleich/core/abc.py index 58a076a..ac298e5 100644 --- a/src/abgleich/core/abc.py +++ b/src/abgleich/core/abc.py @@ -8,7 +8,7 @@ src/abgleich/core/abc.py: Abstract base classes - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,56 +28,64 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import abc +from abc import ABC # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CLASSES # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -class CloneABC(abc.ABC): +class CloneABC(ABC): pass -class CommandABC(abc.ABC): +class CommandABC(ABC): pass -class ComparisonABC(abc.ABC): +class ComparisonDatasetABC(ABC): pass -class ComparisonItemABC(abc.ABC): +class ComparisonItemABC(ABC): pass -class ConfigABC(abc.ABC): +class ComparisonZpoolABC(ABC): pass -class DatasetABC(abc.ABC): +class ConfigABC(ABC): pass -class PropertyABC(abc.ABC): +class ConfigFieldABC(ABC): pass -class SnapshotABC(abc.ABC): +class DatasetABC(ABC): pass -class TransactionABC(abc.ABC): +class PropertyABC(ABC): pass -class TransactionListABC(abc.ABC): +class SnapshotABC(ABC): pass -class TransactionMetaABC(abc.ABC): +class TransactionABC(ABC): pass -class ZpoolABC(abc.ABC): +class TransactionListABC(ABC): + pass + + +class TransactionMetaABC(ABC): + pass + + +class ZpoolABC(ABC): pass diff --git a/src/abgleich/core/command.py b/src/abgleich/core/command.py index d3c3ab0..684e5bd 100644 --- a/src/abgleich/core/command.py +++ b/src/abgleich/core/command.py @@ -8,7 +8,7 @@ src/abgleich/core/command.py: Sub-process wrapper for commands - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,115 +28,145 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import subprocess -import typing +import itertools +from subprocess import Popen, PIPE +from typing import List, Tuple, Union +import shlex -import typeguard +from typeguard import typechecked -from .abc import CommandABC +from .abc import CommandABC, ConfigABC # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CLASS # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked class Command(CommandABC): - def __init__(self, cmd: typing.List[str]): + """ + Immutable. + """ - self._cmd = cmd.copy() + def __init__(self, cmd: List[List[str]]): + + self._cmd = [fragment.copy() for fragment in cmd] + + def __repr__(self) -> str: + + return "" def __str__(self) -> str: - return " ".join([item.replace(" ", "\\ ") for item in self._cmd]) + return " | ".join([shlex.join(fragment) for fragment in self._cmd]) + + def __len__(self) -> int: + + return len(self._cmd) + + def __or__(self, other: CommandABC) -> CommandABC: # pipe + + return type(self)(self.cmd + other.cmd) + + @staticmethod + def _com_to_str(com: Union[str, bytes, None]) -> str: + + if com is None: + return "" + + if isinstance(com, bytes): + return com.decode("utf-8") + + return com + + @staticmethod + def _split_list(data: List, delimiter: str) -> List[List]: + + return [ + list(sub_list) + for is_delimiter, sub_list in itertools.groupby( + data, lambda item: item == delimiter + ) + if not is_delimiter + ] def run( self, returncode: bool = False - ) -> typing.Union[typing.Tuple[str, str], typing.Tuple[str, str, int, Exception]]: + ) -> Union[ + Tuple[List[str], List[str], List[int], Exception], Tuple[List[str], List[str]] + ]: - proc = subprocess.Popen( - self.cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE - ) - output, errors = proc.communicate() - status = not bool(proc.returncode) - output, errors = output.decode("utf-8"), errors.decode("utf-8") + procs = [] # all processes, connected with pipes + + for index, fragment in enumerate(self._cmd): # create & connect processes + + stdin = None if index == 0 else procs[-1].stdout # output of last process + proc = Popen( + fragment, + stdout=PIPE, + stderr=PIPE, + stdin=stdin, + ) + procs.append(proc) + + output, errors, status = [], [], [] + + for proc in procs[::-1]: # inverse order, last process first + + out, err = proc.communicate() + output.append(self._com_to_str(out)) + errors.append(self._com_to_str(err)) + status.append(int(proc.returncode)) + + output.reverse() + errors.reverse() + status.reverse() exception = SystemError("command failed", str(self), output, errors) if returncode: - return output, errors, int(proc.returncode), exception + return output, errors, status, exception - if not status or len(errors.strip()) > 0: + if any((code != 0 for code in status)): # some fragment failed: raise exception return output, errors - def run_pipe(self, other: CommandABC): + def on_side(self, side: str, config: ConfigABC) -> CommandABC: - proc_1 = subprocess.Popen( - self.cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE - ) - proc_2 = subprocess.Popen( - other.cmd, - stdin=proc_1.stdout, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) + if config[f"{side:s}/host"].value == "localhost": + return self - output_2, errors_2 = proc_2.communicate() - status_2 = not bool(proc_2.returncode) - _, errors_1 = proc_1.communicate() - status_1 = not bool(proc_1.returncode) + side_config = config.group(side) + ssh_config = config.group("ssh") - errors_1 = errors_1.decode("utf-8") - output_2, errors_2 = output_2.decode("utf-8"), errors_2.decode("utf-8") - - if any( - ( - not status_1, - len(errors_1.strip()) > 0, - not status_2, - len(errors_2.strip()) > 0, - ) - ): - raise SystemError( - "command pipe failed", - f"{str(self):s} | {str(other):s}", - errors_1, - output_2, - errors_2, - ) + cmd_ssh = [ + "ssh", + "-T", # Disable pseudo-terminal allocation + "-o", # Option parameter + "Compression=yes" if ssh_config["compression"].value else "Compression=no", + ] + if side_config["port"].value != 0: + cmd_ssh.extend(["-p", f'{side_config["port"].value:d}']) + if ssh_config["cipher"].value is not None: + cmd_ssh.extend(("-c", ssh_config["cipher"].value)) + cmd_ssh.extend( + [f'{side_config["user"].value:s}@{side_config["host"].value:s}', str(self)] + ) - return errors_1, output_2, errors_2 + return type(self)([cmd_ssh]) @property - def cmd(self) -> typing.List[str]: + def cmd(self) -> List[List[str]]: - return self._cmd.copy() + return [fragment.copy() for fragment in self._cmd] @classmethod - def on_side( - cls, cmd: typing.List[str], side: str, config: typing.Dict - ) -> CommandABC: + def from_str(cls, cmd: str) -> CommandABC: - if config[side]["host"] == "localhost": - return cls(cmd) - return cls.with_ssh(cmd, side_config=config[side], ssh_config=config["ssh"]) + return cls(cls._split_list(shlex.split(cmd), "|")) @classmethod - def with_ssh( - cls, cmd: typing.List[str], side_config: typing.Dict, ssh_config: typing.Dict - ) -> CommandABC: - - cmd_str = " ".join([item.replace(" ", "\\ ") for item in cmd]) - cmd = [ - "ssh", - "-T", # Disable pseudo-terminal allocation - "-o", - "Compression=yes" if ssh_config["compression"] else "Compression=no", - ] - if ssh_config["cipher"] is not None: - cmd.extend(("-c", ssh_config["cipher"])) - cmd.extend([f'{side_config["user"]:s}@{side_config["host"]:s}', cmd_str]) + def from_list(cls, cmd: List[str]) -> CommandABC: - return cls(cmd) + return cls([cmd]) diff --git a/src/abgleich/core/comparison.py b/src/abgleich/core/comparisondataset.py similarity index 50% rename from src/abgleich/core/comparison.py rename to src/abgleich/core/comparisondataset.py index e9c08b2..06b642b 100644 --- a/src/abgleich/core/comparison.py +++ b/src/abgleich/core/comparisondataset.py @@ -6,9 +6,9 @@ zfs sync tool https://github.com/pleiszenburg/abgleich - src/abgleich/core/comparison.py: ZFS comparison + src/abgleich/core/comparisondataset.py: ZFS dataset comparison - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -29,41 +29,35 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ import itertools -import typing +from typing import Generator, List, Union -import typeguard +from typeguard import typechecked -from .abc import ComparisonABC, ComparisonItemABC, DatasetABC, SnapshotABC, ZpoolABC - -# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -# TYPING -# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -ComparisonParentTypes = typing.Union[ - ZpoolABC, DatasetABC, None, -] -ComparisonMergeTypes = typing.Union[ - typing.Generator[DatasetABC, None, None], typing.Generator[SnapshotABC, None, None], -] -ComparisonItemType = typing.Union[ - DatasetABC, SnapshotABC, None, -] -ComparisonStrictItemType = typing.Union[ - DatasetABC, SnapshotABC, -] +from .abc import ( + ComparisonDatasetABC, + ComparisonItemABC, + ConfigABC, + DatasetABC, + SnapshotABC, +) +from .comparisonitem import ComparisonItem, ComparisonItemType, ComparisonStrictItemType # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CLASS # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked -class Comparison(ComparisonABC): +@typechecked +class ComparisonDataset(ComparisonDatasetABC): + """ + Immutable. + """ + def __init__( self, - a: ComparisonParentTypes, - b: ComparisonParentTypes, - merged: typing.List[ComparisonItemABC], + a: Union[DatasetABC, None], + b: Union[DatasetABC, None], + merged: List[ComparisonItemABC], ): assert a is not None or b is not None @@ -77,20 +71,28 @@ def __len__(self) -> int: return len(self._merged) @property - def a(self) -> ComparisonParentTypes: + def a(self) -> Union[DatasetABC, None]: return self._a @property - def a_head(self) -> typing.List[ComparisonStrictItemType]: + def a_disjoint_head(self) -> List[ComparisonStrictItemType]: - return self._head( + return self._disjoint_head( source=[item.a for item in self._merged], target=[item.b for item in self._merged], ) @property - def a_overlap_tail(self) -> typing.List[ComparisonStrictItemType]: + def a_disjoint_tail(self) -> List[ComparisonStrictItemType]: + + return self._disjoint_head( + source=[item.a for item in self._merged][::-1], + target=[item.b for item in self._merged][::-1], + )[::-1] + + @property + def a_overlap_tail(self) -> List[ComparisonStrictItemType]: return self._overlap_tail( source=[item.a for item in self._merged], @@ -98,20 +100,28 @@ def a_overlap_tail(self) -> typing.List[ComparisonStrictItemType]: ) @property - def b(self) -> ComparisonParentTypes: + def b(self) -> Union[DatasetABC, None]: return self._b @property - def b_head(self) -> typing.List[ComparisonStrictItemType]: + def b_disjoint_head(self) -> List[ComparisonStrictItemType]: - return self._head( + return self._disjoint_head( source=[item.b for item in self._merged], target=[item.a for item in self._merged], ) @property - def b_overlap_tail(self) -> typing.List[ComparisonStrictItemType]: + def b_disjoint_tail(self) -> List[ComparisonStrictItemType]: + + return self._disjoint_head( + source=[item.b for item in self._merged][::-1], + target=[item.a for item in self._merged][::-1], + )[::-1] + + @property + def b_overlap_tail(self) -> List[ComparisonStrictItemType]: return self._overlap_tail( source=[item.b for item in self._merged], @@ -119,16 +129,16 @@ def b_overlap_tail(self) -> typing.List[ComparisonStrictItemType]: ) @property - def merged(self) -> typing.Generator[ComparisonItemABC, None, None]: + def merged(self) -> Generator[ComparisonItemABC, None, None]: return (item for item in self._merged) @classmethod - def _head( + def _disjoint_head( cls, - source: typing.List[ComparisonItemType], - target: typing.List[ComparisonItemType], - ) -> typing.List[ComparisonItemType]: + source: List[ComparisonItemType], + target: List[ComparisonItemType], + ) -> List[ComparisonItemType]: """ Returns new elements from source. If target is empty, returns source. @@ -176,9 +186,9 @@ def _head( @classmethod def _overlap_tail( cls, - source: typing.List[ComparisonItemType], - target: typing.List[ComparisonItemType], - ) -> typing.List[ComparisonItemType]: + source: List[ComparisonItemType], + target: List[ComparisonItemType], + ) -> List[ComparisonItemType]: """ Overlap must include first element of source. """ @@ -218,8 +228,8 @@ def _overlap_tail( @classmethod def _strip_none( - cls, elements: typing.List[ComparisonItemType] - ) -> typing.List[ComparisonItemType]: + cls, elements: List[ComparisonItemType] + ) -> List[ComparisonItemType]: elements = cls._left_strip_none(elements) # left strip elements.reverse() # flip into reverse @@ -230,153 +240,133 @@ def _strip_none( @staticmethod def _left_strip_none( - elements: typing.List[ComparisonItemType], - ) -> typing.List[ComparisonItemType]: + elements: List[ComparisonItemType], + ) -> List[ComparisonItemType]: return list(itertools.dropwhile(lambda element: element is None, elements)) @staticmethod - def _single_items( - items_a: typing.Union[ComparisonMergeTypes, None], - items_b: typing.Union[ComparisonMergeTypes, None], - ) -> typing.List[ComparisonItemABC]: - - assert items_a is not None or items_b is not None - - if items_a is None: - return [ComparisonItem(None, item) for item in items_b] - return [ComparisonItem(item, None) for item in items_a] + def _test_alternations(items: List[Union[SnapshotABC, None]]): - @staticmethod - def _merge_datasets( - items_a: typing.Generator[DatasetABC, None, None], - items_b: typing.Generator[DatasetABC, None, None], - ) -> typing.List[ComparisonItemABC]: + alternations = 0 + state = False # None - items_a = {item.subname: item for item in items_a} - items_b = {item.subname: item for item in items_b} + for item in items: - names = list(items_a.keys() | items_b.keys()) - merged = [ - ComparisonItem(items_a.get(name, None), items_b.get(name, None)) - for name in names - ] - merged.sort(key=lambda item: item.get_item().name) + new_state = item is not None - return merged + if new_state == state: + continue - @classmethod - def from_zpools( - cls, - zpool_a: typing.Union[ZpoolABC, None], - zpool_b: typing.Union[ZpoolABC, None], - ) -> ComparisonABC: + alternations += 1 + state = new_state - assert zpool_a is not None or zpool_b is not None + if alternations > 2: + raise ValueError("gap in snapshot series") - if zpool_a is None or zpool_b is None: - return cls( - a=zpool_a, - b=zpool_b, - merged=cls._single_items( - getattr(zpool_a, "datasets", None), - getattr(zpool_b, "datasets", None), - ), - ) + @staticmethod + def _test_names(items: List[ComparisonItemABC]): - assert zpool_a is not zpool_b - assert zpool_a != zpool_b + for item in items: + if item.a is not None and item.b is not None: + if item.a.name != item.b.name: + raise ValueError("inconsistent snapshot names") - return cls( - a=zpool_a, - b=zpool_b, - merged=cls._merge_datasets(zpool_a.datasets, zpool_b.datasets), + @staticmethod + def _find_name(snapshots: List[SnapshotABC], name: str) -> Union[int, None]: + + return next( + ( + index + for (index, snapshot) in enumerate(snapshots) + if snapshot.name == name + ), + None, # if nothing is found, return None ) @staticmethod + def _squash(snapshots: List[SnapshotABC]) -> List[SnapshotABC]: + + squashed, buffer = [], [] + + for snapshot in snapshots: + + if snapshot.get("abgleich:type").value != "backup": + buffer.append(snapshot) + else: + snapshot.intermediates.clear() + snapshot.intermediates.extend(buffer) + squashed.append(snapshot) + buffer.clear() + + return squashed + + @classmethod def _merge_snapshots( - items_a: typing.Generator[SnapshotABC, None, None], - items_b: typing.Generator[SnapshotABC, None, None], - ) -> typing.List[ComparisonItemABC]: + cls, + items_a: Generator[SnapshotABC, None, None], + items_b: Generator[SnapshotABC, None, None], + config: ConfigABC, + ) -> List[ComparisonItemABC]: - items_a = list(items_a) - items_b = list(items_b) - names_a = [item.name for item in items_a] - names_b = [item.name for item in items_b] + items_a, items_b = list(items_a), list(items_b) - assert len(set(names_a)) == len(items_a) # unique names - assert len(set(names_b)) == len(items_b) # unique names + if config["compatibility/tagging"].value: + items_a, items_b = cls._squash(items_a), cls._squash(items_b) if len(items_a) == 0 and len(items_b) == 0: return [] + + assert len(set({item.name for item in items_a})) == len(items_a) # unique names + assert len(set({item.name for item in items_b})) == len(items_b) # unique names + if len(items_a) == 0: return [ComparisonItem(None, item) for item in items_b] if len(items_b) == 0: return [ComparisonItem(item, None) for item in items_a] - try: - start_b = names_a.index(names_b[0]) - except ValueError: - start_b = None - try: - start_a = names_b.index(names_a[0]) - except ValueError: - start_a = None + start_b = cls._find_name(items_a, items_b[0].name) + start_a = cls._find_name(items_b, items_a[0].name) assert start_a is not None or start_b is not None # overlap - prefix_a = [] if start_a is None else [None for _ in range(start_a)] - prefix_b = [] if start_b is None else [None for _ in range(start_b)] - items_a = prefix_a + items_a - items_b = prefix_b + items_b - suffix_a = ( - [] - if len(items_a) >= len(items_b) - else [None for _ in range(len(items_b) - len(items_a))] - ) - suffix_b = ( - [] - if len(items_b) >= len(items_a) - else [None for _ in range(len(items_a) - len(items_b))] - ) - items_a = items_a + suffix_a - items_b = items_b + suffix_b + if start_a is not None: # fill prefix + items_a = [None for _ in range(start_a)] + items_a + if start_b is not None: # fill prefix + items_b = [None for _ in range(start_b)] + items_b + if len(items_a) < len(items_b): # fill suffix + items_a = items_a + [None for _ in range(len(items_b) - len(items_a))] + if len(items_b) < len(items_a): # fill suffix + items_b = items_b + [None for _ in range(len(items_a) - len(items_b))] assert len(items_a) == len(items_b) - alt_a, alt_b, state_a, state_b = 0, 0, False, False - merged = [] - for item_a, item_b in zip(items_a, items_b): - new_state_a, new_state_b = item_a is not None, item_b is not None - if new_state_a != state_a: - alt_a, state_a = alt_a + 1, new_state_a - if alt_a > 2: - raise ValueError("gap in snapshot series") - if new_state_b != state_b: - alt_b, state_b = alt_b + 1, new_state_b - if alt_b > 2: - raise ValueError("gap in snapshot series") - if state_a and state_b: - if item_a.name != item_b.name: - raise ValueError("inconsistent snapshot names") - merged.append(ComparisonItem(item_a, item_b)) + cls._test_alternations(items_a) + cls._test_alternations(items_b) + + merged = [ + ComparisonItem(item_a, item_b) for item_a, item_b in zip(items_a, items_b) + ] + + cls._test_names(merged) return merged @classmethod def from_datasets( cls, - dataset_a: typing.Union[DatasetABC, None], - dataset_b: typing.Union[DatasetABC, None], - ) -> ComparisonABC: + dataset_a: Union[DatasetABC, None], + dataset_b: Union[DatasetABC, None], + config: ConfigABC, + ) -> ComparisonDatasetABC: assert dataset_a is not None or dataset_b is not None - if dataset_a is None or dataset_b is None: + if (dataset_a is None) ^ (dataset_b is None): return cls( a=dataset_a, b=dataset_b, - merged=cls._single_items( + merged=ComparisonItem.list_from_singles( getattr(dataset_a, "snapshots", None), getattr(dataset_b, "snapshots", None), ), @@ -388,37 +378,7 @@ def from_datasets( return cls( a=dataset_a, b=dataset_b, - merged=cls._merge_snapshots(dataset_a.snapshots, dataset_b.snapshots), + merged=cls._merge_snapshots( + dataset_a.snapshots, dataset_b.snapshots, config + ), ) - - -@typeguard.typechecked -class ComparisonItem(ComparisonItemABC): - def __init__(self, a: ComparisonItemType, b: ComparisonItemType): - - assert a is not None or b is not None - if a is not None and b is not None: - assert type(a) == type(b) - - self._a, self._b = a, b - - def get_item(self) -> ComparisonStrictItemType: - - if self._a is not None: - return self._a - return self._b - - @property - def complete(self) -> bool: - - return self._a is not None and self._b is not None - - @property - def a(self) -> ComparisonItemType: - - return self._a - - @property - def b(self) -> ComparisonItemType: - - return self._b diff --git a/src/abgleich/core/comparisonitem.py b/src/abgleich/core/comparisonitem.py new file mode 100644 index 0000000..ec63d96 --- /dev/null +++ b/src/abgleich/core/comparisonitem.py @@ -0,0 +1,108 @@ +# -*- coding: utf-8 -*- + +""" + +ABGLEICH +zfs sync tool +https://github.com/pleiszenburg/abgleich + + src/abgleich/core/comparisonitem.py: ZFS comparison item + + Copyright (C) 2019-2022 Sebastian M. Ernst + + +The contents of this file are subject to the GNU Lesser General Public License +Version 2.1 ("LGPL" or "License"). You may not use this file except in +compliance with the License. You may obtain a copy of the License at +https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt +https://github.com/pleiszenburg/abgleich/blob/master/LICENSE + +Software distributed under the License is distributed on an "AS IS" basis, +WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the +specific language governing rights and limitations under the License. + + +""" + + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# IMPORT +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +from typing import Generator, List, Union + +from typeguard import typechecked + +from .abc import ComparisonItemABC, DatasetABC, SnapshotABC + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# TYPING +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +ComparisonGeneratorType = Union[ + Generator[DatasetABC, None, None], + Generator[SnapshotABC, None, None], + None, +] +ComparisonItemType = Union[ + DatasetABC, + SnapshotABC, + None, +] +ComparisonStrictItemType = Union[ + DatasetABC, + SnapshotABC, +] + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# CLASS +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + + +@typechecked +class ComparisonItem(ComparisonItemABC): + """ + Immutable. + """ + + def __init__(self, a: ComparisonItemType, b: ComparisonItemType): + + assert a is not None or b is not None + if a is not None and b is not None: + assert type(a) == type(b) + + self._a, self._b = a, b + + def get_item(self) -> ComparisonStrictItemType: + + if self._a is not None: + return self._a + return self._b + + @property + def complete(self) -> bool: + + return self._a is not None and self._b is not None + + @property + def a(self) -> ComparisonItemType: + + return self._a + + @property + def b(self) -> ComparisonItemType: + + return self._b + + @classmethod + def list_from_singles( + cls, + items_a: ComparisonGeneratorType, + items_b: ComparisonGeneratorType, + ) -> List[ComparisonItemABC]: + + assert (items_a is not None) ^ (items_b is not None) + + if items_a is None: + return [cls(None, item) for item in items_b] + return [cls(item, None) for item in items_a] diff --git a/src/abgleich/core/comparisonzpool.py b/src/abgleich/core/comparisonzpool.py new file mode 100644 index 0000000..f976e4d --- /dev/null +++ b/src/abgleich/core/comparisonzpool.py @@ -0,0 +1,125 @@ +# -*- coding: utf-8 -*- + +""" + +ABGLEICH +zfs sync tool +https://github.com/pleiszenburg/abgleich + + src/abgleich/core/comparisonzpool.py: ZFS zpool comparison + + Copyright (C) 2019-2022 Sebastian M. Ernst + + +The contents of this file are subject to the GNU Lesser General Public License +Version 2.1 ("LGPL" or "License"). You may not use this file except in +compliance with the License. You may obtain a copy of the License at +https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt +https://github.com/pleiszenburg/abgleich/blob/master/LICENSE + +Software distributed under the License is distributed on an "AS IS" basis, +WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the +specific language governing rights and limitations under the License. + + +""" + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# IMPORT +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +from typing import Generator, List, Union + +from typeguard import typechecked + +from .abc import ComparisonItemABC, ComparisonZpoolABC, DatasetABC, ZpoolABC +from .comparisonitem import ComparisonItem + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# CLASS +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + + +@typechecked +class ComparisonZpool(ComparisonZpoolABC): + """ + Immutable. No order, just name-based matching. + """ + + def __init__( + self, + a: Union[ZpoolABC, None], + b: Union[ZpoolABC, None], + merged: List[ComparisonItemABC], + ): + + assert a is not None or b is not None + if a is not None and b is not None: + assert type(a) == type(b) + + self._a, self._b, self._merged = a, b, merged + + def __len__(self) -> int: + + return len(self._merged) + + @property + def a(self) -> Union[ZpoolABC, None]: + + return self._a + + @property + def b(self) -> Union[ZpoolABC, None]: + + return self._b + + @property + def merged(self) -> Generator[ComparisonItemABC, None, None]: + + return (item for item in self._merged) + + @staticmethod + def _merge_datasets( + items_a: Generator[DatasetABC, None, None], + items_b: Generator[DatasetABC, None, None], + ) -> List[ComparisonItemABC]: + + items_a = {item.subname: item for item in items_a} + items_b = {item.subname: item for item in items_b} + + names = list(items_a.keys() | items_b.keys()) + merged = [ + ComparisonItem(items_a.get(name, None), items_b.get(name, None)) + for name in names + ] + merged.sort(key=lambda item: item.get_item().name) + + return merged + + @classmethod + def from_zpools( + cls, + zpool_a: Union[ZpoolABC, None], + zpool_b: Union[ZpoolABC, None], + ) -> ComparisonZpoolABC: + + assert zpool_a is not None or zpool_b is not None + + if (zpool_a is None) ^ (zpool_b is None): + return cls( + a=zpool_a, + b=zpool_b, + merged=ComparisonItem.list_from_singles( + getattr(zpool_a, "datasets", None), + getattr(zpool_b, "datasets", None), + ), + ) + + assert zpool_a is not zpool_b + assert zpool_a != zpool_b + + return cls( + a=zpool_a, + b=zpool_b, + merged=cls._merge_datasets(zpool_a.datasets, zpool_b.datasets), + ) diff --git a/src/abgleich/core/config.py b/src/abgleich/core/config.py index 045db00..903c248 100644 --- a/src/abgleich/core/config.py +++ b/src/abgleich/core/config.py @@ -8,7 +8,7 @@ src/abgleich/core/config.py: Handles configuration data - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -29,9 +29,9 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import typing +from typing import Dict, TextIO, Union -import typeguard +from typeguard import typechecked import yaml try: @@ -39,53 +39,79 @@ except ImportError: from yaml import FullLoader as Loader -from .abc import ConfigABC -from .lib import valid_name +from .abc import ConfigABC, ConfigFieldABC +from .configspec import CONFIGSPEC # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CLASS # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked -class Config(ConfigABC, dict): +@typechecked +class Config(ConfigABC): + """ + Immutable. + """ + + def __init__(self, root: Union[str, None] = None, **kwargs: ConfigFieldABC): + + self._root = root + self._fields = kwargs + + def __repr__(self): + + return "" if self._root is None else f'' + + def __getitem__(self, key: str) -> ConfigFieldABC: + + return ( + self._fields[key] + if self._root is None + else self._fields[f"{self._root:s}/{key:s}"] + ) + + def group(self, root: str) -> ConfigABC: + + return type(self)(root=root, **self._fields) + @classmethod - def from_fd(cls, fd: typing.TextIO): - - ssh_schema = { - "compression": lambda v: isinstance(v, bool), - "cipher": lambda v: isinstance(v, str) or v is None, - } - - side_schema = { - "zpool": lambda v: isinstance(v, str) and len(v) > 0, - "prefix": lambda v: isinstance(v, str) or v is None, - "host": lambda v: isinstance(v, str) and len(v) > 0, - "user": lambda v: isinstance(v, str) or v is None, - } - - root_schema = { - "source": lambda v: cls._validate(data=v, schema=side_schema), - "target": lambda v: cls._validate(data=v, schema=side_schema), - "keep_snapshots": lambda v: isinstance(v, int) and v >= 1, - "suffix": lambda v: v is None or (isinstance(v, str) and valid_name(v)), - "digits": lambda v: isinstance(v, int) and v >= 1, - "ignore": lambda v: isinstance(v, list) - and all((isinstance(item, str) and len(item) > 0 for item in v)), - "ssh": lambda v: cls._validate(data=v, schema=ssh_schema), - } - - config = yaml.load(fd.read(), Loader=Loader) - cls._validate(data=config, schema=root_schema) - return cls(config) + def _flatten_dict_tree(cls, data: Dict, root: Union[str, None] = None) -> Dict: + + flat_data = {} + + for key, value in data.items(): + if not isinstance(key, str): + raise TypeError("configuration key is no string", key) + if root is not None: + if len(root) > 0: + key = f"{root:s}/{key:s}" + if isinstance(value, dict): + flat_data.update(cls._flatten_dict_tree(data=value, root=key)) + else: + flat_data[key] = value + + return flat_data @classmethod - def _validate(cls, data: typing.Dict, schema: typing.Dict): + def from_fd(cls, fd: TextIO) -> ConfigABC: + + return cls.from_text(fd.read()) + + @classmethod + def from_text(cls, text: str) -> ConfigABC: + + config = yaml.load(text, Loader=Loader) + + if not isinstance(config, dict): + raise TypeError("config is no dict", config) + + config_fields = {field.name: field.copy() for field in CONFIGSPEC} + for key, value in cls._flatten_dict_tree(data=config).items(): + if key not in config_fields.keys(): + raise ValueError("unknown configuration key", key) + config_fields[key].value = value - for field, validator in schema.items(): - if field not in data.keys(): - raise KeyError(f'missing configuration field "{field:s}"') - if not validator(data[field]): - raise ValueError(f'invalid value in field "{field:s}"') + if any((not field.valid for field in config_fields.values())): + raise ValueError("configuration is not valid") - return True + return cls(**config_fields) diff --git a/src/abgleich/core/configfield.py b/src/abgleich/core/configfield.py new file mode 100644 index 0000000..60de58a --- /dev/null +++ b/src/abgleich/core/configfield.py @@ -0,0 +1,132 @@ +# -*- coding: utf-8 -*- + +""" + +ABGLEICH +zfs sync tool +https://github.com/pleiszenburg/abgleich + + src/abgleich/core/configfield.py: Handles configuration fields + + Copyright (C) 2019-2022 Sebastian M. Ernst + + +The contents of this file are subject to the GNU Lesser General Public License +Version 2.1 ("LGPL" or "License"). You may not use this file except in +compliance with the License. You may obtain a copy of the License at +https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt +https://github.com/pleiszenburg/abgleich/blob/master/LICENSE + +Software distributed under the License is distributed on an "AS IS" basis, +WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the +specific language governing rights and limitations under the License. + + +""" + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# IMPORT +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +from typing import Callable, List, Union + +from typeguard import typechecked + +from .abc import ConfigFieldABC + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# TYPING +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +ConfigValueTypes = Union[List[str], str, int, float, bool, None] + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# CLASS +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + + +@typechecked +class ConfigField(ConfigFieldABC): + """ + Mutable. + """ + + def __init__( + self, + name: str, + validate: Callable, + default: ConfigValueTypes = None, + ): + + self._name = name + self._default = default + self._validate = validate + + if self._default is not None: + if not self._validate(self._default): + raise ValueError(f"invalid default value for {self._name}") + + self._value = None + + def __repr__(self) -> str: + + return ( + "" + ) + + def copy(self) -> ConfigFieldABC: + + return type(self)( + name=self._name, + default=self._default, + validate=self._validate, + ) + + @property + def name(self) -> str: + + return self._name + + @property + def value(self) -> ConfigValueTypes: + + if self._value is not None: + return self._value + + if self._default is None: + raise ValueError(f"required value for {self._name} missing") + + return self._default + + @value.setter + def value(self, value: ConfigValueTypes): + + if self._value is not None: + raise ValueError(f"value for {self._name} has already been set") + if value is None: + return + if not self._validate(value): + raise ValueError(f"invalid value for {self._name}") + + self._value = value + + @property + def valid(self) -> bool: + + return (self._value is not None and self._validate(self._value)) or ( + self._default is not None and self._validate(self._default) + ) + + @property + def required(self) -> bool: + + return self._default is None + + @property + def set(self) -> bool: + + return self._value is not None diff --git a/src/abgleich/core/configspec.py b/src/abgleich/core/configspec.py new file mode 100644 index 0000000..c81125e --- /dev/null +++ b/src/abgleich/core/configspec.py @@ -0,0 +1,148 @@ +# -*- coding: utf-8 -*- + +""" + +ABGLEICH +zfs sync tool +https://github.com/pleiszenburg/abgleich + + src/abgleich/core/configspec.py: Defines configuration fields + + Copyright (C) 2019-2022 Sebastian M. Ernst + + +The contents of this file are subject to the GNU Lesser General Public License +Version 2.1 ("LGPL" or "License"). You may not use this file except in +compliance with the License. You may obtain a copy of the License at +https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt +https://github.com/pleiszenburg/abgleich/blob/master/LICENSE + +Software distributed under the License is distributed on an "AS IS" basis, +WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the +specific language governing rights and limitations under the License. + + +""" + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# IMPORT +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +from .configfield import ConfigField +from .lib import valid_name + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# SPEC +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +CONFIGSPEC = [ + ConfigField( + name="keep_snapshots", + validate=lambda v: isinstance(v, int) and v >= 1, + default=1, + ), + ConfigField( + name="keep_backlog", + validate=lambda v: (isinstance(v, int) and v >= 0) or isinstance(v, bool), + default=True, + ), + ConfigField( + name="suffix", + validate=lambda v: isinstance(v, str) and valid_name(v, min_len=0), + default="", + ), + ConfigField( + name="digits", + validate=lambda v: isinstance(v, int) and v >= 1, + default=2, + ), + ConfigField( + name="always_changed", + validate=lambda v: isinstance(v, bool), + default=False, + ), + ConfigField( + name="written_threshold", + validate=lambda v: isinstance(v, int) and v > 0, + default=1024 ** 2, + ), + ConfigField( + name="check_diff", + validate=lambda v: isinstance(v, bool), + default=True, + ), + ConfigField( + name="ignore", + validate=lambda v: isinstance(v, list) + and all((isinstance(item, str) and len(item) > 0 for item in v)), + default=list(), + ), + ConfigField( + name="include_root", + validate=lambda v: isinstance(v, bool), + default=True, + ), + ConfigField( + name="compatibility/tagging", + validate=lambda v: isinstance(v, bool), + default=False, + ), + ConfigField( + name="compatibility/target_samba_noshare", + validate=lambda v: isinstance(v, bool), + default=False, + ), + ConfigField( + name="compatibility/target_autosnapshot_ignore", + validate=lambda v: isinstance(v, bool), + default=False, + ), + ConfigField( + name="ssh/compression", + validate=lambda v: isinstance(v, bool), + default=False, + ), + ConfigField( + name="ssh/cipher", + validate=lambda v: isinstance(v, str), + default="", + ), +] + +for _side in ("source", "target"): + + CONFIGSPEC.extend( + [ + ConfigField( + name=f"{_side}/zpool", + validate=lambda v: isinstance(v, str) and len(v) > 0, + ), + ConfigField( + name=f"{_side}/prefix", + validate=lambda v: isinstance(v, str), + default="", + ), + ConfigField( + name=f"{_side}/host", + validate=lambda v: isinstance(v, str) and len(v) > 0, + default="localhost", + ), + ConfigField( + name=f"{_side}/user", + validate=lambda v: isinstance(v, str), + default="", + ), + ConfigField( + name=f"{_side}/port", + validate=lambda v: isinstance(v, int) and v >= 0, + default=0, + ), + ConfigField( + name=f"{_side}/processing", + validate=lambda v: isinstance(v, str), + default="", + ), + ] + ) + +del _side diff --git a/src/abgleich/core/dataset.py b/src/abgleich/core/dataset.py index ca0de90..4507b04 100644 --- a/src/abgleich/core/dataset.py +++ b/src/abgleich/core/dataset.py @@ -8,7 +8,7 @@ src/abgleich/core/dataset.py: ZFS dataset - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -29,22 +29,24 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ import datetime -import typing +from typing import Dict, Generator, List, Union # Python <= 3.7.1 "fix" try: from typing import OrderedDict as DictType except ImportError: - from typing import Dict as DictType + DictType = Dict -import typeguard +from typeguard import typechecked -from .abc import ConfigABC, DatasetABC, PropertyABC, TransactionABC, SnapshotABC +from .abc import ConfigABC, DatasetABC, PropertyABC, SnapshotABC, TransactionListABC from .command import Command from .i18n import t from .lib import root from .property import Property -from .transaction import Transaction, TransactionMeta +from .transaction import Transaction +from .transactionlist import TransactionList +from .transactionmeta import TransactionMeta from .snapshot import Snapshot # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ @@ -52,13 +54,17 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked class Dataset(DatasetABC): + """ + Immutable. + """ + def __init__( self, name: str, - properties: typing.Dict[str, PropertyABC], - snapshots: typing.List[SnapshotABC], + properties: Dict[str, PropertyABC], + snapshots: List[SnapshotABC], side: str, config: ConfigABC, ): @@ -69,7 +75,9 @@ def __init__( self._side = side self._config = config - self._root = root(config[side]["zpool"], config[side]["prefix"]) + self._root = root( + config[f"{side:s}/zpool"].value, config[f"{side:s}/prefix"].value + ) assert self._name.startswith(self._root) self._subname = self._name[len(self._root) :].strip("/") @@ -82,7 +90,7 @@ def __len__(self) -> int: return len(self._snapshots) - def __getitem__(self, key: typing.Union[str, int, slice]) -> PropertyABC: + def __getitem__(self, key: Union[str, int, slice]) -> PropertyABC: if isinstance(key, str): return self._properties[key] @@ -90,13 +98,14 @@ def __getitem__(self, key: typing.Union[str, int, slice]) -> PropertyABC: def get( self, - key: typing.Union[str, int, slice], - default: typing.Union[None, PropertyABC] = None, - ) -> typing.Union[None, PropertyABC]: + key: Union[str, int, slice], + default: Union[None, PropertyABC] = None, + ) -> Union[None, PropertyABC]: if isinstance(key, str): return self._properties.get( - key, Property(key, None, None) if default is None else default, + key, + Property(key, None, None) if default is None else default, ) assert isinstance(key, int) or isinstance(key, slice) @@ -111,26 +120,41 @@ def changed(self) -> bool: if len(self) == 0: return True - if self._config["always_changed"]: + if self._config["always_changed"].value: return True + + if self._config["compatibility/tagging"].value: + if self._snapshots[-1].get("abgleich:type").value != "backup": + return True + if self._properties["written"].value == 0: return False if self._properties["type"].value == "volume": return True - if self._config["written_threshold"] is not None: - if self._properties["written"].value > self._config["written_threshold"]: + if self._config["written_threshold"].value is not None: + if ( + self._properties["written"].value + > self._config["written_threshold"].value + ): return True - if not self._config["check_diff"]: + if not self._config["check_diff"].value: return True - output, _ = Command.on_side( - ["zfs", "diff", f"{self._name:s}@{self._snapshots[-1].name:s}"], - self._side, - self._config, - ).run() - return len(output.strip(" \t\n")) > 0 + output, _ = ( + Command.from_list( + ["zfs", "diff", f"{self._name:s}@{self._snapshots[-1].name:s}"] + ) + .on_side(side=self._side, config=self._config) + .run() + ) + return len(output[0].strip(" \t\n")) > 0 + + @property + def ignore(self) -> bool: + + return self._subname in self._config["ignore"].value @property def name(self) -> str: @@ -143,7 +167,7 @@ def subname(self) -> str: return self._subname @property - def snapshots(self) -> typing.Generator[SnapshotABC, None, None]: + def snapshots(self) -> Generator[SnapshotABC, None, None]: return (snapshot for snapshot in self._snapshots) @@ -152,33 +176,40 @@ def root(self) -> str: return self._root - def get_snapshot_transaction(self) -> TransactionABC: + def get_snapshot_transactions(self) -> TransactionListABC: snapshot_name = self._new_snapshot_name() - return Transaction( - TransactionMeta( - **{ - t("type"): t("snapshot"), - t("dataset_subname"): self._subname, - t("snapshot_name"): snapshot_name, - t("written"): self._properties["written"].value, - } - ), - [ - Command.on_side( - ["zfs", "snapshot", f"{self._name:s}@{snapshot_name:s}"], - self._side, - self._config, - ) - ], + command = ["zfs", "snapshot"] + if self._config["compatibility/tagging"].value: + command.extend(["-o", "abgleich:type=backup"]) + command.append(f"{self._name:s}@{snapshot_name:s}") + + return TransactionList( + Transaction( + meta=TransactionMeta( + **{ + t("type"): t("snapshot"), + t("dataset_subname"): self._subname, + t("snapshot_name"): snapshot_name, + t("written"): self._properties["written"].value, + } + ), + command=Command.from_list(command).on_side( + side=self._side, config=self._config + ), + ) ) def _new_snapshot_name(self) -> str: today = datetime.datetime.now().strftime("%Y%m%d") - max_snapshots = (10 ** self._config["digits"]) - 1 - suffix = self._config["suffix"] if self._config["suffix"] is not None else "" + max_snapshots = (10 ** self._config["digits"].value) - 1 + suffix = ( + self._config["suffix"].value + if self._config["suffix"].value is not None + else "" + ) todays_names = [ snapshot.name @@ -188,14 +219,14 @@ def _new_snapshot_name(self) -> str: snapshot.name.startswith(today), snapshot.name.endswith(suffix), len(snapshot.name) - == len(today) + self._config["digits"] + len(suffix), + == len(today) + self._config["digits"].value + len(suffix), ) ) ] todays_numbers = [ - int(name[len(today) : len(today) + self._config["digits"]]) + int(name[len(today) : len(today) + self._config["digits"].value]) for name in todays_names - if name[len(today) : len(today) + self._config["digits"]].isnumeric() + if name[len(today) : len(today) + self._config["digits"].value].isnumeric() ] if len(todays_numbers) != 0: todays_numbers.sort() @@ -211,7 +242,7 @@ def _new_snapshot_name(self) -> str: def from_entities( cls, name: str, - entities: DictType[str, typing.List[typing.List[str]]], + entities: DictType[str, List[List[str]]], side: str, config: ConfigABC, ) -> DatasetABC: @@ -226,7 +257,11 @@ def from_entities( snapshots.extend( ( Snapshot.from_entity( - snapshot_name, entities[snapshot_name], snapshots, side, config, + snapshot_name, + entities[snapshot_name], + snapshots, + side, + config, ) for snapshot_name in entities.keys() ) diff --git a/src/abgleich/core/i18n.py b/src/abgleich/core/i18n.py index 09b5ec9..9620386 100644 --- a/src/abgleich/core/i18n.py +++ b/src/abgleich/core/i18n.py @@ -8,7 +8,7 @@ src/abgleich/core/i18n.py: Translations - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -31,7 +31,7 @@ import locale import os -import typeguard +from typeguard import typechecked import yaml try: @@ -49,8 +49,12 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked class _Lang(dict): + """ + Mutable. + """ + def __init__(self): super().__init__() diff --git a/src/abgleich/core/io.py b/src/abgleich/core/io.py index bd2fae0..947fc5b 100644 --- a/src/abgleich/core/io.py +++ b/src/abgleich/core/io.py @@ -8,7 +8,7 @@ src/abgleich/core/io.py: Command line IO - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,9 +28,9 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import typing +from typing import Union -import typeguard +from typeguard import typechecked # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CONSTANTS @@ -57,14 +57,14 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked def colorize(text: str, col: str) -> str: return c.get(col.upper(), c["GREY"]) + text + c["RESET"] -@typeguard.typechecked +@typechecked def humanize_size( - size: typing.Union[float, int], add_color: bool = False, get_rgb: bool = False + size: Union[float, int], add_color: bool = False, get_rgb: bool = False ) -> str: suffix = "B" diff --git a/src/abgleich/core/lib.py b/src/abgleich/core/lib.py index 3f7b4ad..d54787d 100644 --- a/src/abgleich/core/lib.py +++ b/src/abgleich/core/lib.py @@ -8,7 +8,7 @@ src/abgleich/core/lib.py: ZFS library - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -29,9 +29,9 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ import re -import typing +from typing import Union -import typeguard +from typeguard import typechecked from .abc import ConfigABC from .command import Command @@ -41,19 +41,23 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked def is_host_up(side: str, config: ConfigABC) -> bool: assert side in ("source", "target") - if config[side]["host"] == "localhost": + if config[f"{side:s}/host"].value == "localhost": return True - _, _, returncode, _ = Command.on_side(["exit"], side, config).run(returncode=True) - assert returncode in (0, 255) - return returncode == 0 + _, _, returncode, _ = ( + Command.from_list(["exit"]) + .on_side(side=side, config=config) + .run(returncode=True) + ) + assert returncode[0] in (0, 255) + return returncode[0] == 0 -@typeguard.typechecked +@typechecked def join(*args: str) -> str: if len(args) < 2: @@ -67,22 +71,28 @@ def join(*args: str) -> str: return "/".join(args) -@typeguard.typechecked -def root(zpool: str, prefix: typing.Union[str, None]) -> str: +@typechecked +def root(zpool: str, prefix: Union[str, None]) -> str: if prefix is None: return zpool + if len(prefix) == 0: + return zpool return join(zpool, prefix) _name_re = re.compile("^[A-Za-z0-9_]+$") -@typeguard.typechecked +@typechecked def valid_name(name: str, min_len: int = 1) -> bool: assert min_len >= 0 if len(name) < min_len: return False + + if min_len == 0 and len(name) == 0: + return True + return bool(_name_re.match(name)) diff --git a/src/abgleich/core/property.py b/src/abgleich/core/property.py index ee12bdf..dee76e8 100644 --- a/src/abgleich/core/property.py +++ b/src/abgleich/core/property.py @@ -8,7 +8,7 @@ src/abgleich/core/filesystem.py: ZFS filesystem - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,9 +28,9 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import typing +from typing import Union -import typeguard +from typeguard import typechecked from .abc import PropertyABC @@ -38,17 +38,24 @@ # TYPING # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -PropertyTypes = typing.Union[str, int, float, None] +PropertyTypes = Union[str, int, float, None] # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CLASS # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked class Property(PropertyABC): + """ + Immutable. + """ + def __init__( - self, name: str, value: PropertyTypes, src: PropertyTypes, + self, + name: str, + value: PropertyTypes, + src: PropertyTypes = None, ): self._name = name @@ -63,12 +70,20 @@ def name(self) -> str: def value(self) -> PropertyTypes: return self._value + @property + def value_export(self) -> str: + return self._export(self._value) + @property def src(self) -> PropertyTypes: return self._src + @property + def src_export(self) -> str: + return self._export(self._src) + @classmethod - def _convert(cls, value: str) -> PropertyTypes: + def _import(cls, value: str) -> PropertyTypes: value = value.strip() @@ -85,7 +100,15 @@ def _convert(cls, value: str) -> PropertyTypes: return value + def _export(self, value: PropertyTypes) -> str: + + return "-" if value is None else str(value) # TODO improve! + @classmethod def from_params(cls, name, value, src) -> PropertyABC: - return cls(name=name, value=cls._convert(value), src=cls._convert(src),) + return cls( + name=name, + value=cls._import(value), + src=cls._import(src), + ) diff --git a/src/abgleich/core/snapshot.py b/src/abgleich/core/snapshot.py index 1a5841e..c14c5f3 100644 --- a/src/abgleich/core/snapshot.py +++ b/src/abgleich/core/snapshot.py @@ -8,7 +8,7 @@ src/abgleich/core/snapshot.py: ZFS snapshot - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,30 +28,36 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import typing +from typing import Dict, List, Union -import typeguard +from typeguard import typechecked -from .abc import ConfigABC, PropertyABC, SnapshotABC, TransactionABC +from .abc import ConfigABC, PropertyABC, SnapshotABC, TransactionABC, TransactionListABC from .command import Command from .i18n import t from .lib import root from .property import Property -from .transaction import Transaction, TransactionMeta +from .transaction import Transaction +from .transactionlist import TransactionList +from .transactionmeta import TransactionMeta # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CLASS # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked class Snapshot(SnapshotABC): + """ + Immutable. + """ + def __init__( self, name: str, parent: str, - properties: typing.Dict[str, PropertyABC], - context: typing.List[SnapshotABC], + properties: Dict[str, PropertyABC], + context: List[SnapshotABC], side: str, config: ConfigABC, ): @@ -63,11 +69,15 @@ def __init__( self._side = side self._config = config - self._root = root(config[side]["zpool"], config[side]["prefix"]) + self._root = root( + config[f"{side:s}/zpool"].value, config[f"{side:s}/prefix"].value + ) assert self._parent.startswith(self._root) self._subparent = self._parent[len(self._root) :].strip("/") + self._intermediates = [] # for namespaces / tagging + def __eq__(self, other: SnapshotABC) -> bool: return self.subparent == other.subparent and self.name == other.name @@ -76,68 +86,101 @@ def __getitem__(self, name: str) -> PropertyABC: return self._properties[name] - def get_cleanup_transaction(self) -> TransactionABC: + def get( + self, + key: str, + default: Union[None, PropertyABC] = None, + ) -> Union[None, PropertyABC]: - assert self._side == "source" + return self._properties.get( + key, + Property(key, None, None) if default is None else default, + ) - return Transaction( - meta=TransactionMeta( - **{ - t("type"): t("cleanup_snapshot"), - t("snapshot_subparent"): self._subparent, - t("snapshot_name"): self._name, - } - ), - commands=[ - Command.on_side( - ["zfs", "destroy", f"{self._parent:s}@{self._name:s}"], - self._side, - self._config, - ) - ], + def get_cleanup_transactions(self) -> TransactionListABC: + + return TransactionList( + Transaction( + meta=TransactionMeta( + **{ + t("type"): t("cleanup_snapshot"), + t("snapshot_subparent"): self._subparent, + t("snapshot_name"): self._name, + } + ), + command=Command.from_list( + ["zfs", "destroy", f"{self._parent:s}@{self._name:s}"] + ).on_side(side=self._side, config=self._config), + ) ) - def get_backup_transaction( - self, source_dataset: str, target_dataset: str, - ) -> TransactionABC: + def get_backup_transactions( + self, + source_dataset: str, + target_dataset: str, + ) -> TransactionListABC: assert self._side == "source" ancestor = self.ancestor - commands = [ - Command.on_side( - ["zfs", "send", "-c", f"{source_dataset:s}@{self.name:s}",] - if ancestor is None - else [ - "zfs", - "send", - "-c", - "-i", - f"{source_dataset:s}@{ancestor.name:s}", - f"{source_dataset:s}@{self.name:s}", - ], - "source", - self._config, - ), - Command.on_side( - ["zfs", "receive", f"{target_dataset:s}"], "target", self._config - ), - ] - - return Transaction( - meta=TransactionMeta( - **{ - t("type"): t("transfer_snapshot") - if ancestor is None - else t("transfer_snapshot_incremental"), - t("snapshot_subparent"): self._subparent, - t("ancestor_name"): "" if ancestor is None else ancestor.name, - t("snapshot_name"): self.name, - } - ), - commands=commands, + send = Command.from_list( + [ + "zfs", + "send", + "-c", + f"{source_dataset:s}@{self.name:s}", + ] + if ancestor is None + else [ + "zfs", + "send", + "-c", + "-i", + f"{source_dataset:s}@{ancestor.name:s}", + f"{source_dataset:s}@{self.name:s}", + ] ) + receive = Command.from_list(["zfs", "receive", f"{target_dataset:s}"]) + + if self._config["source/processing"].set: + send = send | Command.from_str(self._config["source/processing"].value) + if self._config["target/processing"].set: + receive = ( + Command.from_str(self._config["target/processing"].value) | receive + ) + + command = send.on_side(side="source", config=self._config) | receive.on_side( + side="target", config=self._config + ) + + transactions = TransactionList( + Transaction( + meta=TransactionMeta( + **{ + t("type"): t("transfer_snapshot") + if ancestor is None + else t("transfer_snapshot_incremental"), + t("snapshot_subparent"): self._subparent, + t("ancestor_name"): "" if ancestor is None else ancestor.name, + t("snapshot_name"): self.name, + } + ), + command=command, + ) + ) + + if self._config["compatibility/tagging"].value: + transactions.append( + Transaction.set_property( + item=f"{target_dataset:s}@{self.name:s}", + property=Property(name="abgleich:type", value="backup"), + side="target", + config=self._config, + ) + ) + + return transactions @property def name(self) -> str: @@ -155,7 +198,7 @@ def subparent(self) -> str: return self._subparent @property - def ancestor(self) -> typing.Union[None, SnapshotABC]: + def ancestor(self) -> Union[None, SnapshotABC]: assert self in self._context self_index = self._context.index(self) @@ -169,12 +212,17 @@ def root(self) -> str: return self._root + @property + def intermediates(self) -> List[SnapshotABC]: + + return self._intermediates + @classmethod def from_entity( cls, name: str, - entity: typing.List[typing.List[str]], - context: typing.List[SnapshotABC], + entity: List[List[str]], + context: List[SnapshotABC], side: str, config: ConfigABC, ) -> SnapshotABC: diff --git a/src/abgleich/core/transaction.py b/src/abgleich/core/transaction.py index 3c2e558..d5293c1 100644 --- a/src/abgleich/core/transaction.py +++ b/src/abgleich/core/transaction.py @@ -8,7 +8,7 @@ src/abgleich/core/transaction.py: ZFS transactions - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,29 +28,34 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import typing +from typing import Callable, Union -from tabulate import tabulate -import typeguard +from typeguard import typechecked -from .abc import CommandABC, TransactionABC, TransactionListABC, TransactionMetaABC +from .abc import CommandABC, ConfigABC, PropertyABC, TransactionABC, TransactionMetaABC +from .command import Command from .i18n import t -from .io import colorize, humanize_size +from .transactionmeta import TransactionMeta + # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CLASS # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked class Transaction(TransactionABC): + """ + Mutable. + """ + def __init__( - self, meta: TransactionMetaABC, commands: typing.List[CommandABC], + self, + meta: TransactionMetaABC, + command: CommandABC, ): - assert len(commands) in (1, 2) - - self._meta, self._commands = meta, commands + self._meta, self._command = meta, command self._complete = False self._running = False @@ -59,12 +64,12 @@ def __init__( self._changed = None @property - def changed(self) -> typing.Union[None, typing.Callable]: + def changed(self) -> Union[None, Callable]: return self._changed @changed.setter - def changed(self, value: typing.Union[None, typing.Callable]): + def changed(self, value: Union[None, Callable]): self._changed = value @@ -74,12 +79,12 @@ def complete(self) -> bool: return self._complete @property - def commands(self) -> typing.Tuple[CommandABC]: + def command(self) -> CommandABC: - return self._commands + return self._command @property - def error(self) -> typing.Union[Exception, None]: + def error(self) -> Union[Exception, None]: return self._error @@ -103,12 +108,7 @@ def run(self): self._changed() try: - if len(self._commands) == 1: - output, errors = self._commands[0].run() - else: - errors_1, output_2, errors_2 = self._commands[0].run_pipe( - self._commands[1] - ) + _, _ = self._command.run() except SystemError as error: self._error = error finally: @@ -117,185 +117,28 @@ def run(self): if self._changed is not None: self._changed() - -MetaTypes = typing.Union[str, int, float] -MetaNoneTypes = typing.Union[str, int, float, None] - - -@typeguard.typechecked -class TransactionMeta(TransactionMetaABC): - def __init__(self, **kwargs: MetaTypes): - - self._meta = kwargs - - def __getitem__(self, key: str) -> MetaTypes: - - return self._meta[key] - - def __len__(self) -> int: - - return len(self._meta) - - def get(self, key: str) -> MetaNoneTypes: - - return self._meta.get(key, None) - - def keys(self) -> typing.Generator[str, None, None]: - - return (key for key in self._meta.keys()) - - -TransactionIterableTypes = typing.Union[ - typing.Generator[TransactionABC, None, None], - typing.List[TransactionABC], - typing.Tuple[TransactionABC], -] - - -@typeguard.typechecked -class TransactionList(TransactionListABC): - def __init__(self): - - self._transactions = [] - self._changed = None - - def __len__(self) -> int: - - return len(self._transactions) - - def __getitem__(self, index: int) -> TransactionABC: - - return self._transactions[index] - - @property - def changed(self) -> typing.Union[None, typing.Callable]: - - return self._changed - - @changed.setter - def changed(self, value: typing.Union[None, typing.Callable]): - - self._changed = value - - @property - def table_columns(self) -> typing.List[str]: - - headers = set() - for transaction in self._transactions: - keys = list(transaction.meta.keys()) - assert t("type") in keys - headers.update(keys) - headers = list(headers) - headers.sort() - - if len(headers) == 0: - return headers - - type_index = headers.index(t("type")) - if type_index != 0: - headers.pop(type_index) - headers.insert(0, t("type")) - - return headers - - @property - def table_rows(self) -> typing.List[str]: - - return [f'{t("transaction"):s} #{index:d}' for index in range(1, len(self) + 1)] - - def append(self, transaction: TransactionABC): - - self._transactions.append(transaction) - if self._changed is not None: - self._link_transaction(transaction) - - def extend(self, transactions: TransactionIterableTypes): - - transactions = list(transactions) - self._transactions.extend(transactions) - if self._changed is not None: - for transaction in transactions: - self._link_transaction(transaction) - - def clear(self): - - self._transactions.clear() - self._changed() - - def _link_transaction(self, transaction: TransactionABC): - - transaction.changed = lambda: self._changed( - self._transactions.index(transaction) - ) - transaction.changed() - - def print_table(self): - - if len(self) == 0: - return - - table_columns = self.table_columns - colalign = self._table_colalign(table_columns) - - table = [ - [ - self._table_format_cell(header, transaction.meta.get(header)) - for header in table_columns - ] - for transaction in self._transactions - ] - - print( - tabulate( - table, headers=table_columns, tablefmt="github", colalign=colalign, - ) + @classmethod + def set_property( + cls, + item: str, + property: PropertyABC, + side: str, + config: ConfigABC, + ) -> TransactionABC: + + return cls( + meta=TransactionMeta( + **{ + t("type"): t("set_property"), + t("item"): item, + } + ), + command=Command.from_list( + [ + "zfs", + "set", + f"{property.name:s}={property.value_export:s}", + item, + ] + ).on_side(side=side, config=config), ) - - @staticmethod - def _table_format_cell(header: str, value: MetaNoneTypes) -> str: - - FORMAT = { - t("written"): lambda v: humanize_size(v, add_color=True), - } - - return FORMAT.get(header, str)(value) - - @staticmethod - def _table_colalign(headers: typing.List[str]) -> typing.List[str]: - - RIGHT = (t("written"),) - DECIMAL = tuple() - - colalign = [] - for header in headers: - if header in RIGHT: - colalign.append("right") - elif header in DECIMAL: - colalign.append("decimal") - else: - colalign.append("left") - - return colalign - - def run(self): - - for transaction in self._transactions: - - print( - f'({colorize(transaction.meta[t("type")], "white"):s}) ' - f'{colorize(" | ".join([str(command) for command in transaction.commands]), "yellow"):s}' - ) - - assert not transaction.running - assert not transaction.complete - - transaction.run() - - assert not transaction.running - assert transaction.complete - - if transaction.error is not None: - print(colorize(t("FAILED"), "red")) - raise transaction.error - else: - print(colorize(t("OK"), "green")) diff --git a/src/abgleich/core/transactionlist.py b/src/abgleich/core/transactionlist.py new file mode 100644 index 0000000..ab6ffe1 --- /dev/null +++ b/src/abgleich/core/transactionlist.py @@ -0,0 +1,211 @@ +# -*- coding: utf-8 -*- + +""" + +ABGLEICH +zfs sync tool +https://github.com/pleiszenburg/abgleich + + src/abgleich/core/transactionlist.py: ZFS transaction list + + Copyright (C) 2019-2022 Sebastian M. Ernst + + +The contents of this file are subject to the GNU Lesser General Public License +Version 2.1 ("LGPL" or "License"). You may not use this file except in +compliance with the License. You may obtain a copy of the License at +https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt +https://github.com/pleiszenburg/abgleich/blob/master/LICENSE + +Software distributed under the License is distributed on an "AS IS" basis, +WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the +specific language governing rights and limitations under the License. + + +""" + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# IMPORT +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +from typing import Callable, Generator, List, Union + +from tabulate import tabulate +from typeguard import typechecked + +from .abc import TransactionABC, TransactionListABC +from .i18n import t +from .io import colorize, humanize_size +from .transactionmeta import TransactionMetaTypes + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# CLASS +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + + +@typechecked +class TransactionList(TransactionListABC): + """ + Mutable. + """ + + def __init__(self, *transactions: TransactionABC): + + self._transactions = list(transactions) + self._changed = None + + def __len__(self) -> int: + + return len(self._transactions) + + def __add__(self, other: TransactionListABC) -> TransactionListABC: + + return type(self)(*(tuple(self.transactions) + tuple(other.transactions))) + + def __getitem__(self, index: int) -> TransactionABC: + + return self._transactions[index] + + @property + def changed(self) -> Union[None, Callable]: + + return self._changed + + @changed.setter + def changed(self, value: Union[None, Callable]): + + self._changed = value + + @property + def table_columns(self) -> List[str]: + + headers = set() + for transaction in self._transactions: + keys = list(transaction.meta.keys()) + assert t("type") in keys + headers.update(keys) + headers = list(headers) + headers.sort() + + if len(headers) == 0: + return headers + + type_index = headers.index(t("type")) + if type_index != 0: + headers.pop(type_index) + headers.insert(0, t("type")) + + return headers + + @property + def table_rows(self) -> List[str]: + + return [f'{t("transaction"):s} #{index:d}' for index in range(1, len(self) + 1)] + + @property + def transactions(self) -> Generator[TransactionABC, None, None]: + + return (transaction for transaction in self._transactions) + + def append(self, transaction: TransactionABC): + + self._transactions.append(transaction) + if self._changed is not None: + self._link_transaction(transaction) + + def extend(self, transactions: TransactionListABC): + + transactions = list(transactions.transactions) + self._transactions.extend(transactions) + if self._changed is not None: + for transaction in transactions: + self._link_transaction(transaction) + + def clear(self): + + self._transactions.clear() + if self._changed is not None: + self._changed() + + def _link_transaction(self, transaction: TransactionABC): + + transaction.changed = lambda: self._changed( + self._transactions.index(transaction) + ) + transaction.changed() + + def print_table(self): + + if len(self) == 0: + return + + table_columns = self.table_columns + colalign = self._table_colalign(table_columns) + + table = [ + [ + self._table_format_cell(header, transaction.meta.get(header)) + for header in table_columns + ] + for transaction in self._transactions + ] + + print( + tabulate( + table, + headers=table_columns, + tablefmt="github", + colalign=colalign, + ) + ) + + @staticmethod + def _table_format_cell( + header: str, value: Union[TransactionMetaTypes, None] + ) -> str: + + FORMAT = { + t("written"): lambda v: humanize_size(v, add_color=True), + } + + return FORMAT.get(header, str)(value) + + @staticmethod + def _table_colalign(headers: List[str]) -> List[str]: + + RIGHT = (t("written"),) + DECIMAL = tuple() + + colalign = [] + for header in headers: + if header in RIGHT: + colalign.append("right") + elif header in DECIMAL: + colalign.append("decimal") + else: + colalign.append("left") + + return colalign + + def run(self): + + for transaction in self._transactions: + + print( + f'({colorize(transaction.meta[t("type")], "white"):s}) ' + f'{colorize(str(transaction.command), "yellow"):s}' + ) + + assert not transaction.running + assert not transaction.complete + + transaction.run() + + assert not transaction.running + assert transaction.complete + + if transaction.error is not None: + print(colorize(t("FAILED"), "red")) + raise transaction.error + else: + print(colorize(t("OK"), "green")) diff --git a/src/abgleich/core/transactionmeta.py b/src/abgleich/core/transactionmeta.py new file mode 100644 index 0000000..bba64f6 --- /dev/null +++ b/src/abgleich/core/transactionmeta.py @@ -0,0 +1,72 @@ +# -*- coding: utf-8 -*- + +""" + +ABGLEICH +zfs sync tool +https://github.com/pleiszenburg/abgleich + + src/abgleich/core/transactionmeta.py: ZFS transaction meta + + Copyright (C) 2019-2022 Sebastian M. Ernst + + +The contents of this file are subject to the GNU Lesser General Public License +Version 2.1 ("LGPL" or "License"). You may not use this file except in +compliance with the License. You may obtain a copy of the License at +https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt +https://github.com/pleiszenburg/abgleich/blob/master/LICENSE + +Software distributed under the License is distributed on an "AS IS" basis, +WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the +specific language governing rights and limitations under the License. + + +""" + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# IMPORT +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +from typing import Generator, Union + +from typeguard import typechecked + +from .abc import TransactionMetaABC + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# TYPES +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +TransactionMetaTypes = Union[str, int, float] + +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +# CLASS +# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + + +@typechecked +class TransactionMeta(TransactionMetaABC): + """ + Immutable. + """ + + def __init__(self, **kwargs: TransactionMetaTypes): + + self._meta = kwargs + + def __getitem__(self, key: str) -> TransactionMetaTypes: + + return self._meta[key] + + def __len__(self) -> int: + + return len(self._meta) + + def get(self, key: str) -> Union[TransactionMetaTypes, None]: + + return self._meta.get(key, None) + + def keys(self) -> Generator[str, None, None]: + + return (key for key in self._meta.keys()) diff --git a/src/abgleich/core/zpool.py b/src/abgleich/core/zpool.py index 5cefc47..bf7b2b3 100644 --- a/src/abgleich/core/zpool.py +++ b/src/abgleich/core/zpool.py @@ -8,7 +8,7 @@ src/abgleich/core/zpool.py: ZFS zpool - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -29,52 +29,62 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ from collections import OrderedDict -import typing +from typing import Generator, List, Tuple, Union from tabulate import tabulate -import typeguard +from typeguard import typechecked from .abc import ( ComparisonItemABC, ConfigABC, DatasetABC, SnapshotABC, - TransactionABC, TransactionListABC, ZpoolABC, ) from .command import Command -from .comparison import Comparison +from .comparisondataset import ComparisonDataset +from .comparisonzpool import ComparisonZpool from .dataset import Dataset from .i18n import t from .io import colorize, humanize_size from .lib import join, root from .property import Property -from .transaction import TransactionList +from .transaction import Transaction +from .transactionlist import TransactionList # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # CLASS # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked class Zpool(ZpoolABC): + """ + Immutable. + """ + def __init__( - self, datasets: typing.List[DatasetABC], side: str, config: ConfigABC, + self, + datasets: List[DatasetABC], + side: str, + config: ConfigABC, ): self._datasets = datasets self._side = side self._config = config - self._root = root(config[side]["zpool"], config[side]["prefix"]) + self._root = root( + config[f"{side:s}/zpool"].value, config[f"{side:s}/prefix"].value + ) def __eq__(self, other: ZpoolABC) -> bool: return self.side == other.side @property - def datasets(self) -> typing.Generator[DatasetABC, None, None]: + def datasets(self) -> Generator[DatasetABC, None, None]: return (dataset for dataset in self._datasets) @@ -88,40 +98,33 @@ def root(self) -> str: return self._root - def get_cleanup_transactions(self, other: ZpoolABC,) -> TransactionListABC: + def get_cleanup_transactions( + self, + other: ZpoolABC, + ) -> TransactionListABC: - assert self.side == "source" - assert other.side == "target" + assert self.side != other.side + assert self.side in ("source", "target") + assert other.side in ("source", "target") - zpool_comparison = Comparison.from_zpools(self, other) + zpool_comparison = ComparisonZpool.from_zpools(self, other) transactions = TransactionList() for dataset_item in zpool_comparison.merged: - - cleanup_transactions = self._get_cleanup_from_datasetitem(dataset_item) - if cleanup_transactions is None: - continue - transactions.extend(cleanup_transactions) + transactions.extend(self._get_cleanup_from_datasetitem(dataset_item)) return transactions def generate_cleanup_transactions( - self, other: ZpoolABC, - ) -> typing.Generator[ - typing.Tuple[ - int, - typing.Union[ - None, typing.Union[None, typing.Generator[TransactionABC, None, None]] - ], - ], - None, - None, - ]: + self, + other: ZpoolABC, + ) -> Generator[Tuple[int, Union[None, TransactionListABC]], None, None]: - assert self.side == "source" - assert other.side == "target" + assert self.side != other.side + assert self.side in ("source", "target") + assert other.side in ("source", "target") - zpool_comparison = Comparison.from_zpools(self, other) + zpool_comparison = ComparisonZpool.from_zpools(self, other) yield len(zpool_comparison), None @@ -129,54 +132,68 @@ def generate_cleanup_transactions( yield index, self._get_cleanup_from_datasetitem(dataset_item) def _get_cleanup_from_datasetitem( - self, dataset_item: ComparisonItemABC, - ) -> typing.Union[None, typing.Generator[TransactionABC, None, None]]: + self, + dataset_item: ComparisonItemABC, + ) -> TransactionListABC: - if dataset_item.get_item().subname in self._config["ignore"]: - return + if dataset_item.get_item().ignore: + return TransactionList() if dataset_item.a is None or dataset_item.b is None: - return + return TransactionList() + if self.side == "target" and self._config["keep_backlog"].value == True: + return TransactionList() + + dataset_comparison = ComparisonDataset.from_datasets( + dataset_item.a, dataset_item.b, self._config + ) # TODO namespace + + if self.side == "source": + snapshots = dataset_comparison.a_overlap_tail[ + : -self._config["keep_snapshots"].value + ] + else: # target + if self._config["keep_backlog"].value in (False, 0): + keep_backlog = None + else: + keep_backlog = -self._config["keep_backlog"].value + snapshots = dataset_comparison.a_disjoint_tail[:keep_backlog] - dataset_comparison = Comparison.from_datasets(dataset_item.a, dataset_item.b) - snapshots = dataset_comparison.a_overlap_tail[: -self._config["keep_snapshots"]] + transactions = TransactionList() - return (snapshot.get_cleanup_transaction() for snapshot in snapshots) + for snapshot in snapshots: + transactions.extend(snapshot.get_cleanup_transactions()) - def get_backup_transactions(self, other: ZpoolABC,) -> TransactionListABC: + return transactions + + def get_backup_transactions( + self, + other: ZpoolABC, + ) -> TransactionListABC: assert self.side == "source" assert other.side == "target" - zpool_comparison = Comparison.from_zpools(self, other) + zpool_comparison = ComparisonZpool.from_zpools(self, other) transactions = TransactionList() for dataset_item in zpool_comparison.merged: - backup_transactions = self._get_backup_transactions_from_datasetitem( - other, dataset_item + transactions.extend( + self._get_backup_transactions_from_datasetitem(other, dataset_item) ) - if backup_transactions is None: - continue - transactions.extend(backup_transactions) + + transactions.extend(self._get_backup_propery_transactions(other)) return transactions def generate_backup_transactions( - self, other: ZpoolABC, - ) -> typing.Generator[ - typing.Tuple[ - int, - typing.Union[ - None, typing.Union[None, typing.Generator[TransactionABC, None, None]] - ], - ], - None, - None, - ]: + self, + other: ZpoolABC, + ) -> Generator[Tuple[int, Union[None, TransactionListABC]], None, None]: assert self.side == "source" assert other.side == "target" - zpool_comparison = Comparison.from_zpools(self, other) + zpool_comparison = ComparisonZpool.from_zpools(self, other) yield len(zpool_comparison), None @@ -185,25 +202,30 @@ def generate_backup_transactions( other, dataset_item ) + for transaction in self._get_backup_propery_transactions(other): + yield len(zpool_comparison) - 1, transaction + def _get_backup_transactions_from_datasetitem( - self, other: ZpoolABC, dataset_item: ComparisonItemABC, - ) -> typing.Union[None, typing.Generator[TransactionABC, None, None]]: + self, + other: ZpoolABC, + dataset_item: ComparisonItemABC, + ) -> TransactionListABC: - if dataset_item.get_item().subname in self._config["ignore"]: - return + if dataset_item.get_item().ignore: + return TransactionList() if dataset_item.a is None: - return + return TransactionList() if dataset_item.b is None: snapshots = list(dataset_item.a.snapshots) else: - dataset_comparison = Comparison.from_datasets( - dataset_item.a, dataset_item.b - ) - snapshots = dataset_comparison.a_head + dataset_comparison = ComparisonDataset.from_datasets( + dataset_item.a, dataset_item.b, self._config + ) # TODO namespace + snapshots = dataset_comparison.a_disjoint_head if len(snapshots) == 0: - return + return TransactionList() source_dataset = ( self.root @@ -216,10 +238,43 @@ def _get_backup_transactions_from_datasetitem( else join(other.root, dataset_item.a.subname) ) - return ( - snapshot.get_backup_transaction(source_dataset, target_dataset,) - for snapshot in snapshots - ) + transactions = TransactionList() + + for snapshot in snapshots: + transactions.extend( + snapshot.get_backup_transactions( + source_dataset, + target_dataset, + ) + ) + + return transactions + + def _get_backup_propery_transactions(self, other: ZpoolABC) -> TransactionListABC: + + transactions = TransactionList() + + if self._config["compatibility/target_samba_noshare"].value: + transactions.append( + Transaction.set_property( + item=other.root, + property=Property(name="sharesmb", value="off"), + side="target", + config=self._config, + ) + ) + + if self._config["compatibility/target_autosnapshot_ignore"].value: + transactions.append( + Transaction.set_property( + item=other.root, + property=Property(name="com.sun:auto-snapshot", value="false"), + side="target", + config=self._config, + ) + ) + + return transactions def get_snapshot_transactions(self) -> TransactionListABC: @@ -228,18 +283,13 @@ def get_snapshot_transactions(self) -> TransactionListABC: transactions = TransactionList() for dataset in self._datasets: - transaction = self._get_snapshot_transactions_from_dataset(dataset) - if transaction is None: - continue - transactions.append(dataset.get_snapshot_transaction()) + transactions.extend(self._get_snapshot_transactions_from_dataset(dataset)) return transactions def generate_snapshot_transactions( self, - ) -> typing.Generator[ - typing.Tuple[int, typing.Union[None, TransactionABC]], None, None - ]: + ) -> Generator[Tuple[int, Union[None, TransactionListABC]], None, None]: assert self._side == "source" @@ -250,30 +300,30 @@ def generate_snapshot_transactions( def _get_snapshot_transactions_from_dataset( self, dataset: DatasetABC - ) -> typing.Union[None, TransactionABC]: + ) -> TransactionListABC: - if dataset.subname in self._config["ignore"]: - return + if dataset.ignore: + return TransactionList() if ( dataset.get("mountpoint").value is None and dataset["type"].value == "filesystem" ): - return - if not dataset.changed: - return + return TransactionList() + if not dataset.changed: # TODO namespace + return TransactionList() - return dataset.get_snapshot_transaction() + return dataset.get_snapshot_transactions() def print_table(self): table = [] for dataset in self._datasets: - table.append(self._table_row(dataset)) + table.append(self._table_row(dataset, ignore=dataset.ignore)) for snapshot in dataset.snapshots: table.append(self._table_row(snapshot)) if len(table) == 0: - print('(empty)') + print("(empty)") return print( @@ -286,12 +336,16 @@ def print_table(self): ) @staticmethod - def _table_row(entity: typing.Union[SnapshotABC, DatasetABC]) -> typing.List[str]: + def _table_row( + entity: Union[SnapshotABC, DatasetABC], ignore: bool = False + ) -> List[str]: + + color = "white" if not ignore else "red" return [ "- " + colorize(entity.name, "grey") if isinstance(entity, SnapshotABC) - else colorize(entity.name, "white"), + else colorize(entity.name, color), humanize_size(entity["used"].value, add_color=True), humanize_size(entity["referenced"].value, add_color=True), f'{entity["compressratio"].value:.02f}', @@ -299,21 +353,33 @@ def _table_row(entity: typing.Union[SnapshotABC, DatasetABC]) -> typing.List[str def print_comparison_table(self, other: ZpoolABC): - zpool_comparison = Comparison.from_zpools(self, other) + zpool_comparison = ComparisonZpool.from_zpools(self, other) table = [] for dataset_item in zpool_comparison.merged: - table.append(self._comparison_table_row(dataset_item)) + table.append( + self._comparison_table_row( + dataset_item, ignore=dataset_item.get_item().ignore + ) + ) if dataset_item.complete: - dataset_comparison = Comparison.from_datasets( - dataset_item.a, dataset_item.b + dataset_comparison = ComparisonDataset.from_datasets( + dataset_item.a, dataset_item.b, self._config ) elif dataset_item.a is not None: - dataset_comparison = Comparison.from_datasets(dataset_item.a, None) + dataset_comparison = ComparisonDataset.from_datasets( + dataset_item.a, None, self._config + ) else: - dataset_comparison = Comparison.from_datasets(None, dataset_item.b) + dataset_comparison = ComparisonDataset.from_datasets( + None, dataset_item.b, self._config + ) for snapshot_item in dataset_comparison.merged: - table.append(self._comparison_table_row(snapshot_item)) + table.append( + self._comparison_table_row( + snapshot_item, ignore=dataset_item.get_item().ignore + ) + ) print( tabulate( @@ -324,66 +390,105 @@ def print_comparison_table(self, other: ZpoolABC): ) @staticmethod - def _comparison_table_row(item: ComparisonItemABC) -> typing.List[str]: + def _comparison_table_row( + item: ComparisonItemABC, ignore: bool = False + ) -> List[str]: + + color1, color2 = ("white", "grey") if not ignore else ("red", "yellow") + colorR, colorG, colorB = ( + ("red", "green", "blue") if not ignore else ("grey", "grey", "grey") + ) + + symbol = "X" entity = item.get_item() name = entity.name if isinstance(entity, SnapshotABC) else entity.subname if item.a is not None and item.b is not None: - a, b = colorize("X", "green"), colorize("X", "green") + a, b = colorize(symbol, colorG), colorize(symbol, colorG) elif item.a is None and item.b is not None: - a, b = "", colorize("X", "blue") + a, b = "", colorize(symbol, colorB) elif item.a is not None and item.b is None: - a, b = colorize("X", "red"), "" + a, b = colorize(symbol, colorR), "" return [ - "- " + colorize(name, "grey") + "- " + colorize(name, color2) if isinstance(entity, SnapshotABC) - else colorize(name, "white"), + else colorize(name, color1), a, b, ] @staticmethod - def available(side: str, config: ConfigABC,) -> int: - - output, _ = Command.on_side( - [ - "zfs", - "get", - "available", - "-H", - "-p", - root(config[side]["zpool"], config[side]["prefix"]), - ], - side, - config, - ).run() - - return Property.from_params(*output.strip().split("\t")[1:]).value + def available( + side: str, + config: ConfigABC, + ) -> int: + + output, _ = ( + Command.from_list( + [ + "zfs", + "get", + "available", + "-H", + "-p", + root( + config[f"{side:s}/zpool"].value, + config[f"{side:s}/prefix"].value, + ), + ] + ) + .on_side(side=side, config=config) + .run() + ) - @classmethod - def from_config(cls, side: str, config: ConfigABC,) -> ZpoolABC: + return Property.from_params(*output[0].strip().split("\t")[1:]).value - root_dataset = root(config[side]["zpool"], config[side]["prefix"]) + @classmethod + def from_config( + cls, + side: str, + config: ConfigABC, + ) -> ZpoolABC: + + root_dataset = root( + config[f"{side:s}/zpool"].value, config[f"{side:s}/prefix"].value + ) - output, errors, returncode, exception = Command.on_side( - ["zfs", "get", "all", "-r", "-H", "-p", root_dataset,], side, config, - ).run(returncode = True) + output, errors, returncode, exception = ( + Command.from_list( + [ + "zfs", + "get", + "all", + "-r", + "-H", + "-p", + root_dataset, + ] + ) + .on_side(side=side, config=config) + .run(returncode=True) + ) - if returncode != 0 and 'dataset does not exist' in errors: - return cls(datasets=[], side=side, config=config,) - if returncode != 0: + if returncode[0] != 0 and "dataset does not exist" in errors[0]: + return cls( + datasets=[], + side=side, + config=config, + ) + if returncode[0] != 0: raise exception output = [ - line.split("\t") for line in output.split("\n") if len(line.strip()) > 0 + line.split("\t") for line in output[0].split("\n") if len(line.strip()) > 0 ] entities = OrderedDict((line[0], []) for line in output) for line_list in output: entities[line_list[0]].append(line_list[1:]) - if not config.get("include_root", True): + if not config["include_root"].value: entities.pop(root_dataset) for name in [ snapshot @@ -408,4 +513,8 @@ def from_config(cls, side: str, config: ConfigABC,) -> ZpoolABC: ] datasets.sort(key=lambda dataset: dataset.name) - return cls(datasets=datasets, side=side, config=config,) + return cls( + datasets=datasets, + side=side, + config=config, + ) diff --git a/src/abgleich/gui/__init__.py b/src/abgleich/gui/__init__.py index d82915c..83c7d20 100644 --- a/src/abgleich/gui/__init__.py +++ b/src/abgleich/gui/__init__.py @@ -8,7 +8,7 @@ src/abgleich/gui/__init__.py: GUI package root - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/gui/abc.py b/src/abgleich/gui/abc.py index c73b29c..0b008b0 100644 --- a/src/abgleich/gui/abc.py +++ b/src/abgleich/gui/abc.py @@ -8,7 +8,7 @@ src/abgleich/gui/abc.py: Abstract base classes - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License diff --git a/src/abgleich/gui/lib.py b/src/abgleich/gui/lib.py index 6df61bc..c8ffb9f 100644 --- a/src/abgleich/gui/lib.py +++ b/src/abgleich/gui/lib.py @@ -8,7 +8,7 @@ src/abgleich/gui/lib.py: gui library - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,11 +28,11 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import typing +from typing import Type import sys from PyQt5.QtWidgets import QApplication, QDialog -import typeguard +from typeguard import typechecked from ..core.abc import ConfigABC @@ -41,8 +41,8 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked -def run_app(Window: typing.Type[QDialog], config: ConfigABC): +@typechecked +def run_app(Window: Type[QDialog], config: ConfigABC): app = QApplication(sys.argv) window = Window(config) diff --git a/src/abgleich/gui/transaction.py b/src/abgleich/gui/transaction.py index a94f1d9..62af09e 100644 --- a/src/abgleich/gui/transaction.py +++ b/src/abgleich/gui/transaction.py @@ -8,7 +8,7 @@ src/abgleich/gui/transaction.py: ZFS transactions - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -28,9 +28,9 @@ # IMPORT # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -import typing +from typing import Callable, Union -import typeguard +from typeguard import typechecked from PyQt5.QtCore import QAbstractTableModel, QModelIndex, Qt from PyQt5.QtGui import QColor @@ -43,11 +43,13 @@ # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -@typeguard.typechecked +@typechecked class TransactionListModel(QAbstractTableModel): - def __init__( - self, transactions: TransactionListABC, parent_changed: typing.Callable - ): + """ + Mutable. + """ + + def __init__(self, transactions: TransactionListABC, parent_changed: Callable): super().__init__() self._transactions = transactions @@ -59,7 +61,7 @@ def __init__( def data( self, index: QModelIndex, role: int - ) -> typing.Union[None, str, QColor]: # TODO return type + ) -> Union[None, str, QColor]: # TODO return type row, col = index.row(), index.column() col_key = self._cols[col] @@ -94,7 +96,7 @@ def data( def headerData( self, section: int, orientation: Qt.Orientation, role: int - ) -> typing.Union[None, str]: + ) -> Union[None, str]: if role == Qt.DisplayRole: @@ -112,7 +114,7 @@ def columnCount(self, index: QModelIndex) -> int: return len(self._cols) - def _transactions_changed(self, row: typing.Union[None, int] = None): + def _transactions_changed(self, row: Union[None, int] = None): old_rows, old_cols = self._rows, self._cols self._update_labels() diff --git a/src/abgleich/gui/wizard.py b/src/abgleich/gui/wizard.py index 5e446b6..7cfa014 100644 --- a/src/abgleich/gui/wizard.py +++ b/src/abgleich/gui/wizard.py @@ -8,7 +8,7 @@ src/abgleich/gui/wizard.py: wizard gui - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -37,7 +37,7 @@ from .transaction import TransactionListModel from .wizard_base import WizardUiBase from ..core.abc import ConfigABC -from ..core.transaction import TransactionList +from ..core.transactionlist import TransactionList from ..core.i18n import t from ..core.zpool import Zpool from .. import __version__ @@ -49,6 +49,11 @@ @typechecked class WizardUi(WizardUiBase): + """ + UI events and logic. + Mutable. + """ + def __init__(self, config: ConfigABC): super().__init__() @@ -199,7 +204,7 @@ def _finish_step(self, index: int): self._init_step(index + 1) - def _prepare_snap(self): + def _prepare_snap(self) -> bool: zpool = Zpool.from_config("source", config=self._config) @@ -210,14 +215,13 @@ def _prepare_snap(self): QApplication.processEvents() for number, transaction in gen: - if transaction is not None: - self._transactions.append(transaction) + self._transactions.extend(transaction) self._ui["progress"].setValue(number + 1) QApplication.processEvents() return len(self._transactions) > 0 - def _prepare(self, action: str): + def _prepare(self, action: str) -> bool: source_zpool = Zpool.from_config("source", config=self._config) target_zpool = Zpool.from_config("target", config=self._config) @@ -229,8 +233,7 @@ def _prepare(self, action: str): QApplication.processEvents() for number, transactions in gen: - if transactions is not None: - self._transactions.extend(transactions) + self._transactions.extend(transactions) self._ui["progress"].setValue(number + 1) QApplication.processEvents() diff --git a/src/abgleich/gui/wizard_base.py b/src/abgleich/gui/wizard_base.py index f15e1ab..1696199 100644 --- a/src/abgleich/gui/wizard_base.py +++ b/src/abgleich/gui/wizard_base.py @@ -8,7 +8,7 @@ src/abgleich/gui/wizard_base.py: wizard gui base - Copyright (C) 2019-2020 Sebastian M. Ernst + Copyright (C) 2019-2022 Sebastian M. Ernst The contents of this file are subject to the GNU Lesser General Public License @@ -46,6 +46,11 @@ @typechecked class WizardUiBase(QDialog): + """ + UI definition. + Mutable. + """ + def __init__(self): super().__init__() # skip WizardUiBaseABC