From 4e3c41f5ba7b01184bffa529708e5509eba7c85d Mon Sep 17 00:00:00 2001 From: strawberry Date: Sun, 15 Dec 2024 22:46:56 -0500 Subject: [PATCH] update some more docs Signed-off-by: strawberry --- debian/README.md | 2 ++ docs/deploying/docker.md | 3 ++ docs/deploying/generic.md | 43 ++++++++++++++++++++------- docs/deploying/nixos.md | 6 ++++ docs/maintenance.md | 61 ++++++++++++++++++++++++++++++--------- docs/troubleshooting.md | 10 ------- 6 files changed, 91 insertions(+), 34 deletions(-) diff --git a/debian/README.md b/debian/README.md index 893544696..800a2e090 100644 --- a/debian/README.md +++ b/debian/README.md @@ -9,6 +9,8 @@ It is recommended to see the [generic deployment guide](../deploying/generic.md) for further information if needed as usage of the Debian package is generally related. +No `apt` repository is currently offered yet, it is in the works/development. + ### Configuration When installed, the example config is placed at `/etc/conduwuit/conduwuit.toml` diff --git a/docs/deploying/docker.md b/docs/deploying/docker.md index d0aa13b37..bdbfb59c8 100644 --- a/docs/deploying/docker.md +++ b/docs/deploying/docker.md @@ -24,6 +24,9 @@ OCI images for conduwuit are available in the registries listed below. [shield-latest]: https://img.shields.io/docker/image-size/girlbossceo/conduwuit/latest [shield-main]: https://img.shields.io/docker/image-size/girlbossceo/conduwuit/main +OCI image `.tar.gz` files are also hosted directly at when uploaded by CI with a +commit hash/revision or a tagged release: + Use ```bash diff --git a/docs/deploying/generic.md b/docs/deploying/generic.md index 398ba67f6..cc50544e4 100644 --- a/docs/deploying/generic.md +++ b/docs/deploying/generic.md @@ -1,6 +1,6 @@ # Generic deployment documentation -> ## Getting help +> ### Getting help > > If you run into any problems while setting up conduwuit, ask us in > `#conduwuit:puppygock.gay` or [open an issue on @@ -8,29 +8,50 @@ ## Installing conduwuit -You may simply download the binary that fits your machine. Run `uname -m` to see -what you need. +### Static prebuilt binary + +You may simply download the binary that fits your machine architecture (x86_64 +or aarch64). Run `uname -m` to see what you need. Prebuilt fully static musl binaries can be downloaded from the latest tagged release [here](https://github.com/girlbossceo/conduwuit/releases/latest) or -`main` CI branch workflow artifact output. These also include Debian/Ubuntu packages. +`main` CI branch workflow artifact output. These also include Debian/Ubuntu +packages. + +Binaries are also available on my website directly at: + +These can be curl'd directly from. `ci-bins` are CI workflow binaries by commit +hash/revision, and `releases` are tagged releases. Sort by descending last +modified for the latest. These binaries have jemalloc and io_uring statically linked and included with them, so no additional dynamic dependencies need to be installed. +For the **best** performance; if using an `x86_64` CPU made in the last ~15 years, +we recommend using the `-haswell-` optimised binaries. This sets +`-march=haswell` which is the most compatible and highest performance with +optimised binaries. The database backend, RocksDB, most benefits from this as it +will then use hardware accelerated CRC32 hashing/checksumming which is critical +for performance. + +### Compiling + Alternatively, you may compile the binary yourself. We recommend using -Nix (or [Lix](https://lix.systems)) to build conduwuit as this has the most guaranteed -reproducibiltiy and easiest to get a build environment and output going. This also -allows easy cross-compilation. +Nix (or [Lix](https://lix.systems)) to build conduwuit as this has the most +guaranteed reproducibiltiy and easiest to get a build environment and output +going. This also allows easy cross-compilation. You can run the `nix build -L .#static-x86_64-linux-musl-all-features` or `nix build -L .#static-aarch64-linux-musl-all-features` commands based on architecture to cross-compile the necessary static binary located at -`result/bin/conduwuit`. This is reproducible with the static binaries produced in our CI. +`result/bin/conduwuit`. This is reproducible with the static binaries produced +in our CI. + +If wanting to build using standard Rust toolchains, make sure you install: +- `liburing-dev` on the compiling machine, and `liburing` on the target host +- LLVM and libclang for RocksDB -Otherwise, follow standard Rust project build guides (installing git and cloning -the repo, getting the Rust toolchain via rustup, installing LLVM toolchain + -libclang for RocksDB, installing liburing for io_uring and RocksDB, etc). +You can build conduwuit using `cargo build --release --all-features` ## Migrating from Conduit diff --git a/docs/deploying/nixos.md b/docs/deploying/nixos.md index 0372228df..3c5b0e699 100644 --- a/docs/deploying/nixos.md +++ b/docs/deploying/nixos.md @@ -29,6 +29,12 @@ conduit:Isq8FGyEC6FOXH6nD+BOeAA+bKp6X6UIbupSlGEPuOg= conduwuit:lYPVh7o1hLu1idH4Xt2QHaRa49WRGSAqzcfFd94aOTw= ``` +If needed, we have a binary cache on Cachix but it is only limited to 5GB: + +``` +https://conduwuit.cachix.org +conduwuit.cachix.org-1:MFRm6jcnfTf0jSAbmvLfhO3KBMt4px+1xaereWXp8Xg= +``` If specifying a Git remote URL in your flake, you can use any remotes that are specified on the README (the mirrors), such as the GitHub: `github:girlbossceo/conduwuit` diff --git a/docs/maintenance.md b/docs/maintenance.md index c8df95af7..378d8d4a9 100644 --- a/docs/maintenance.md +++ b/docs/maintenance.md @@ -22,23 +22,56 @@ conduwuit has moderation admin commands for: Any commands with `-list` in them will require a codeblock in the message with each object being newline delimited. An example of doing this is: -```` !admin rooms moderation ban-list-of-rooms ``` !roomid1:server.name -!roomid2:server.name !roomid3:server.name ``` ```` +```` +!admin rooms moderation ban-list-of-rooms +``` +!roomid1:server.name +!roomid2:server.name !roomid3:server.name +``` +```` -## Database +## Database (RocksDB) -If using RocksDB, there's very little you need to do. Compaction is ran -automatically based on various defined thresholds tuned for conduwuit to be high -performance with the least I/O amplifcation or overhead. Manually running -compaction is not recommended, or compaction via a timer. RocksDB is built with -io_uring support via liburing for async read I/O. +Generally there is very little you need to do. [Compaction][rocksdb-compaction] +is ran automatically based on various defined thresholds tuned for conduwuit to +be high performance with the least I/O amplifcation or overhead. Manually +running compaction is not recommended, or compaction via a timer, due to +creating unnecessary I/O amplification. RocksDB is built with io_uring support +via liburing for improved read performance. -Some RocksDB settings can be adjusted such as the compression method chosen. See -the RocksDB section in the [example config](configuration/examples.md). btrfs -users may benefit from disabling compression on RocksDB if CoW is in use. +RocksDB troubleshooting can be found [in the RocksDB section of troubleshooting](troubleshooting.md). + +### Compression -RocksDB troubleshooting can be found [in the RocksDB section of -troubleshooting](troubleshooting.md). +Some RocksDB settings can be adjusted such as the compression method chosen. See +the RocksDB section in the [example config](configuration/examples.md). + +btrfs users have reported that database compression does not need to be disabled +on conduwuit as the filesystem already does not attempt to compress. This can be +validated by using `filefrag -v` on a `.SST` file in your database, and ensure +the `physical_offset` matches (no filesystem compression). It is very important +to ensure no additional filesystem compression takes place as this can render +unbuffered Direct IO inoperable, significantly slowing down read and write +performance. See + +> Compression is done using the COW mechanism so it’s incompatible with +> nodatacow. Direct IO read works on compressed files but will fall back to +> buffered writes and leads to no compression even if force compression is set. +> Currently nodatasum and compression don’t work together. + +### Files in database + +Do not touch any of the files in the database directory. This must be said due +to users being mislead by the `.log` files in the RocksDB directory, thinking +they're server logs or database logs, however they are critical RocksDB files +related to WAL tracking. + +The only safe files that can be deleted are the `LOG` files (all caps). These +are the real RocksDB telemetry/log files, however conduwuit has already +configured to only store up to 3 RocksDB `LOG` files due to generall being +useless for average users unless troubleshooting something low-level. If you +would like to store nearly none at all, see the `rocksdb_max_log_files` +config option. ## Backups @@ -95,3 +128,5 @@ Built-in S3 support is also planned, but for now using a "S3 filesystem" on `media/` works. conduwuit also sends a `Cache-Control` header of 1 year and immutable for all media requests (download and thumbnail) to reduce unnecessary media requests from browsers, reduce bandwidth usage, and reduce load. + +[rocksdb-compaction]: https://github.com/facebook/rocksdb/wiki/Compaction diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 283cdeeed..d25c97621 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -91,16 +91,6 @@ reliability at a slight performance cost due to TCP overhead. ## RocksDB / database issues -#### Direct IO - -Some filesystems may not like RocksDB using [Direct -IO](https://github.com/facebook/rocksdb/wiki/Direct-IO). Direct IO is for -non-buffered I/O which improves conduwuit performance and reduces system CPU -usage, but at least FUSE and possibly ZFS are filesystems potentially known -to not like this. See the [example config](configuration/examples.md) for -disabling it if needed. Issues from Direct IO on unsupported filesystems are -usually shown as startup errors. - #### Database corruption If your database is corrupted *and* is failing to start (e.g. checksum