Skip to content

Commit

Permalink
split readme into blocks and services
Browse files Browse the repository at this point in the history
  • Loading branch information
Pierre Penninckx authored Nov 21, 2023
1 parent d928071 commit d6a325f
Showing 1 changed file with 155 additions and 119 deletions.
274 changes: 155 additions & 119 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,44 +3,33 @@
*Building blocks for self-hosting with battery included.*

SHB's (Self Host Blocks) goal is to provide a lower entry-bar for self-hosting. I intend to achieve
this by providing opinionated building blocks fitting together to self-host a wide range of
services. Also, the design will be extendable to allow users to add services not provided by SHB.
this by providing opinionated [building blocks](#building-blocks) fitting together to self-host any service
you'd want. Some [common services](#provided-services) are provided out of the box.

For each service, I intend to provide turn-key Nix options to setup:
The building blocks allow you to easily setup:
- Access through a subdomain ([Nginx](https://www.nginx.com/)).
- HTTPS access ([Nginx](https://www.nginx.com/) + [Letsencrypt](https://letsencrypt.org/)).
- Backup ([Borgmatic](https://torsion.org/borgmatic/) and/or [Restic](https://restic.net/)).
- Single sign-on ([Authelia](https://www.authelia.com/)).
- LDAP user management ([LLDAP](https://github.com/lldap/lldap)).
- Metrics, logs and alerting ([Grafana](https://grafana.com/) + [Prometheus](https://prometheus.io/) + [Loki](https://grafana.com/oss/loki/) + [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) + [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)).
- Database setup (Only [Postgresql](https://www.postgresql.org/) so far).
- VPN tunnels with optional proxys ([OpenVPN](https://openvpn.net/) with [Tinyproxy](http://tinyproxy.github.io/)).

The provided services will have all those integrated. Progress is detailed in the [Supported Features](#supported-features) section.

You should know that although I am using everything in this repo for my personal production server, this is
really just a one person effort for now and there are most certainly bugs that I didn't discover yet.

## TOC

<!--toc:start-->
- [Supported Features](#supported-features)
- [Usage](#usage)
- [Building Blocks](#building-blocks)
- [Demos](#demos)
- [Provided Services](#provided-services)
- [Add SSL configuration](#add-ssl-configuration)
- [Add LDAP and Authelia services](#add-ldap-and-authelia-services)
- [Deploy the full Grafana, Prometheus and Loki suite](#deploy-the-full-grafana-prometheus-and-loki-suite)
- [Deploy a Nextcloud Instance](#deploy-a-nextcloud-instance)
- [Enable verbose Nginx logging](#enable-verbose-nginx-logging)
- [Deploy an hledger Instance with LDAP and SSO support](#deploy-an-hledger-instance-with-ldap-and-sso-support)
- [Deploy a Jellyfin instance with LDAP and SSO support](#deploy-a-jellyfin-instance-with-ldap-and-sso-support)
- [Deploy a Home Assistant instance with LDAP support](#deploy-a-home-assistant-instance-with-ldap-support)
- [Set up network tunnel with VPN and Proxy](#set-up-network-tunnel-with-vpn-and-proxy)
- [Demos](#demos)
- [Import selfhostblocks](#import-selfhostblocks)
- [Tips](#tips)
- [Run tests](#run-tests)
- [Deploy using colmena](#deploy-using-colmena)
- [Use a local version of selfhostblocks](#use-a-local-version-of-selfhostblocks)
- [Diff changes](#diff-changes)
- [What is deployed](#what-is-deployed)
- [What will get deployed](#what-will-get-deployed)
- [Get the full diff](#get-the-full-diff)
- [Get version bumps](#get-version-bumps)
- [Generate random secret](#generate-random-secret)
- [TODOs](#todos)
- [Links that helped](#links-that-helped)
- [License](#license)
Expand Down Expand Up @@ -76,8 +65,9 @@ Currently supported services and features are:
- [X] LDAP auth, unfortunately we need to configure this manually.
- [ ] Declarative setup.
- [ ] SSO auth.
- [ ] Declarative setup.
- [X] Backup support.
- [x] Optional tracing debug.
- [X] Optional tracing debug.
- [ ] Export traces to Prometheus.
- [ ] Export metrics to Prometheus.
- [X] Home Assistant.
Expand Down Expand Up @@ -106,89 +96,40 @@ Currently supported services and features are:
- [ ] Gitea to deploy
- [ ] Scrutiny to monitor hard drives health
- [ ] Export metrics to Prometheus.
- [x] Tests
- [x] QoL
- [x] Unit tests for modules.
- [x] Running in CI.
- [ ] Integration tests with real nodes.

## Usage

The top-level `flake.nix` just outputs a nixos module that gathers all other modules from the [`modules/`](./modules/) directory.

Some provided modules are low-level and some are high-level that re-use those low-level ones. For
example, the nextcloud module re-uses the backup and nginx ones.

You want to use this repo as a flake input to your own repo. The `inputs` field of your `flake.nix`
file in your repo should look like so:

```nix
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
sops-nix.url = "github:Mic92/sops-nix";
selfhostblocks.url = "github:ibizaman/selfhostblocks";
selfhostblocks.inputs.nixpkgs.follows = "nixpkgs";
selfhostblocks.inputs.sops-nix.follows = "sops-nix";
};
```

`sops-nix` is used to setup passwords and secrets. Currently `selfhostblocks` has a strong
dependency on it but I'm working on removing that so you could use any secret provider.

The snippet above makes `selfhostblocks`' inputs follow yours. This is not maintainable though
because options that `selfhostblocks` rely on can change or disappear and you have no control on
that. Later, I intend to make `selfhostblocks` provide its own `nixpkgs` input and update it myself
through CI.

How you actually deploy using selfhostblocks depends on what system you choose. If you use
[colmena](https://colmena.cli.rs), this is what your `outputs` field could look like:

```nix
outputs = inputs@{ self, nixpkgs, ... }: {
colmena = {
meta = {
nixpkgs = import inputs.nixpkgs {
system = "x86_64-linux";
};
specialArgs = inputs;
};
myserver = import ./machines/myserver.nix;
};
}
```

Now, what goes inside this `./machines/myserver.nix` file? First, import `selfhostblocks` and
`sops-nix`:

```nix
imports = [
selfhostblocks.nixosModules.x86_64-linux.default
sops-nix.nixosModules.default
]
```

For how to configure the services, check the sections below.
- [ ] Self published documentation for options.
- [ ] Examples for all building blocks.

## Building Blocks

These building blocks are the reason selfhostblocks exist. You can pick and choose what building
blocks you need to self host a service of your choosing. Some services are already provided out of
the box by selfhostblocks but you might not want to use those if for example you want to integrate
with existing services or slowly transition to NixOS.
The building blocks are the foundation selfhostblocks intend to provide to allow you to self host
easily and with best practices any service of your choosing. Some services are already provided out of
the box but you might not want to use those if for example you want to integrate with existing
services or slowly transition to NixOS.

Following somewhat the Unix principle, each block has one goal and does it correctly. They also are
independent of each other, you can use only one or combine them to your liking.

Although these blocks provide options that encourage best practices, these are just NixOS modules that
configure other modules provided by nixpkgs. Would you need to make tweaks, you can always
access those underlying modules directly, like for any NixOS module.

## Demos
- [`authelia.nix`](./modules/blocks/authelia.nix) for Single Sign On.
- [`backup.nix`](./modules/blocks/backup.nix).
- [`ldap.nix`](./modules/blocks/ldap.nix) for user management.
- [`monitoring.nix`](./modules/blocks/monitoring.nix) for dashboards, logs and alerts.
- [`nginx.nix`](./modules/blocks/nginx.nix) for reverse proxy with SSL termination.
- [`postgresql.nix`](./modules/blocks/postgresql.nix) for database setup.
- [`ssl.nix`](./modules/blocks/ssl.nix) for maintaining SSL certificates provided by letsencrypt.
- [`tinyproxy.nix`](./modules/blocks/tinyproxy.nix) to forward traffic to a VPN tunnel.
- [`vpn.nix`](./modules/blocks/vpn.nix) to setup a VPN tunnel.

Demos that start and deploy on a Virtual Machine on your computer are located under the
[demo](./demo/) folder. These show the onboarding experience you would get if you deployed
selfhostblocks on your own server.

## Provided Services

I plan to have documentation for all options provided by selfhostblocks. For now,
I have a few examples:
The best way for now to understand how to use those modules is to read the code linked above and see
how they are used in the [provided services](#provided-services) and in the [demos](#demos). Also, here are a
few examples taken from my personal usage of selfhostblocks.

### Add SSL configuration

Expand Down Expand Up @@ -422,6 +363,69 @@ MaxFileSec=day
'';
```

### Set up network tunnel with VPN and Proxy

```nix
shb.vpn.nordvpnus = {
enable = true;
# Only "nordvpn" supported for now.
provider = "nordvpn";
dev = "tun1";
# Must be unique per VPN instance.
routingNumber = 10;
# Change to the one you want to connect to
remoteServerIP = "1.2.3.4";
sopsFile = ./secrets/vpn.yaml;
proxyPort = 12000;
};
```

This sets up a tunnel interface `tun1` that connects to the VPN provider, here NordVPN. Also, if the
`proxyPort` option is not null, this will spin up a `tinyproxy` instance that listens on the given
port and redirects all traffic through that VPN.

```bash
$ curl 'https://api.ipify.org?format=json'
{"ip":"107.21.107.115"}
$ curl --interface tun1 'https://api.ipify.org?format=json'
{"ip":"46.12.123.113"}
$ curl --proxy 127.0.0.1:12000 'https://api.ipify.org?format=json'
{"ip":"46.12.123.113"}
```

## Provided Services

- [`arr.nix`](./modules/services/arr.nix) for finding media https://wiki.servarr.com/.
- [`deluge.nix`](./modules/services/deluge.nix) for downloading linux isos https://deluge-torrent.org/.
- [`hledger.nix`](./modules/services/hledger.nix) for managing finances https://hledger.org/.
- [`home-assistant.nix`](./modules/services/home-assistant.nix) for private IoT https://www.home-assistant.io/.
- [`jellyfin.nix`](./modules/services/jellyfin.nix) for watching media https://jellyfin.org/.
- [`nextcloud-server.nix`](./modules/services/nextcloud-server.nix) for private documents, contacts, calendar, etc https://nextcloud.com.
- [`vaultwarden.nix`](./modules/services/vaultwarden.nix) for passwords https://github.com/dani-garcia/vaultwarden.

The services above are those I am using myself. I intend to add more.

The best way for now to understand how to use those modules is to read the code linked above and see
how they are used in the demos. Also, here are a
few examples taken from my personal usage of selfhostblocks.

### Common Options

Some common options are provided for all services.

- `enable` (bool). Set to true to deploy and run the service.
- `subdomain` (string). Subdomain under which to serve the service.
- `domain` (string). Domain under which to server the service.

Some other common options are the following. I am not satisfied with how those are expressed so those will most certainly change.
- LDAP and OIDC options for SSO, authentication and authorization.
- Secrets.
- Backups.

Note that for backups, every service exposes what directory should be backed up, you must merely choose when those backups will take place and where they will be stored.

### Deploy a Nextcloud Instance

```nix
Expand Down Expand Up @@ -607,38 +611,70 @@ home-assistant: |
longitude_home: "-0.01234567890123"
```

### Set up network tunnel with VPN and Proxy
## Demos

Demos that start and deploy a service on a Virtual Machine on your computer are located under the
[demo](./demo/) folder. These show the onboarding experience you would get if you deployed
one of the services on your own server.

## Import selfhostblocks

Ready to start using selfhostblocks? Thank you for trusting selfhostblocks. Please raise any
question you have or hurdle you encounter by creating an issue.

The top-level `flake.nix` just outputs a nixos module that gathers all other modules from
the [`modules/`](./modules/) directory. Use this repo as a flake input to your own repo.
The `inputs` field of your `flake.nix` file in your repo should look like so:

```nix
shb.vpn.nordvpnus = {
enable = true;
# Only "nordvpn" supported for now.
provider = "nordvpn";
dev = "tun1";
# Must be unique per VPN instance.
routingNumber = 10;
# Change to the one you want to connect to
remoteServerIP = "1.2.3.4";
sopsFile = ./secrets/vpn.yaml;
proxyPort = 12000;
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
sops-nix.url = "github:Mic92/sops-nix";
selfhostblocks.url = "github:ibizaman/selfhostblocks";
selfhostblocks.inputs.nixpkgs.follows = "nixpkgs";
selfhostblocks.inputs.sops-nix.follows = "sops-nix";
};
```

This sets up a tunnel interface `tun1` that connects to the VPN provider, here NordVPN. Also, if the
`proxyPort` option is not null, this will spin up a `tinyproxy` instance that listens on the given
port and redirects all traffic through that VPN.
`sops-nix` is used to setup passwords and secrets. Currently `selfhostblocks` has a strong
dependency on it but I'm working on removing that so you could use any secret provider.

```bash
$ curl 'https://api.ipify.org?format=json'
{"ip":"107.21.107.115"}
The snippet above makes `selfhostblocks`' inputs follow yours. This is not maintainable though
because options that `selfhostblocks` rely on can change or disappear and you have no control on
that. Later, I intend to make `selfhostblocks` provide its own `nixpkgs` input and update it myself
through CI.

$ curl --interface tun1 'https://api.ipify.org?format=json'
{"ip":"46.12.123.113"}
How you actually deploy using selfhostblocks depends on what system you choose. If you use
[colmena](https://colmena.cli.rs), this is what your `outputs` field could look like:

$ curl --proxy 127.0.0.1:12000 'https://api.ipify.org?format=json'
{"ip":"46.12.123.113"}
```nix
outputs = inputs@{ self, nixpkgs, ... }: {
colmena = {
meta = {
nixpkgs = import inputs.nixpkgs {
system = "x86_64-linux";
};
specialArgs = inputs;
};
myserver = import ./machines/myserver.nix;
};
}
```

Now, what goes inside this `./machines/myserver.nix` file? First, import `selfhostblocks` and
`sops-nix`:

```nix
imports = [
selfhostblocks.nixosModules.x86_64-linux.default
sops-nix.nixosModules.default
]
```

For the rest, see the

## Tips

### Run tests
Expand Down

0 comments on commit d6a325f

Please sign in to comment.