Change docs, changelog generation and mkdocs container (et/somenergia…

…-jardiner!112)

* add edit and view raw features to mkdocs

* fix bad references between articles in docs

* change mkdocs container workflow and add plugins

Some plugins for mkdocs needed extra dependencies, and this commit simplifies the process based on learnings from mercat.

The new image does not need the app builder image.

The mkdocs image uses python 3.11 so pyproject.toml now is scoped to <3.12 to include this fact.

* Change information about using ORMs as admonition

* fix typo in docs

* add plant reader to documentation index

* change documentation structure and format a little

this reorder directories, update links and sometimes format a little more.

* Change project version to calver and formalize changelog generation

This updates README.md and creates a CONTRIBUTING.md and updates CHANGELOG.md file while stablishing a changelog generation workflow with `make changelog`

.editorconfig

@@ -19,3 +19,7 @@ indent_size = 2
 
 [*.md]
 trim_trailing_whitespace = true
+
+[Makefile]
+indent_style = tab
+

.git-changelog.toml

@@ -0,0 +1,13 @@
+bump = "auto"
+convention = "basic"
+in-place = false
+# filter-commits = "0.5.0.."
+# marker-line = "<!-- insertion marker -->"
+parse-refs = true
+parse-trailers = true
+repository = "."
+sections = ["add","remove","fix","doc"]
+template = "keepachangelog"
+version-regex = "^## \\\\[(?P<version>v?[^\\\\]]+)"
+provider = "gitlab"
+zerover = true

CONTRIBUTING.md

@@ -0,0 +1,69 @@
+# Contributing to this project
+
+This project is a data pipeline that takes data from the Som Energia plants and processes it to be used in the BI tools. This documentation serves as an onboarding guide for new developers and as a reference for the current ones.
+
+## Using `make`
+
+Check the `Makefile` for the available commands. You can use `make` to run the tests, serve the documentation, and more. Use `make help` to see the available commands.
+
+## Project versioning (experimental)
+
+Since jardiner is not a library but rather a complex system, we version the project using the `git` tags. We use the (`calendar versioning`)[https://calver.org/] to tag the releases. We also use the `CHANGELOG.md` to keep track of the changes along with `git-changelog` to generate the changelog.
+
+Git changelog is a tool that generates a changelog from git tags and the commit messages. We _try_ to follow the [Basic convention](https://pawamoy.github.io/git-changelog/usage/#basic-convention) format.
+
+A way to tag a release is to use the following commands:
+
+```bash
+$ git tag $(date +'%Y.%m.%d') -m "$(date +'%Y.%m.%d')"
+$ make changelog > CHANGELOG.md
+```
+
+## Install dependencies
+
+We use `poetry` to manage the dependencies. You can install it via pipx or the official installer. Check the poetry documentation for more.
+
+In short,
+
+- `poetry add <package>` to add package
+- `poetry install` to install `poetry.lock` packages
+- `poetry show --tree` will show the dependencies tree.
+- Additionally `deptry .` will analyze the project and find inconsistencies between project and dependencies.
+
+### Known issues
+
+If you get wheel errors on manylinux2014, update your `pip` to solve it. `poetry` doesn't fetch wheels from manylinux2014. `orjson` will cause this issue with pip 20 for example.
+
+## Testing
+
+Run `pytest` on the root directory. Also, you can test the data models with `dbt test --target pre --target-dir dbt_jardiner`
+
+## Run
+
+`typer` will tell you what arguments you need to run the notify_alarms script. You will have to provide a dbapi string or the placeholders `prod` or `pre` which will read `.env.prod` and `.env.pre` respectivelly
+
+```bash
+python ./jardiner/notify_alarms --help
+```
+
+## Deploy
+
+We deploy using local gitlab runners as part of the CI/CD pipeline. Check the `.gitlab-ci.yml` file for more information.
+
+This project features many docker images hosted in our private registry at <https://harbor.somenergia.coop>. Check the `docker-compose.yml` files for more information.
+
+## Update requirements
+
+We use poetry to maintain the requirements, but we can update the requirements like so:
+
+```bash
+poetry export --without=dev --without-hashes --format=requirements.txt > requirements.txt
+```
+
+## Update documentation
+
+We use mkdocs to serve extra documentation and ADRs (Architecture Decision Records). You can serve the documentation with the following command
+
+```bash
+make mkdocs.serve
+```

Makefile

@@ -6,7 +6,6 @@ app_compose_env_file := ./.env
 local_airflow_compose_file := containers/airflow-local/docker-compose.airflow-local.yml
 local_airflow_compose_env_file := containers/airflow-local/.airflow-local.env
 mkdocs_compose_file := containers/mkdocs/docker-compose.mkdocs.yml
-mkdocs_compose_env_file := containers/mkdocs/.mkdocs.env
 
 # taken from https://container-solutions.com/tagging-docker-images-the-right-way/
 
@@ -46,19 +45,19 @@ mkdocs.requirements.txt: ## update requirements.txt file from pyproject.toml
 	@echo "poetry-mkdocs-requirements.txt file updated"
 
 mkdocs.serve: ## serve the mkdocs documentation
-	@docker compose -f $(mkdocs_compose_file) --env-file $(mkdocs_compose_env_file) up
+	@docker compose --env-file $(mkdocs_compose_env_file) up
 
 mkdocs.build_image: ## build the mkdocs image
-	@docker compose -f $(mkdocs_compose_file) --env-file $(mkdocs_compose_env_file) build mkdocs
+	@docker compose --env-file $(mkdocs_compose_env_file) build mkdocs
 
 mkdocs.push_image: ## push the mkdocs image with tag: latest
-	@docker compose -f $(mkdocs_compose_file) --env-file $(mkdocs_compose_env_file) push mkdocs
+	@docker compose --env-file $(mkdocs_compose_env_file) push mkdocs
 
 mkdocs.build_docs: ## build the mkdocs documentation
-	@docker compose -f $(mkdocs_compose_file) --env-file $(mkdocs_compose_env_file) run --rm mkdocs build
+	@docker compose --env-file $(mkdocs_compose_env_file) run --rm mkdocs build
 
 mkdocs.logs: ## show the logs of the mkdocs container
-	@docker compose -f $(mkdocs_compose_file) --env-file $(mkdocs_compose_env_file) logs -ft mkdocs
+	@docker compose --env-file $(mkdocs_compose_env_file) logs -ft mkdocs
 
 # ---------------------------------------------------------------------------- #
 #                               dbt-docs commands                              #
@@ -86,4 +85,5 @@ dbt_docs.logs: ## show the logs of the dbt-docs container
 local.re_data_models.dev: ## Run re_data models
 	@(cd dbt_jardiner && dbt run --target dev --models package:re_data)
 
-
+changelog: ## generate changelog
+	@docker compose -f $(app_compose_file) --env-file $(app_compose_env_file) run --rm -it --entrypoint git-changelog app-dev

README.md

@@ -6,61 +6,10 @@ Mending and nurturing of green energy plants. See its changelog at [CHANGELOG.md
 
 - [Mkdocs documentation](https://et.pages.somenergia.coop/somenergia-jardiner/docs/)
 - [dbt documentation](https://et.pages.somenergia.coop/somenergia-jardiner/dbt_docs/)
-- [Elementary documentation](https://et.pages.somenergia.coop/somenergia-jardiner/elementary_docs/)
+- [Elementary documentation](https://jardiner-elementary.moll.somenergia.coop/)
 
 ## Usuàries de Gestió d'actius
 
-Gestió d'actius! Aneu [aquí](/somenergia-jardiner/projecte/2023-06-03-macrofase roadmap/)
+- Gestió d'actius! Aneu [aquí](/somenergia-jardiner/projecte/2023-06-03-macrofase roadmap/)
+- SomEnergia Devs! Aneu [aquí](/somenergia-jardiner/desenvolupadors/2023-06-10-overview-devs/)
 
-SomEnergia Devs! Aneu [aquí](/somenergia-jardiner/desenvolupadors/2023-06-10-overview-devs/)
-
-## install dependencies
-
-The appropriate way to install poetry is in its own isolated environment
-
-```bash
-curl -sSL https://install.python-poetry.org | python3 -
-```
-
-Otherwise [poetry will delete itself](https://github.com/python-poetry/poetry/issues/3957) when running `poetry install --sync`
-
-alternativelly you can do
-
-```bash
-pip install -r requirements.txt
-```
-
-Check the poetry documentation for more, shortlist `poetry add <package>` to add package, `poetry install` to install `poetry.lock` packages
-and `poetry show --tree` will show the dependencies tree. Additionally `deptry .` will analyze the project and find inconsistencies between project and dependencies.
-
-If you get wheel errors on manylinux2014, update your `pip` to solve it. `poetry` doesn't fetch wheels from manylinux2014. `orjson` will cause this issue with pip 20 for example.
-
-## testing
-
-just run `pytest` on the root directory
-
-also you can test the data models with `dbt test --target pre --target-dir dbt_jardiner`
-
-## run
-
-`typer` will tell you what arguments you need to run the notify_alarms script. You will have to provide a dbapi string or the placeholders `prod` or `pre` which will read `.env.prod` and `.env.pre` respectivelly
-
-`python ./jardiner/notify_alarms --help`
-
-## deploy
-
-deploy is in continuos delivery on main branch, but dbt models must be manually ran at the moment.
-
-`dbt run --target prod --target-dir dbt_jardiner`
-
-## update requirements
-
-We use poetry to maintain the requirements, but we can update the requirements like so:
-
-`poetry export --without=dev --without-hashes --format=requirements.txt > requirements.txt`
-
-## update documentation
-
-We use mkdocs to serve extra documentation and adrs
-
-`mkdocs serve`

containers/mkdocs/Dockerfile

@@ -1,10 +1,5 @@
-ARG WORKDIR="/app"
-
-FROM harbor.somenergia.coop/dades/somenergia-jardiner-builder:latest as builder
-ARG WORKDIR
-RUN poetry export -f requirements.txt --output ${WORKDIR}/mkdocs-requirements.txt --without-hashes
-
 FROM squidfunk/mkdocs-material:latest
-ARG WORKDIR
-COPY --from=builder ${WORKDIR}/mkdocs-requirements.txt .
-RUN pip install --no-cache-dir -r mkdocs-requirements.txt
+
+# Install dependencies
+COPY poetry-mkdocs-requirements.txt .
+RUN pip install --no-cache-dir -r poetry-mkdocs-requirements.txt

containers/mkdocs/README.md

@@ -1,3 +1,9 @@
 # About
 
-Deploy a mkdocs site to a container.
+Deploy a mkdocs site to a container.
+
+Dependencies are managed by poetry. At the root level of the repo, run:
+
+```bash
+make mkdocs.requirements.txt
+```

containers/mkdocs/docker-compose.mkdocs.yml

@@ -2,7 +2,7 @@
 version: "3.7"
 services:
   mkdocs:
-    image: ${SOMENERGIA_REGISTRY}/${SOMENERGIA_PROJECT_NAME?Variable not set}-mkdocs:${SOMENERGIA_DOCKER_TAG?Variable not set}
+    image: harbor.somenergia.coop/dades/somenergia-jardiner-mkdocs:${SOMENERGIA_DOCKER_TAG-latest}
     build:
       context: ../../
       dockerfile: containers/mkdocs/Dockerfile
@@ -13,4 +13,4 @@ services:
       - ../../mkdocs.yml:/docs/mkdocs.yml
       - ../../docs:/docs/docs
       - ./README.md:/docs/README.md
-      - ../../git:/docs/.git:ro # needed for plugins
+      - ../../.git:/docs/.git:ro # needed for plugins

docs/adr/2022-10-11-dbt_workflow_meter_alarms.md

@@ -81,4 +81,4 @@ Chosen option: triem l'opció 2. per suportar calcular alarmes en el passat
 ### Negative Consequences
 
 * Requereix materialitzar una taula amb la última lectura processada (o tenir un staging on s'inserten les noves)
-* Incrementa la complexitat de la query SQL
+* Incrementa la complexitat de la query SQL

docs/adr/2023-06-13-pipeline_dades_llargues.md

@@ -10,7 +10,7 @@ https://trello.com/c/rmCTOJrH/205-decidir-com-farem-lobt-i-el-cincminutal-r%C3%A
 
 Passem a una macrotaula de dades llargues clau-valor per a les dades cincminutals?
 
-Com serà la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem la discussió de la obt a [un adr diferent](/somenergia-jardiner/adr/2023-06-13-obt_plantes)
+Com serà la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem la discussió de la obt a [un adr diferent](../adr/2023-06-13-obt_plantes.md)
 
 ## Context and Problem Statement
 
@@ -31,7 +31,7 @@ Ara que sabem molt més dbt i que incorporarem un proveïdor de dades cincminuta
 ## Decision Outcome
 
 Chosen option: opció 3
-
+<!--
 ### Positive Consequences <!-- optional -->
 
 * [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
@@ -74,4 +74,4 @@ Chosen option: opció 3
 ## Links <!-- optional -->
 
 * [Link type] [Link to ADR] <!-- example: Refined by [ADR-0005](0005-example.md) -->
-* … <!-- numbers of links can vary -->
+* … numbers of links can vary -->

docs/desenvolupadors/deprecated/TODOs.md → docs/deprecated/TODOs.md

@@ -37,7 +37,6 @@ sum(a/x) + sum(b/y) ... mensual wrong
 
 sum((a+b+c)/(x+y+z)) -- mensual correct
 
-
 # views per limitar l'accés a la info dels tècnics
 
 No cal fer-ho, limitarem l'accés a la wide table o farem servir els permisos del superset o ja veurem
@@ -68,4 +67,4 @@ view_sensorirradiationregistry_ercam.sql
 ```
 zero_inverter_power_at_daylight.sql
 zero_sonda_irradiation_at_daylight.sql
-```
+```

docs/desenvolupadors/tutorials/novu_notifications.md → docs/desenvolupadors/2023-02-08-novu_notifications.md

@@ -1,3 +1,7 @@
+title: Gestionar notificacions a Novu
+description: Com interactuar amb el sistema de notificacions Novu
+date: 2023-07-10
+
 # Novu helpers and quickstart
 
 ## Swagger API

docs/desenvolupadors/2023-06-10-overview-devs.md

@@ -1,13 +1,20 @@
+title: Overview for devs
+description: Overview of the project for developers
+date: 2023-06-10
+tags: dbt, airflow, plantmonitor, plantreader, jardiner
+
 # Overview for devs
 
 ## quickstart
 
-To fully deploy the dbt models you can do
+To fully deploy the dbt models to the production schema you can do
 
 ```bash
 dbt build -s tag:jardiner,tag:legacy --store-failures --full-refresh --target prod
 ```
 
+You will need to have the `prod` profile in your `~/.dbt/profiles.yml` file.
+
 ## Current State of the project
 
 ### Pieces
@@ -18,7 +25,9 @@ dbt build -s tag:jardiner,tag:legacy --store-failures --full-refresh --target pr
 
 Fa d'ORM de les plantes, conté la api i és codi de les raspberrypis. A més a més també té el codi d'ingesta d'apis de tercers.
 
-La meitat del plantmonitor acabarà deprecated pel proveïdor de dades.
+!!! warning
+
+    La meitat del plantmonitor acabarà deprecated pel proveïdor de dades
 
 [Plant Reader](https://github.com/Som-Energia/somenergia-plant-reader)
 
@@ -92,6 +101,6 @@ union -- notify.py --> alerts
 obt --> datasets
 ```
 
-Després de l'[ADR d'estructura de pipeline](/somenergia-jardiner/adr/2023-06-13-pipeline_dades_llargues)
+Després de l'[ADR d'estructura de pipeline](../adr/2023-06-13-pipeline_dades_llargues.md)
 
-See [Roadmap](/somenergia-jardiner/projecte/2023-06-03-macrofase roadmap/)
+See [Roadmap](""../projecte/2023-06-03-macrofase roadmap.md")

docs/desenvolupadors/tutorials/dbt_cmds_utils.md → docs/desenvolupadors/2023-07-10-dbt_cmds_utils.md

@@ -1,11 +1,15 @@
+title: Comandes de dbt comunes
+description: Comandes de dbt que es fan servir sovint
+date: 2023-07-10
+
 # dbt commands
 
 1. Generar i servir la documentació per a poder veure el directed graph.
 
 ```bash
-dbt docs generate --project-dir dbt_jardiner --target pre
+$ dbt docs generate --project-dir dbt_jardiner --target pre
 
-dbt docs serve --project-dir dbt_jardiner --target pre --port 8010
+$ dbt docs serve --project-dir dbt_jardiner --target pre --port 8010
 ```
 
 2. Pujar tots els CSVs de seeds
@@ -23,4 +27,4 @@ Per fer testing amb un csv determinat:
 
 ```bash
 dbt run --project-dir project_x --target testing --vars '{"test_sample":"nom_del_csv"}' -m model_x
-```
+```

docs/desenvolupadors/tutorials/codi_reciclable.md → docs/desenvolupadors/2023-07-13-codi_reciclable.md

@@ -1,3 +1,7 @@
+title: SQL reutilitzable
+description: SQLs que es poden reutilitzar en diferents contexts
+date: 2023-07-10
+
 # Reusuable sqls
 
 ## dbt sql for rolling window