Skip to content

Commit

Permalink
fix: update mkdocs specific syntax, reconcile some headers contents
Browse files Browse the repository at this point in the history
  • Loading branch information
IL-Minh committed Nov 14, 2023
1 parent fcd7e1f commit 33d347e
Show file tree
Hide file tree
Showing 5 changed files with 61 additions and 42 deletions.
22 changes: 13 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,28 +9,32 @@

Automate and streamline the notification process for dbt test results using this versatile CLI companion tool. Receive detailed alerts seamlessly on various platforms, promoting improved collaboration on dbt project issues 🐞🚀.

Features
## Features

- Automated alerts and notifications based on recorded dbt test results.
- Built-in support for dq-tools and custom query input.
- Auto-labels `deprecated` tests for quick & easy identification.
- Sends compact and succinct messages to a dedicated Slack channel.
- Sends succinct and informative messages to a dedicated Slack channel.
- Creates and updates Jira tickets with the latest tests' metadata.

Supported input modules (dbt test results):
### Supported input modules (dbt test results):

- [![dq-tools](https://img.shields.io/badge/dq--tools-hub-FF694B?logo=dbt&logoColor=FF694B)](https://hub.getdbt.com/infinitelambda/dq_tools)
- Custom query

Supported output modules (alert/ notification):
- CSV
### Supported output modules (alert/ notification):

- Jira
- Slack
- CSV

### Supported [![dq-tools](https://img.shields.io/badge/dq--tools-hub-FF694B?logo=dbt&logoColor=FF694B)](https://hub.getdbt.com/infinitelambda/dq_tools) metadata

Supported [![dq-tools](https://img.shields.io/badge/dq--tools-hub-FF694B?logo=dbt&logoColor=FF694B)](https://hub.getdbt.com/infinitelambda/dq_tools) metadata:
- Latest & historical tests' statuses, timestamps, row failed counts, row scanned counts, and failed rate.
- Latest & historical tests' statuses, timestamps, row failed counts, row scanned counts and failed rate.
- `deprecated` status for tests that are not executed and recorded in x days.
- Tests labels (dq_issue_type, kpi_category)

> dbt alert rant 🟢 🟡 🔴 ⚫ </br>
> dbt alert rant 🟢 🟡 🔴 ⚫ - by `diqu` OG Contributors </br>
>_Run errors are red,_</br>
>_Test warnings are yellow,_</br>
>_Where's my bug alert,_</br>
Expand All @@ -45,7 +49,7 @@ pip install diqu [--upgrade]
```

📓 _NOTE_: The Data Warehouse (DWH) module should already be installed if you are using `diqu` in a working dbt project. If not, please perform additional steps to install these DWH modules.
For example, if you're using snowflake:
For example, if you're using Snowflake:

```bash
pip install "snowflake-connector-python[pandas]"
Expand Down
22 changes: 13 additions & 9 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,28 @@

Automate and streamline the notification process for dbt test results using this versatile CLI companion tool. Receive detailed alerts seamlessly on various platforms, promoting improved collaboration on dbt project issues 🐞🚀.

Features
## Features

- Automated alerts and notifications based on recorded dbt test results.
- Built-in support for dq-tools and custom query input.
- Auto-labels `deprecated` tests for quick & easy identification.
- Sends compact and succinct messages to a dedicated Slack channel.
- Sends succinct and informative messages to a dedicated Slack channel.
- Creates and updates Jira tickets with the latest tests' metadata.

Supported input modules (dbt test results):
### Supported input modules (dbt test results):

- [![dq-tools](https://img.shields.io/badge/dq--tools-hub-FF694B?logo=dbt&logoColor=FF694B)](https://hub.getdbt.com/infinitelambda/dq_tools)
- Custom query

Supported output modules (alert/ notification):
- CSV
### Supported output modules (alert/ notification):

- Jira
- Slack
- CSV

### Supported [![dq-tools](https://img.shields.io/badge/dq--tools-hub-FF694B?logo=dbt&logoColor=FF694B)](https://hub.getdbt.com/infinitelambda/dq_tools) metadata

Supported [![dq-tools](https://img.shields.io/badge/dq--tools-hub-FF694B?logo=dbt&logoColor=FF694B)](https://hub.getdbt.com/infinitelambda/dq_tools) metadata:
- Latest & historical tests' statuses, timestamps, row failed counts, row scanned counts, and failed rate.
- Latest & historical tests' statuses, timestamps, row failed counts, row scanned counts and failed rate.
- `deprecated` status for tests that are not executed and recorded in x days.
- Tests labels (dq_issue_type, kpi_category)

Expand All @@ -41,7 +45,7 @@ Supported [![dq-tools](https://img.shields.io/badge/dq--tools-hub-FF694B?logo=db
</div>

📓 _NOTE_: The Data Warehouse (DWH) module should already be installed if you are using `diqu` in a working dbt project. If not, please perform additional steps to install these DWH modules.
For example, if you're using snowflake:
For example, if you're using Snowflake:

```bash
pip install "snowflake-connector-python[pandas]"
Expand All @@ -50,7 +54,7 @@ pip install "snowflake-connector-python[secure-local-storage]"

## Concept

!!! quote "dbt alert rant 🟢 🟡 🔴 ⚫"
!!! quote "dbt alert rant 🟢 🟡 🔴 ⚫ - by `diqu` OG Contributors"
_Run errors are red,_

_Test warnings are yellow,_
Expand Down
40 changes: 22 additions & 18 deletions docs/nav/guide/config/packages/custom.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,22 @@ The steps are as follows:

Assuming we have a custom script named `incidents.sql` which is located in the current directory.

The script needs to provide the expected columns required for all modules below:

- `test_id`:
- `test_status`: status of the test: 'pass', 'warn', 'fail', 'deprecated'
- `test_title`: test's title to be shown in your module (e.g in the Jira module, `test_title` is used as issue's summary: '🟡 | Warning in test: accepted_values_my_first_dbt_model_id__False__1__2.ee252c12b8 [dq-tools]')
- `check_timestamp`: test execution timestamp
- `no_of_records_scanned`: number of rows scanned
- `no_of_records_failed`: number of rows that did not pass the test
- `failed_rate`: percentage of no_of_records_failed / no_of_records_scanned (e.g `no_of_records_failed / nullif(no_of_records_scanned, 0) as failed_rate`)
- `tag_1`: your 1st test tag (e.g 'accepted_value')
- `tag_2`: your 2nd test tag (e.g `Accuracy)
- `prev_statuses`: an array of previous statuses (e.g `array_agg(test_status_emoji) within group (order by check_timestamp desc) as prev_statuses`)
- `prev_check_timestamps`: an array of previous execution times of this test (e.g `array_agg(check_timestamp) within group (order by check_timestamp desc) as prev_check_timestamps`)
- `prev_no_of_records_scanned`: an array of previous row scanned (e.g `array_agg(no_of_records_scanned) within group (order by check_timestamp desc) as prev_no_of_records_scanned`)
- `prev_no_of_records_failed`: an array of previous row failed (e.g `array_agg(no_of_records_failed) within group (order by check_timestamp desc) as prev_no_of_records_failed`)
- `priority`: your priority level for each test. This field will be used in 'order by' clause in all modules. It is recommended to use numbers to simplify the process.
The script has to provide the expected columns required for all modules below:

- `test_id`: the unique identifier of the test.
- `test_status`: status of the test: 'pass', 'warn', 'fail', 'deprecated'. Example: `accepted_values_my_first_dbt_model_id__False__1__2.ee252c12b8`
- `test_title`: test's title to be shown in your module (e.g in the Jira module, `test_title` is used as the issue's summary: `🟡 | Warning in test: test_id [dq-tools]`).
- `check_timestamp`: test execution timestamp.
- `no_of_records_scanned`: number of rows scanned.
- `no_of_records_failed`: number of rows that did not pass the test.
- `failed_rate`: percentage of no_of_records_failed / no_of_records_scanned (e.g `no_of_records_failed / nullif(no_of_records_scanned, 0) as failed_rate`).
- `tag_1`: your 1st test tag (e.g 'accepted_value').
- `tag_2`: your 2nd test tag (e.g `Accuracy).
- `prev_statuses`: an array of previous statuses (e.g `array_agg(test_status_emoji) within group (order by check_timestamp desc) as prev_statuses`).
- `prev_check_timestamps`: an array of previous execution times of this test (e.g `array_agg(check_timestamp) within group (order by check_timestamp desc) as prev_check_timestamps`).
- `prev_no_of_records_scanned`: an array of previous row scanned (e.g `array_agg(no_of_records_scanned) within group (order by check_timestamp desc) as prev_no_of_records_scanned`).
- `prev_no_of_records_failed`: an array of previous row failed (e.g `array_agg(no_of_records_failed) within group (order by check_timestamp desc) as prev_no_of_records_failed`).
- `priority`: your priority level for each test. This field will be used for ordering the issues list in all modules. It is recommended to use numbers to simplify the process.

Let's build your `SELECT` query:

Expand All @@ -37,11 +37,15 @@ from your_table
```

## 2. Alerting
Check out the modules' configuration for specific env vars setup. Example:

See [JIRA Configuration](https://diqu.iflambda.com/latest/nav/guide/config/alerts/jira.html) for more details, following is a sample command:
- [JIRA Configuration](https://diqu.iflambda.com/latest/nav/guide/config/alerts/jira.html)
- [SLACK Configuration](https://diqu.iflambda.com/latest/nav/guide/config/alerts/slack.html)

Sample command:

```bash
# prepare the env vars here first
# prepare the env vars first
...
# run alerting
diqu alert --query-file incident.sql
Expand Down
11 changes: 7 additions & 4 deletions docs/nav/guide/config/packages/dq-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,20 +42,23 @@ By default, it is `False`, therefore let's enable it to have data flew in.

```bash
# init the dq-tools' models
dbt run -s dq_tools
dbt run -s dq_tools
# build your dbt models with saving the test results
dbt build --vars '{dq_tools_enable_store_test_results: true}'
```

## 4. Alerting

See [JIRA Configuration](https://diqu.iflambda.com/latest/nav/guide/config/alerts/jira.html) for more details, following is a sample command:
Check out the modules' configuration for specific env vars setup. Example:

- [JIRA Configuration](https://diqu.iflambda.com/latest/nav/guide/config/alerts/jira.html)
- [SLACK Configuration](https://diqu.iflambda.com/latest/nav/guide/config/alerts/slack.html)

```bash
# prepare the env vars here first
# prepare the env vars first
...
# run alerting
diqu alert --query-schema AUDIT
```

> We need to use `--query-schema` here because we previously configure the `dbt_dq_tool_schema` variable
> `--query-schema` is required here because we previously configured the `dbt_dq_tool_schema` variable
8 changes: 6 additions & 2 deletions docs/nav/guide/config/usage_stats.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
# Anonymous usage stats

We aim to enhance the functionality of diqu by gaining insights into user interactions. As part of this effort, we have implemented straightforward event tracking within diqu, utilizing Snowplow.

It is important to note that we do not collect credentials or any other type of private info, as we consider this information outside the scope of our interest.
The potential uses for usage statistics encompass package maintenance & future module development
By default, event tracking is enabled. However, users of diqu can choose to opt out of this feature at any time by setting the environment variable `DO_NOT_TRACK` = true (default = false)

The potential uses for usage statistics encompass package maintenance & future module development.

By default, event tracking is enabled. However, users of diqu can choose to opt out of this feature at any time by setting the following environment variable

```bash
export DO_NOT_TRACK=true
```

0 comments on commit 33d347e

Please sign in to comment.