diff --git a/0.1/404.html b/0.1/404.html index c809a61..202ea2c 100644 --- a/0.1/404.html +++ b/0.1/404.html @@ -1 +1 @@ - Data Quality CLI for the Auto-Alerts

404 - Not found

\ No newline at end of file + Data Quality CLI for the Auto-Alerts

404 - Not found

\ No newline at end of file diff --git a/0.1/index.html b/0.1/index.html index 5adca6c..db7f0b5 100644 --- a/0.1/index.html +++ b/0.1/index.html @@ -1,4 +1,4 @@ - Quick Start - Data Quality CLI for the Auto-Alerts
Skip to content

diqu

PyPI version python-cli License: Apache 2.0 python codecov

CLI companion tool supporting the Alert / Notification for dq-tools and more.

Installation

pip install diqu --upgrade Successfully installed diqu restart ↻

📓 NOTE: The DWH module should get installed already if you use diqu in a dbt project, if not, please perform additional step, for example, to install snowflake module:

pip install "snowflake-connector-python[pandas]"
+ Quick Start - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
        
diqu
 
PyPI version python-cli License: Apache 2.0 python codecov
 
CLI companion tool supporting the Alert / Notification for dq-tools and more.
 
Installation
 
  pip install diqu --upgrade  Successfully installed diqu restart ↻ 
 
📓 NOTE: The DWH module should get installed already if you use diqu in a dbt project, if not, please perform additional step, for example, to install snowflake module:
 
pip install "snowflake-connector-python[pandas]"
 pip install "snowflake-connector-python[secure-local-storage]"
 
 
Concept
 
 
Alert Romance 🔴 🟡 ⚫ ✅
 
In the realm where circuits hum and wires entwine,
 
A bug’s life, oh, how it’s truly divine.
 
In the world of engineering, where chaos thrives,
 
Auto Alert, our savior, arrives.
 
 
diqu Concept 
 
In the efforts of making our bug's life easier, diqu CLI is born with a significant impact, streamlining collaboration and enhancing agility in our daily tasks. This tool takes charge of handling the beloved "chick" of our engineering world: Anomalies or Incidents 🐞
 
Alongside our methods for detecting these anomalies in our data, it's important to ensure the Auto Alert trigger is firmly in place, ready to notify us promptly. Together, these innovations pave the way for a more efficient and seamless bug-fighting experience 🚀
 
Usage
 
dbt run -s dq_tools # optional
 diqu alert --to jira
@@ -19,4 +19,4 @@
 export JIRA_INCIDENT_TICKET_TYPE=your_ticket_type, default to "[System] Incident"
 export JIRA_OPEN_TICKETS_FILTER=your_ticket_filter_on_title, default to "*dq_tools"
 diqu alert --to jira
-
 
How to Contribute
 
👉 See CONTRIBUTING.md
 
About Infinite Lambda
 
Infinite Lambda is a cloud and data consultancy. We build strategies, help organisations implement them and pass on the expertise to look after the infrastructure.
 
We are an Elite Snowflake Partner, a Platinum dbt Partner and two-times Fivetran Innovation Partner of the Year for EMEA.
 
Naturally, we love exploring innovative solutions and sharing knowledge, so go ahead and:
 
🔧 Take a look around our Git
 
✏️ Browse our tech blog
 
We are also chatty, so:
 
👀 Follow us on LinkedIn
 
👋🏼 Or just get in touch
 
About IL
  
 
 
  
  
 
 
 
         
\ No newline at end of file
+

How to Contribute

👉 See CONTRIBUTING.md

About Infinite Lambda

Infinite Lambda is a cloud and data consultancy. We build strategies, help organisations implement them and pass on the expertise to look after the infrastructure.

We are an Elite Snowflake Partner, a Platinum dbt Partner and two-times Fivetran Innovation Partner of the Year for EMEA.

Naturally, we love exploring innovative solutions and sharing knowledge, so go ahead and:

🔧 Take a look around our Git

✏️ Browse our tech blog

We are also chatty, so:

👀 Follow us on LinkedIn

👋🏼 Or just get in touch

About IL

\ No newline at end of file diff --git a/0.1/nav/dev/contributing.html b/0.1/nav/dev/contributing.html index 65586dd..89cbf8e 100644 --- a/0.1/nav/dev/contributing.html +++ b/0.1/nav/dev/contributing.html @@ -1,4 +1,4 @@ - Contribution Guideline - Data Quality CLI for the Auto-Alerts
Skip to content

Contributing to diqu

diqu is open source software. Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.

About this document

There are many ways to contribute to the ongoing development of diqu, such as by participating in discussions and issues.

The rest of this document serves as a more granular guide for contributing code changes to diqu (this repository). It is not intended as a guide for using diqu, and some pieces assume a level of familiarity with Python development with poetry. Specific code snippets in this guide assume you are using macOS or Linux and are comfortable with the command line.

  • Branches: All pull requests from community contributors should target the main branch (default). If the change is needed as a patch for a minor version of dbt that has already been released (or is already a release candidate), a maintainer will backport the changes in your PR to the relevant "latest" release branch (1.0.<latest>, 1.1.<latest>, ...). If an issue fix applies to a release branch, that fix should be first committed to the development branch and then to the release branch (rarely release-branch fixes may not apply to main).
  • Releases: Before releasing a new minor version, we prepare a series of beta release candidates to allow users to test the new version in live environments. This is an important quality assurance step, as it exposes the new code to a wide variety of complicated deployments and can surface bugs before official release. Releases are accessible via pip.

Getting the code

Installing git

You will need git in order to download and modify the diqu source code. On macOS, the best way to download git is to just install Xcode.

External contributors

You can contribute to diqu by forking the diqu repository. For a detailed overview on forking, check out the GitHub docs on forking. In short, you will need to:

  1. Fork the diqu repository
  2. Clone your fork locally
  3. Check out a new branch for your proposed changes
  4. Push changes to your fork
  5. Open a pull request against infintelambda/diqu from your forked repository

Setting up an environment

There are some tools that will be helpful to you in developing locally. While this is the list relevant for diqu development, many of these tools are used commonly across open-source python projects.

Tools

We will buy poetry in diqu development and testing.

So first install poetry via pip:

python3 -m pip install poetry --upgrade
+ Contribution Guideline - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Contributing to diqu
 
diqu is open source software. Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.
  
About this document
 
There are many ways to contribute to the ongoing development of diqu, such as by participating in discussions and issues.
 
The rest of this document serves as a more granular guide for contributing code changes to diqu (this repository). It is not intended as a guide for using diqu, and some pieces assume a level of familiarity with Python development with poetry. Specific code snippets in this guide assume you are using macOS or Linux and are comfortable with the command line.
 
     
  • Branches: All pull requests from community contributors should target the main branch (default). If the change is needed as a patch for a minor version of dbt that has already been released (or is already a release candidate), a maintainer will backport the changes in your PR to the relevant "latest" release branch (1.0.<latest>, 1.1.<latest>, ...). If an issue fix applies to a release branch, that fix should be first committed to the development branch and then to the release branch (rarely release-branch fixes may not apply to main).
    •  
  • Releases: Before releasing a new minor version, we prepare a series of beta release candidates to allow users to test the new version in live environments. This is an important quality assurance step, as it exposes the new code to a wide variety of complicated deployments and can surface bugs before official release. Releases are accessible via pip.
    •  
 
Getting the code
 
Installing git
 
You will need git in order to download and modify the diqu source code. On macOS, the best way to download git is to just install Xcode.
 
External contributors
 
You can contribute to diqu by forking the diqu repository. For a detailed overview on forking, check out the GitHub docs on forking. In short, you will need to:
 
     
  1. Fork the diqu repository
    2.  
  2. Clone your fork locally
    3.  
  3. Check out a new branch for your proposed changes
    4.  
  4. Push changes to your fork
    5.  
  5. Open a pull request against infintelambda/diqu from your forked repository
    6.  
 
Setting up an environment
 
There are some tools that will be helpful to you in developing locally. While this is the list relevant for diqu development, many of these tools are used commonly across open-source python projects.
 
Tools
 
We will buy poetry in diqu development and testing.
 
So first install poetry via pip:
 
python3 -m pip install poetry --upgrade
 
 
then, start installing the local environment:
 
python3 -m poetry install
 python3 -m poetry shell
 poe git-hooks
diff --git a/0.1/nav/guide/cli.html b/0.1/nav/guide/cli.html
index de35b82..2b81505 100644
--- a/0.1/nav/guide/cli.html
+++ b/0.1/nav/guide/cli.html
@@ -1,4 +1,4 @@
- CLI Reference - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
        
CLI Reference (diqu)
 
Run diqu --help or diqu -h to see the basic guideline for CLI Reference
 
 diqu -h Usage: diqu [OPTIONS] COMMAND [ARGS]...
 
 CLI companion tool to support dq-tools package and more
 
 Options:
 --version Show the version and exit.
 --help, -h Show this message and exit.
 
 Commands:
 alert Alert the incidents
 
 Specify one of these sub-commands and you can find more help from there.  
 
diqu alert
 
Alert the incidents to JIRA Board
 
Examples:
 
 
 
 
diqu alert
+ CLI Reference - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
        
CLI Reference (diqu)
 
Run diqu --help or diqu -h to see the basic guideline for CLI Reference
 
 diqu -h Usage: diqu [OPTIONS] COMMAND [ARGS]...
 
 CLI companion tool to support dq-tools package and more
 
 Options:
 --version Show the version and exit.
 --help, -h Show this message and exit.
 
 Commands:
 alert Alert the incidents
 
 Specify one of these sub-commands and you can find more help from there.  
 
diqu alert
 
Alert the incidents to JIRA Board
 
Examples:
 
 
 
 
diqu alert
 
 
 
 
diqu alert --project-dir /path/to/dbt
 
 
 
 
 
Send alerts to other channels
 
 
 
 
diqu alert --to <your channel module>
 
 
 
 
 
Current supported channels could be found in (repo)/diqu/alerts/:
 
     
  • JIRA: default
    •  
  • Slack: diqu alert --to slack
    •  
 
Customize the alert's query
 
 
 
 
diqu alert \
diff --git a/0.1/nav/guide/common.html b/0.1/nav/guide/common.html
index 94af914..ed03563 100644
--- a/0.1/nav/guide/common.html
+++ b/0.1/nav/guide/common.html
@@ -1 +1 @@
- User Guide - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
User Guide
 
diqu CLI is built in the modular approach with 3 main one:
 
     
  1. Alert module: Define how to alert **_/ notify the Incidents _**to which channels e.g. Jira, Slack
    2.  
  2. Source module: Build the Data Source connection e.g. Snowflake, Csv file
    3.  
  3. Package module: Manage the Query of the Incidents captured
    4.  
 
👉 See the next pages for more information of how to configure the modules
  
 
 
  
  
 
 
 
         
\ No newline at end of file
+ General - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
User Guide
 
diqu CLI is built in the modular approach with 3 main one:
 
     
  1. Alert module: Define how to alert **_/ notify the Incidents _**to which channels e.g. Jira, Slack
    2.  
  2. Source module: Build the Data Source connection e.g. Snowflake, Csv file
    3.  
  3. Package module: Manage the Query of the Incidents captured
    4.  
 
👉 See the next pages for more information of how to configure the modules
  
 
 
  
  
 
 
 
         
\ No newline at end of file
diff --git a/0.1/nav/guide/config/alerts/jira.html b/0.1/nav/guide/config/alerts/jira.html
index 9b631bc..5421281 100644
--- a/0.1/nav/guide/config/alerts/jira.html
+++ b/0.1/nav/guide/config/alerts/jira.html
@@ -1 +1 @@
- JIRA - Data Quality CLI for the Auto-Alerts      
 
 
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
JIRA
  
 
 
  
  
 
 
 
         
\ No newline at end of file
+ JIRA - Data Quality CLI for the Auto-Alerts      
 
 
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
JIRA
  
 
 
  
  
 
 
 
         
\ No newline at end of file
diff --git a/0.1/nav/guide/config/alerts/slack.html b/0.1/nav/guide/config/alerts/slack.html
index 159d22f..936235c 100644
--- a/0.1/nav/guide/config/alerts/slack.html
+++ b/0.1/nav/guide/config/alerts/slack.html
@@ -1 +1 @@
- Slack - Data Quality CLI for the Auto-Alerts      
 
 
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Slack
  
 
 
  
  
 
 
 
         
\ No newline at end of file
+ Slack - Data Quality CLI for the Auto-Alerts      
 
 
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Slack
  
 
 
  
  
 
 
 
         
\ No newline at end of file
diff --git a/0.1/nav/guide/config/packages/custom.html b/0.1/nav/guide/config/packages/custom.html
index 40e2f6e..5baf1b9 100644
--- a/0.1/nav/guide/config/packages/custom.html
+++ b/0.1/nav/guide/config/packages/custom.html
@@ -1,4 +1,4 @@
- Custom query - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Configuration for Custom query
 
If you're not using dq-tools package or even not using dbt, no problem, we're supporting a custom query directly to your data table/view.
 
The steps are as following:
 
1. Prepare SQL script
 
Assuming we have a custom script named incidents.sql which is located in the current directory.
 
The script need to provide the expected columns required for each channels, for example, for JIRA, we need:
 
     
  • Title
    •  
  • Description
    •  
  • Label
    •  
  • ...(to be updated)
    •  
 
Let's build your SELECT query:
 
-- incident.sql
+ Custom query - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Configuration for Custom query
 
If you're not using dq-tools package or even not using dbt, no problem, we're supporting a custom query directly to your data table/view.
 
The steps are as following:
 
1. Prepare SQL script
 
Assuming we have a custom script named incidents.sql which is located in the current directory.
 
The script need to provide the expected columns required for each channels, for example, for JIRA, we need:
 
     
  • Title
    •  
  • Description
    •  
  • Label
    •  
  • ...(to be updated)
    •  
 
Let's build your SELECT query:
 
-- incident.sql
 
 select  'your_value' as title,
         ...
diff --git a/0.1/nav/guide/config/packages/dq-tools.html b/0.1/nav/guide/config/packages/dq-tools.html
index bbfe96a..1a233fe 100644
--- a/0.1/nav/guide/config/packages/dq-tools.html
+++ b/0.1/nav/guide/config/packages/dq-tools.html
@@ -1,4 +1,4 @@
- DQ Tools - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Configuration for dq-tools dbt package
 
This is the essential steps to start alerting the Incidents based on the test results which are captured by dq-tools package in a dbt project.
 
1. Install dq-tools package
 
dbt version required: >=1.6.0
 
Include the following in your packages.yml file:
 
packages:
+ DQ Tools - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Configuration for dq-tools dbt package
 
This is the essential steps to start alerting the Incidents based on the test results which are captured by dq-tools package in a dbt project.
 
1. Install dq-tools package
 
dbt version required: >=1.6.0
 
Include the following in your packages.yml file:
 
packages:
   - package: infinitelambda/dq_tools
     version: 1.4.2
 
 
Run dbt deps to install the package.
 
For more information on using packages in your dbt project, check out the dbt Documentation.
 
2. Configure the log table & the hook
 
Log table is containing all the test results produced by dbt Jobs, and surely can be configured by specifying the database or/and the schema. By default, these info will be getting from the dbt profiles.yml.
 
And the hook is to save the test result if any.
 
In dbt_project.yml file:
 
vars:
diff --git a/0.1/nav/guide/config/sources/custom.html b/0.1/nav/guide/config/sources/custom.html
index 95b6d61..5a97436 100644
--- a/0.1/nav/guide/config/sources/custom.html
+++ b/0.1/nav/guide/config/sources/custom.html
@@ -1,4 +1,4 @@
- Custom via CSV file - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Configuration for Other connection via CSV file
 
Currently, the only supported connection is Snowflake, we'll see more added in the near further.
 
In the meantime, using CSV file is a good option alternatively.
 
1. Configure the dbt profile
 
Let's create a new target for CSV in dbt profiles.yml file:
 
ci:
+ Custom via CSV file - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Configuration for Other connection via CSV file
 
Currently, the only supported connection is Snowflake, we'll see more added in the near further.
 
In the meantime, using CSV file is a good option alternatively.
 
1. Configure the dbt profile
 
Let's create a new target for CSV in dbt profiles.yml file:
 
ci:
   target: dev
   outputs:
     dev:
diff --git a/0.1/nav/guide/config/sources/snowflake.html b/0.1/nav/guide/config/sources/snowflake.html
index e186df4..313f654 100644
--- a/0.1/nav/guide/config/sources/snowflake.html
+++ b/0.1/nav/guide/config/sources/snowflake.html
@@ -1,4 +1,4 @@
- Snowflake - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Configuration for Snowflake connection
 
In order to use Snowflake as the data source, we need to get the additional dependencies after installing diqu:
 
pip install "snowflake-connector-python[pandas]"
+ Snowflake - Data Quality CLI for the Auto-Alerts       
 
  
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
       
Configuration for Snowflake connection
 
In order to use Snowflake as the data source, we need to get the additional dependencies after installing diqu:
 
pip install "snowflake-connector-python[pandas]"
 pip install "snowflake-connector-python[secure-local-storage]"
 
 
diqu will try to reuse dbt profile configuration with supporting 2 authentication methods:
 
User / Password authentication
 
# ~/.dbt/profiles.yml
 my-snowflake-db:
diff --git a/0.1/sitemap.xml.gz b/0.1/sitemap.xml.gz
index e0e6762..b586474 100644
Binary files a/0.1/sitemap.xml.gz and b/0.1/sitemap.xml.gz differ