Skip to content

Commit

Permalink
add docs
Browse files Browse the repository at this point in the history
  • Loading branch information
timkpaine committed Aug 24, 2024
1 parent bf3678a commit 349b9ce
Show file tree
Hide file tree
Showing 2 changed files with 57 additions and 0 deletions.
57 changes: 57 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,63 @@ High Availability (HA) DAG Utility

## Overview

This library provides an operator called `HighAvailabilityOperator`, which inherits from `PythonSensor` and does the following:

- runs a user-provided `python_callable` as a sensor
- if this returns `"done"`, mark the DAG as passed and finish
- if this returns `"running"`, keep checking
- if this returns `"failed"`, mark the DAG as failed and re-run
- if the sensor times out, mark the DAG as passed and re-run

Consider the following DAG:

```python
from datetime import datetime, timedelta
from random import choice

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow_ha import HighAvailabilityOperator


with DAG(
dag_id="test-high-availability",
description="Test HA Operator",
schedule=timedelta(days=1),
start_date=datetime(2024, 1, 1),
catchup=False,
):
ha = HighAvailabilityOperator(
task_id="ha",
timeout=30,
poke_interval=5,
python_callable=lambda **kwargs: choice(("done", "failed", "running", ""))
)

pre = PythonOperator(task_id="pre", python_callable=lambda **kwargs: "test")
pre >> ha

fail = PythonOperator(task_id="fail", python_callable=lambda **kwargs: "test")
ha.failed >> fail

passed = PythonOperator(task_id="passed", python_callable=lambda **kwargs: "test")
ha.passed >> passed

done = PythonOperator(task_id="done", python_callable=lambda **kwargs: "test")
ha.done >> done
```

This produces a DAG with the following topology:

<img src="./docs/src/top.png" />

This DAG exhibits cool behavior.
If a check fails or the interval elapses, the DAG will re-trigger itself.
If the check passes, the DAG will finish.
This allows the one to build "always-on" DAGs without having individual long blocking tasks.

This library is used to build [airflow-supervisor](https://github.com/airflow-laminar/airflow-supervisor), which uses [supervisor](http://supervisord.org) as a process-monitor while checking and restarting jobs via `airflow-ha`.

## License

This software is licensed under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.
Binary file added docs/src/top.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 349b9ce

Please sign in to comment.