Release 0.20 docs (#1842)

* feat(configuration guide): local file watcher * docs: draft * Update apps.md * Update apps.md * Update apps.md * Update apps.md * Create custom-dashboards.md * docs continued * Update apps.md * docs: dashboards * Update apps.md * Update apps.md * Update apps.md * Update apps.md * docs: continued * Update apps.md * docs: system labels * Update apps.md * Update apps.md * docs: document storage per tenant kestra-io/kestra-ee#588 * Update 03.tenants.md * docs: sys labels * fix: app docs * fix: wording * [docs] add docker-composition edge case * [docs] add precision for local file name * [docs] add announcement documentation * [fix] typo * [docs] add bookmark doc * [docs] add bookmark screenshot * docs: add sla * docs: errorLogs() * Update apps.md * keep it DRY: keep all app code in the repo * fix apps docs * fix: dashboards * [docs] add flow trigger documentation * Update announcement.md * Update announcement.md * docs: invites * plural * fix: image paths * Update invitations.md * Update announcements.md * Update 09.bookmarks.md * fix(docs): manual pause to guide * Update content/docs/15.how-to-guides/local-file-sync.md Co-authored-by: Will Russell <[email protected]> --------- Co-authored-by: Will Russell <[email protected]> Co-authored-by: Anna Geller <[email protected]> Co-authored-by: Benoit <[email protected]>
kestra-io · Dec 3, 2024 · b009018 · b009018
1 parent aa4f9e6
commit b009018
Show file tree

Hide file tree

Showing 28 changed files with 904 additions and 91 deletions.
diff --git a/content/docs/04.workflow-components/01.flow.md b/content/docs/04.workflow-components/01.flow.md
@@ -29,7 +29,7 @@ Optionally, a flow can also have:
 - [pluginDefaults](./09.plugin-defaults.md)
 - [errors](./11.errors.md)
 - [retries](./12.retries.md)
-- [timeout](./13.timeout.md)
+- [sla](./18.sla.md)
 - [concurrency](./14.concurrency.md)
 - [descriptions](./15.descriptions.md)
 - [disabled](./16.disabled.md)

diff --git a/content/docs/04.workflow-components/07.triggers/02.flow-trigger.md b/content/docs/04.workflow-components/07.triggers/02.flow-trigger.md
@@ -11,141 +11,218 @@ Flow triggers allows you to trigger a flow after another flow execution, enablin
 type: "io.kestra.plugin.core.trigger.Flow"
 ```
 
-Kestra is able to trigger one flow after another one. This allows the chaining of flows without the need to update the base flows. With this capacity, you can break responsibility between different flows to different teams.
+Kestra is able to trigger a flow as soon as another flow ends. This allows to add implicit dependencies between multiple flows, which can often be managed by different teams.
 
 Check the [Flow trigger](/plugins/core/triggers/io.kestra.plugin.core.trigger.Flow) documentation for the list of all properties.
 
-## Conditions
+## Preconditions
 
-You can provide conditions to determine when your Flow should be executed. Along with the [core trigger conditions](./index.md#conditions), you can use the following:
+A flow trigger must have preconditions which filter on other flow executions on a time window.
 
-- [ExecutionFlowCondition](/plugins/core/conditions/io.kestra.plugin.core.condition.ExecutionFlowCondition)
-- [ExecutionNamespaceCondition](/plugins/core/conditions/io.kestra.plugin.core.condition.ExecutionNamespaceCondition)
-- [ExecutionLabelsCondition](/plugins/core/conditions/io.kestra.plugin.core.condition.executionlabelscondition)
-- [ExecutionStatusCondition](/plugins/core/conditions/io.kestra.plugin.core.condition.ExecutionStatusCondition)
-- [ExecutionOutputsCondition](/plugins/core/conditions/io.kestra.plugin.core.condition.executionoutputscondition)
-- [ExpressionCondition](/plugins/core/conditions/io.kestra.plugin.core.condition.ExpressionCondition)
 
-## Example
+### Filters
 
-This flow will be triggered after each successful execution of the flow `io.kestra.tests.trigger-flow` and forward the `uri` output of the `my-task` task.
-```yaml
-id: trigger_flow_listener
-namespace: company.team
 
-inputs:
-  - id: fromParent
-    type: STRING
+- `flows`
+A list of preconditions to met, in the form of upstream flows.
+
+For example, the following flow will trigger as soon as `flow_a` finish in SUCCESS state.
+
+```yaml
+id: flow_b
+namespace: kestra.sandbox
 
 tasks:
-  - id: onlyNoInput
-    type: io.kestra.plugin.core.debug.Return
-    format: "v1: {{ trigger.executionId }}"
+  - id: hello
+    type: io.kestra.plugin.core.log.Log
+    message: "Hello World!"
 
 triggers:
-  - id: listenFlow
+  - id: upstream_dependancy
     type: io.kestra.plugin.core.trigger.Flow
-    inputs:
-      fromParent: '{{ outputs.myTask.uri }}'
-    conditions:
-      - type: io.kestra.plugin.core.condition.ExecutionFlowCondition
-        namespace: company.team
-        flowId: trigger_flow
-      - type: io.kestra.plugin.core.condition.ExecutionStatusCondition
-        in:
-          - SUCCESS
+    preconditions:
+      id: flow_trigger
+      flows:
+        - namespace: kestra.sandbox
+          flowId: flow_a
+          states: [SUCCESS]
 ```
-Parent flow:
-```yaml
-id: trigger_flow
-namespace: company.team
 
-tasks:
-  - id: myTask
-    type: io.kestra.plugin.core.http.Download
-    uri: https://dummyjson.com/products
+- `where`
+
+You can also add execution filters on fields `FLOW_ID`, `NAMESPACE`, `STATE` and `EXPRESSION`.
+
+For example, the following Flow Trigger will trigger on for execution from flows in FAILED or WARNING state in namespaces starting with "company":
+
+```yaml
+triggers:
+  - id: alert_on_failure
+    type: io.kestra.plugin.core.trigger.Flow
+    states:
+      - FAILED
+      - WARNING
+    preconditions:
+      id: company_namespace
+      where:
+        - id: company
+          filters:
+            - field: NAMESPACE
+              type: STARTS_WITH
+              value: company
 ```
 
+### Time Window & SLA
+
+You can set the `timeWindow` property in different ways:
 
+`DURATION_WINDOW`: this is the default type. It uses a start time (windowAdvance) and end time (window) that are moving forward to the next interval whenever the evaluation time reaches the end time, based on the defined duration window. 
 
-This flow will be triggered after the successful execution of both flows `flow-a` and `flow-b` during the current day. When the conditions are met, the counter is reset and can be re-triggered during the same day. See [MultipleCondition](/plugins/core/conditions/io.kestra.plugin.core.condition.MultipleCondition) for more details
+For example, with a 1-day window (the default option: window: PT1D), the SLA conditions are always evaluated during 24h starting at midnight (i.e. at time 00:00:00) each day. If you set windowAdvance: PT6H, the window will start at 6 AM each day. If you set windowAdvance: PT6H and you also override the window property to PT6H, the window will start at 6 AM and last for 6 hours — as a result, Kestra will check the SLA conditions during the following time periods: 06:00 to 12:00, 12:00 to 18:00, 18:00 to 00:00, and 00:00 to 06:00, and so on.
+
+
+`SLIDING_WINDOW`: this option also evaluates SLA conditions over a fixed time window, but it always goes backward from the current time. For example, a sliding window of 1 hour (window: PT1H) will evaluate executions for the past hour (so between now and one hour before now). It uses a default window of 1 day.
+
+For example, the flow below will evaluate every hour if the flow `flow_a` is in SUCCESS state. If so it will trigger the `flow_b` passing corresponding inputs (reading `flow_a` outputs).
 
 ```yaml
-id: trigger-multiplecondition-listener
-namespace: company.team
+id: flow_b
+namespace: kestra.sandbox
+
+inputs:
+  - id: value_from_a
+    type: STRING
 
 tasks:
-  - id: onlyListener
-    type: io.kestra.plugin.core.debug.Return
-    format: "let's go "
+  - id: hello
+    type: io.kestra.plugin.core.log.Log
+    message: "{{ inputs.value_from_a }}"
 
 triggers:
-  - id: multipleListenFlow
+  - id: upstream_dep
     type: io.kestra.plugin.core.trigger.Flow
-    conditions:
-      - id: multiple
-        type: io.kestra.plugin.core.condition.MultipleCondition
-        window: P1D
-        windowAdvance: P0D
-        conditions:
-          flow-a:
-            type: io.kestra.plugin.core.condition.ExecutionFlowCondition
-            namespace: company.team
-            flowId: trigger-multiplecondition-flow-a
-          flow-b:
-            type: io.kestra.plugin.core.condition.ExecutionFlowCondition
-            namespace: company.team
-            flowId: trigger-multiplecondition-flow-b
+    inputs:
+      value_from_a: "{{ trigger.outputs.return_value }}"
+    preconditions:
+      id: test
+      flows:
+        - namespace: kestra.sandbox
+          flowId: flow_a
+          states: [SUCCESS]
+      timeWindow: 
+        type: SLIDING_WINDOW
+        window: PT1H
 ```
 
-Simply execute the two flows below to trigger `trigger-multiplecondition-listener`:
+Here is for reference the  `flow_a`:
+
 ```yaml
-id: trigger-multiplecondition-flow-a
-namespace: company.team
+id: flow_a
+namespace: kestra.sandbox
 
 tasks:
   - id: hello
     type: io.kestra.plugin.core.log.Log
-    message: Trigger A
+    message: Hello World! 🚀
+
+outputs:
+  - id: return_value
+    type: STRING
+    value: "Flow A run succesfully"
 ```
 
+
+`DAILY_TIME_DEADLINE`: this option declares that some SLA conditions should be met “before a specific time in a day”. With the string property deadline, you can configure a daily cutoff for checking conditions. For example, deadline: "09:00:00.00Z" means that the defined SLA conditions should be met from midnight until 9 AM each day; otherwise, the flow will not be triggered.
+
+For the example, this trigger definition will only trigger the flow if  `flow_a` is in SUCCESS state before 9:00 AM every day.
+
 ```yaml
-id: trigger-multiplecondition-flow-b
-namespace: company.team
+triggers:
+  - id: upstream_dep
+    type: io.kestra.plugin.core.trigger.Flow
+    preconditions:
+      id: should_be_success_by_nine
+      flows:
+        - namespace: kestra.sandbox
+          flowId: flow_a
+          states: [SUCCESS]
+      timeWindow: 
+        type: DAILY_TIME_DEADLINE
+        deadline: "09:00:00.00Z"
+```
+
+
+`DAILY_TIME_WINDOW`: this option declares that some SLA conditions should be met “within a given time range in a day”. For example, a window from startTime: "06:00:00" to endTime: "09:00:00" evaluates executions within that interval each day. This option is particularly useful for declarative definition of freshness conditions when building data pipelines. For example, if you only need one successful execution within a given time range to guarantee that some data has been successfully refreshed in order for you to proceed with the next steps of your pipeline, this option can be more useful than a strict DAG-based approach. Usually, each failure in your flow would block the entire pipeline, whereas with this option, you can proceed with the next steps of the pipeline as soon as the data is successfully refreshed at least once within the given time range.
+
+```yaml
+triggers:
+  - id: upstream_dep
+    type: io.kestra.plugin.core.trigger.Flow
+    inputs:
+      value_from_a: "{{ trigger.outputs.return_value }}"
+    preconditions:
+      id: test
+      flows:
+        - namespace: kestra.sandbox
+          flowId: flow_a
+          states: [SUCCESS]
+      timeWindow: 
+        type: DAILY_TIME_WINDOW
+        startTime: "06:00:00"
+        endTime: "12:00:00"
+```
+
+
+
+## Example
+
+Trigger the `silver_layer` flow once the `bronze_layer` flow finishes successfully by 9 AM. The deadline time string must include the timezone offset. This ensures that no new executions are triggered past the deadline. Here is the `silver_layer` flow:
 
+```yaml
+id: silver_layer
+namespace: company.team
 tasks:
-  - id: hello
+  - id: transform_data
     type: io.kestra.plugin.core.log.Log
-    message: Trigger B
+    message: deduplication, cleaning, and minor aggregations
+triggers:
+  - id: flow_trigger
+    type: io.kestra.plugin.core.trigger.Flow
+    preconditions:
+      id: bronze_layer
+      timeWindow:
+        type: DAILY_TIME_DEADLINE
+        deadline: "09:00:00+01:00"
+      flows:
+        - namespace: company.team
+          flowId: bronze_layer
+          states: [SUCCESS]
 ```
 
 ## Example: Alerting
 
-In this example, the Flow Trigger conditions are set to execute the flow when any workflow execution has a warning or failed status. We can configure this flow to send a notification to Slack (or any other platform) with information around the failure. Using this pattern, you can manage alerts on failure all in one place.
+Create a `System Flow` to send a Slack alert on any failure or warning state within the `company` namespace. This example uses the Slack webhook secret to notify the `#general` channel about the failed flow.
 
 ```yaml
-id: failure_alert_slack
+id: alert
 namespace: system
-
 tasks:
   - id: send_alert
     type: io.kestra.plugin.notifications.slack.SlackExecution
-    url: "{{ secret('SLACK_WEBHOOK') }}"
+    url: "{{secret('SLACK_WEBHOOK')}}" # format: https://hooks.slack.com/services/xzy/xyz/xyz
     channel: "#general"
-    executionId: "{{ trigger.executionId }}"
-
+    executionId: "{{trigger.executionId}}"
 triggers:
-  - id: on_failure
+  - id: alert_on_failure
     type: io.kestra.plugin.core.trigger.Flow
-    conditions:
-      - type: io.kestra.plugin.core.condition.ExecutionStatusCondition
-        in:
-          - FAILED
-          - WARNING
-```
-
-Check out the Blueprint [here](https://kestra.io/blueprints/failure-alert-slack).
-
-<div class="video-container">
-  <iframe src="https://www.youtube.com/embed/qwZUdtsngvQ?si=ss7AVZik8FeojdHk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
-</div>
+    states:
+      - FAILED
+      - WARNING
+    preconditions:
+      id: company_namespace
+      where:
+        - id: company
+          filters:
+            - field: NAMESPACE
+              type: STARTS_WITH
+              value: company
+```
diff --git a/content/docs/04.workflow-components/13.timeout.md b/content/docs/04.workflow-components/13.timeout.md
@@ -1,5 +1,5 @@
 ---
-title: Timeout
+title: Task timeout
 icon: /docs/icons/flow.svg
 ---
 
@@ -45,3 +45,7 @@ tasks:
       - sleep 10
     timeout: PT5S
 ```
+
+## Flow-level timeout
+
+There is no flow-level timeout. If you want to cancel a workflow execution if it exceeds a certain duration, you can leverage `MAX_DURATION`-type [SLA](./18.sla.md).