230
-231
-232
-233
-234
-235
-236
-237
-238
-239
-240
-241
-242
-243
-244
-245
-246
-247
-248
-249
-250
-251
-252
-253
-254
+ | def add_tasks(
- self,
- tasks: (
- OrbiterOperator
- | OrbiterTaskGroup
- | Iterable[OrbiterOperator | OrbiterTaskGroup]
- ),
-) -> "OrbiterDAG":
- """
- Add one or more [`OrbiterOperators`][orbiter.objects.task.OrbiterOperator] to the DAG
-
- ```pycon
- >>> from orbiter.objects.operators.empty import OrbiterEmptyOperator
- >>> OrbiterDAG(file_path="", dag_id="foo").add_tasks(OrbiterEmptyOperator(task_id="bar")).tasks
- {'bar': bar_task = EmptyOperator(task_id='bar')}
-
- >>> OrbiterDAG(file_path="", dag_id="foo").add_tasks([OrbiterEmptyOperator(task_id="bar")]).tasks
- {'bar': bar_task = EmptyOperator(task_id='bar')}
-
- ```
-
- !!! tip
-
- Validation requires a `OrbiterTaskGroup`, `OrbiterOperator` (or subclass), or list of either to be passed
- ```pycon
- >>> # noinspection PyTypeChecker
- ... OrbiterDAG(file_path="", dag_id="foo").add_tasks("bar")
- ... # doctest: +IGNORE_EXCEPTION_DETAIL
- Traceback (most recent call last):
- pydantic_core._pydantic_core.ValidationError: ...
- >>> # noinspection PyTypeChecker
- ... OrbiterDAG(file_path="", dag_id="foo").add_tasks(["bar"])
- ... # doctest: +IGNORE_EXCEPTION_DETAIL
- Traceback (most recent call last):
- pydantic_core._pydantic_core.ValidationError: ...
-
- ```
- :param tasks: List of [OrbiterOperator][orbiter.objects.task.OrbiterOperator], or OrbiterTaskGroup or subclass
- :type tasks: OrbiterOperator | OrbiterTaskGroup | Iterable[OrbiterOperator | OrbiterTaskGroup]
- :return: self
- :rtype: OrbiterProject
- """
- if (
- isinstance(tasks, OrbiterOperator)
- or isinstance(tasks, OrbiterTaskGroup)
- or issubclass(type(tasks), OrbiterOperator)
- ):
- tasks = [tasks]
-
- for task in tasks:
- try:
- task_id = getattr(task, "task_id", None) or getattr(
- task, "task_group_id"
- )
- except AttributeError:
- raise AttributeError(
- f"Task {task} does not have a task_id or task_group_id attribute"
- )
- self.tasks[task_id] = task
- return self
+289
+290
+291
+292
+293
+294
+295
+296
+297
+298
+299
+300
+301
+302
+303
+304
+305
+306
+307
+308
+309
+310
+311
+312
+313
| def add_tasks(
+ self,
+ tasks: (
+ OrbiterOperator
+ | OrbiterTaskGroup
+ | Iterable[OrbiterOperator | OrbiterTaskGroup]
+ ),
+) -> "OrbiterDAG":
+ """
+ Add one or more [`OrbiterOperators`][orbiter.objects.task.OrbiterOperator] to the DAG
+
+ ```pycon
+ >>> from orbiter.objects.operators.empty import OrbiterEmptyOperator
+ >>> OrbiterDAG(file_path="", dag_id="foo").add_tasks(OrbiterEmptyOperator(task_id="bar")).tasks
+ {'bar': bar_task = EmptyOperator(task_id='bar')}
+
+ >>> OrbiterDAG(file_path="", dag_id="foo").add_tasks([OrbiterEmptyOperator(task_id="bar")]).tasks
+ {'bar': bar_task = EmptyOperator(task_id='bar')}
+
+ ```
+
+ !!! tip
+
+ Validation requires a `OrbiterTaskGroup`, `OrbiterOperator` (or subclass), or list of either to be passed
+ ```pycon
+ >>> # noinspection PyTypeChecker
+ ... OrbiterDAG(file_path="", dag_id="foo").add_tasks("bar")
+ ... # doctest: +IGNORE_EXCEPTION_DETAIL
+ Traceback (most recent call last):
+ pydantic_core._pydantic_core.ValidationError: ...
+ >>> # noinspection PyTypeChecker
+ ... OrbiterDAG(file_path="", dag_id="foo").add_tasks(["bar"])
+ ... # doctest: +IGNORE_EXCEPTION_DETAIL
+ Traceback (most recent call last):
+ pydantic_core._pydantic_core.ValidationError: ...
+
+ ```
+ :param tasks: List of [OrbiterOperator][orbiter.objects.task.OrbiterOperator], or OrbiterTaskGroup or subclass
+ :type tasks: OrbiterOperator | OrbiterTaskGroup | Iterable[OrbiterOperator | OrbiterTaskGroup]
+ :return: self
+ :rtype: OrbiterProject
+ """
+ if (
+ isinstance(tasks, OrbiterOperator)
+ or isinstance(tasks, OrbiterTaskGroup)
+ or issubclass(type(tasks), OrbiterOperator)
+ ):
+ tasks = [tasks]
+
+ for task in tasks:
+ try:
+ task_id = getattr(task, "task_id", None) or getattr(
+ task, "task_group_id"
+ )
+ except AttributeError:
+ raise AttributeError(
+ f"Task {task} does not have a task_id or task_group_id attribute"
+ )
+ self.tasks[task_id] = task
+ return self
|
Supported Origins for services to create translations
-create an issue in our orbiter-community-translations repository
-write a TranslationRuleset and submit a pull request to share your translations with the community
+create an issue in our orbiter-community-translations repository
+write a TranslationRuleset Template and submit a pull request to share your translations with the community
diff --git a/search/search_index.json b/search/search_index.json
index 275012b..2db8f2b 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":" Astronomer Orbiter can land legacy workloads safely down in a new home on Apache Airflow! "},{"location":"#what-is-orbiter","title":"What is Orbiter?","text":"Orbiter is both a CLI and Framework for converting workflows from other orchestration tools to Apache Airflow. Generally it can be thoughts of as: flowchart LR\n origin{{ XML/JSON/YAML/Etc Workflows }}\n origin -->| \u2728 Translations \u2728 | airflow{{ Apache Airflow Project }}
The framework is a set of Rules and Objects that can translate workflows from an Origin system to an Airflow project."},{"location":"#installation","title":"Installation","text":"Install the orbiter CLI, if you have Python >= 3.10 installed via pip: pip install astronomer-orbiter\n
If you do not have a compatible Python environment, pre-built binary executables of the orbiter CLI are available for download on the Releases page."},{"location":"#translate","title":"Translate","text":"Utilize the orbiter CLI with existing translations to convert workflows from other systems to an Airflow project. - Set up a new folder, and create a
workflow/ folder. Add your workflows files to it .\n\u2514\u2500\u2500 workflow/\n \u251c\u2500\u2500 workflow_a.json\n \u251c\u2500\u2500 workflow_b.json\n \u2514\u2500\u2500 ...\n
- Determine the specific translation ruleset via:
- the Origins documentation
- the
orbiter list-rulesets command - or by creating a translation ruleset, if one does not exist
- Install the translation ruleset via the
orbiter install command (substituting <REPOSITORY> with the value in the last step) orbiter install --repo=<REPOSITORY>\n
- Use the
orbiter translate command with the <RULESET> determined in the last step This will produce output to an output/ folder: orbiter translate workflow/ --ruleset <RULESET> output/\n
- Review the contents of the
output/ folder. If extensions or customizations are required, review how to extend a translation ruleset - (optional) Utilize the
astro CLI to run Airflow instance with your migrated workloads - (optional) Deploy to Astro to run your translated workflows in production! \ud83d\ude80
"},{"location":"#authoring-rulesets-customization","title":"Authoring Rulesets & Customization","text":"Orbiter can be extended to fit specific needs, patterns, or to support additional origins. Read more specifics about how to use the framework at Rules and Objects "},{"location":"#extend-or-customize","title":"Extend or Customize","text":"To extend or customize an existing ruleset, you can easily modify it with simple Python code. - Set up your workspace as described in steps 1+2 of the Translate instructions
- Create a Python script, named
override.py .\n\u251c\u2500\u2500 override.py\n\u2514\u2500\u2500 workflow/\n \u251c\u2500\u2500 workflow_a.json\n \u251c\u2500\u2500 workflow_b.json\n \u2514\u2500\u2500 ...\n
-
Add contents to override.py: override.py from orbiter_community_translations.dag_factory import translation_ruleset # (1)!\nfrom orbiter.objects.operators.ssh import OrbiterSSHOperator # (2)!\nfrom orbiter.rules import task_rule # (3)!\n\n\n@task_rule(priority=99) # (4)!\ndef ssh_rule(val: dict):\n \"\"\"Demonstration of overriding rulesets, by switching DAG Factory BashOperators to SSHOperators\"\"\"\n if val.pop(\"operator\", \"\") == \"BashOperator\": # (5)!\n return OrbiterSSHOperator( # (6)!\n command=val.pop(\"bash_command\"),\n doc=\"Hello World!\",\n **{k: v for k, v in val if k != \"dependencies\"},\n )\n else:\n return None\n\n\ntranslation_ruleset.task_ruleset.ruleset.append(ssh_rule) # (7)!\n
- Importing specific translation ruleset, determined via the Origins page
- Importing required Objects
- Importing required Rule types
- Create one or more
@rule functions, as required. A higher priority means this rule will be applied first. @task_rule Reference Rules have an if/else statement - they must always return a single thing or nothingOrbiterSSHOperator Reference- Append the new Rule to the
translation_ruleset -
Invoke the orbiter CLI, pointing it at your customized ruleset, and writing output to an output/ folder: orbiter translate workflow/ output/ --ruleset override.translation_ruleset\n
- Follow the remaining steps of the Translate instructions
"},{"location":"#authoring-a-new-ruleset","title":"Authoring a new Ruleset","text":"You can utilize the TranslationRuleset Template to create a new TranslationRuleset. "},{"location":"#faq","title":"FAQ","text":" -
Can this tool convert my workflows from tool X to Airflow? If you don't see your tool listed in Supported Origins, contact us for services to create translations, create an issue in the orbiter-community-translations repository, or write a TranslationRuleset and submit a pull request to share your translations with the community. -
Are the results of this tool under any guarantee of correctness? No. This tool is provided as-is, with no guarantee of correctness. It is your responsibility to verify the results. We accept Pull Requests to improve parsing, and strive to make the tool easily configurable to handle your specific use-case. Artwork Orbiter logo by Ivan Colic used with permission from The Noun Project under Creative Commons. "},{"location":"cli/","title":"CLI","text":""},{"location":"cli/#orbiter","title":"orbiter","text":"Orbiter is a CLI that converts other workflows to Airflow Projects. Usage: orbiter [OPTIONS] COMMAND [ARGS]...\n
Options: Name Type Description Default --version boolean Show the version and exit. False --help boolean Show this message and exit. False"},{"location":"cli/#install","title":"install","text":"Install a new Translation Ruleset from a repository Usage: orbiter install [OPTIONS]\n
Options: Name Type Description Default -r, --repo choice (astronomer-orbiter-translations | orbiter-community-translations) Choose a repository to install (will prompt, if not given) None -k, --key text [Optional] License Key to use for the translation ruleset. Should look like AAAA-BBBB-1111-2222-3333-XXXX-YYYY-ZZZZ None --help boolean Show this message and exit. False"},{"location":"cli/#list-rulesets","title":"list-rulesets","text":"List available Translation Rulesets Usage: orbiter list-rulesets [OPTIONS]\n
Options: Name Type Description Default --help boolean Show this message and exit. False"},{"location":"cli/#translate","title":"translate","text":"Translate workflows in an INPUT_DIR to an OUTPUT_DIR Airflow Project. Provide a specific ruleset with the --ruleset flag. Run orbiter list-rulesets to see available rulesets. INPUT_DIR defaults to $CWD/workflow.
OUTPUT_DIR defaults to $CWD/output
Formats output with Ruff (https://astral.sh/ruff), by default. Usage: orbiter translate [OPTIONS] INPUT_DIR OUTPUT_DIR\n
Options: Name Type Description Default -r, --ruleset text Qualified name of a TranslationRuleset _required --format / --no-format boolean [optional] format the output with Ruff True --help boolean Show this message and exit. False"},{"location":"cli/#logging","title":"Logging","text":"You can alter the verbosity of the CLI by setting the LOG_LEVEL environment variable. The default is INFO. export LOG_LEVEL=DEBUG\n
"},{"location":"origins/","title":"Origins","text":"An Origin is a source system that contains workflows that can be translated to an Apache Airflow project. "},{"location":"origins/#supported-origins","title":"Supported Origins","text":"Origin Maintainer Repository Ruleset DAG Equivalent Task Equivalent DAG Factory Community orbiter-community-translations orbiter_translations.dag_factory.yaml_base.translation_ruleset DAG Task Control M Astronomer astronomer-orbiter-translations orbiter_translations.control_m.json_base.translation_ruleset Folder Job \u2800\u2800 \u2800 \u2800 orbiter_translations.control_m.json_ssh.translation_ruleset \u2800 \u2800 \u2800 \u2800 \u2800 orbiter_translations.control_m.xml_base.translation_ruleset \u2800 \u2800 \u2800 \u2800 \u2800 orbiter_translations.control_m.xml_ssh.translation_ruleset \u2800 \u2800 \u2800 \u2800 orbiter-community-translations orbiter_translations.control_m.xml_demo.translation_ruleset \u2800 \u2800 Automic Astronomer astronomer-orbiter-translations WIP Job Plan Job Autosys Astronomer astronomer-orbiter-translations WIP \u2800 \u2800 JAMS Astronomer astronomer-orbiter-translations WIP Folder Job SSIS Astronomer astronomer-orbiter-translations orbiter_translations.ssis.xml_base.translation_ruleset Pipeline Component \u2800 \u2800 orbiter-community-translations orbiter_translations.oozie.xml_demo.translation_ruleset \u2800 \u2800 Oozie Astronomer astronomer-orbiter-translations orbiter_translations.oozie.xml_base.translation_ruleset Workflow Node \u2800 \u2800\u2800 orbiter-community-translations orbiter_translations.oozie.xml_demo.translation_ruleset \u2800 \u2800 & more! \u2800 \u2800 \u2800 \u2800 \u2800 For Astronomer maintained Translation Rulesets, please contact us for access to the most up-to-date versions. If you don't see your Origin system listed, please either: - contact us for services to create translations
- create an issue in our
orbiter-community-translations repository - write a
TranslationRuleset and submit a pull request to share your translations with the community "},{"location":"Rules_and_Rulesets/","title":"Overview","text":"The brain of the Orbiter framework is in it's Rules and the Rulesets that contain them. - A
Rule is a python function that is evaluated and produces something (typically an Object) or nothing - A
Ruleset is a collection of Rules that are evaluated in priority order - A
TranslationRuleset is a collection of Rulesets, relating to an Origin and FileType, with a translate_fn which determines how to apply the rulesets. Different Rules are applied in different scenarios, such as: - converting input to an Airflow DAG (
@dag_rule), - converting input to a specific Airflow Operator (
@task_rule), - filtering entries from the input data (
@dag_filter_rule, @task_filter_rule). Tip To map the following input {\n \"id\": \"my_task\",\n \"command\": \"echo 'hi'\"\n}\n
to an Airflow BashOperator, a Rule could parse it as follows: @task_rule\ndef my_rule(val):\n if 'command' in val:\n return OrbiterBashOperator(task_id=val['id'], bash_command=val['command'])\n else:\n return None\n
This returns a OrbiterBashOperator, which will become an Airflow BashOperator when the translation completes. "},{"location":"Rules_and_Rulesets/#orbiter.rules.rulesets.translate","title":"orbiter.rules.rulesets.translate","text":"translate(\n translation_ruleset, input_dir: Path\n) -> OrbiterProject\n
Orbiter expects a folder containing text files which may have a structure like: {\"<workflow name>\": { ...<workflow properties>, \"<task name>\": { ...<task properties>} }}\n
The standard translation function performs the following steps: - Find all files with the expected
TranslationRuleset.file_type (.json, .xml, .yaml, etc.) in the input folder. - Load each file and turn it into a Python Dictionary.
- For each file: Apply the
TranslationRuleset.dag_filter_ruleset to filter down to entries that can translate to a DAG, in priority order. - For each: Apply the
TranslationRuleset.dag_ruleset, to convert the object to an OrbiterDAG, in priority-order, stopping when the first rule returns a match. If no rule returns a match, the entry is filtered. - Apply the
TranslationRuleset.task_filter_ruleset to filter down to entries in the DAG that can translate to a Task, in priority-order. - For each: Apply the
TranslationRuleset.task_ruleset, to convert the object to a specific Task, in priority-order, stopping when the first rule returns a match. If no rule returns a match, the entry is filtered. - After the DAG and Tasks are mapped, the
TranslationRuleset.task_dependency_ruleset is applied in priority-order, stopping when the first rule returns a match, to create a list of OrbiterTaskDependency, which are then added to each task in the OrbiterDAG - Apply the
TranslationRuleset.post_processing_ruleset, against the OrbiterProject, which can make modifications after all other rules have been applied. - After translation - the
OrbiterProject is rendered to the output folder. Source code in orbiter/rules/rulesets.py @validate_call\ndef translate(translation_ruleset, input_dir: Path) -> OrbiterProject:\n \"\"\"\n Orbiter expects a folder containing text files which may have a structure like:\n ```json\n {\"<workflow name>\": { ...<workflow properties>, \"<task name>\": { ...<task properties>} }}\n ```\n\n The standard translation function performs the following steps:\n\n \n\n 1. [**Find all files**][orbiter.rules.rulesets.TranslationRuleset.get_files_with_extension] with the expected\n [`TranslationRuleset.file_type`][orbiter.rules.rulesets.TranslationRuleset]\n (`.json`, `.xml`, `.yaml`, etc.) in the input folder.\n - [**Load each file**][orbiter.rules.rulesets.TranslationRuleset.loads] and turn it into a Python Dictionary.\n 2. **For each file:** Apply the [`TranslationRuleset.dag_filter_ruleset`][orbiter.rules.rulesets.DAGFilterRuleset]\n to filter down to entries that can translate to a DAG, in priority order.\n - **For each**: Apply the [`TranslationRuleset.dag_ruleset`][orbiter.rules.rulesets.DAGRuleset],\n to convert the object to an [`OrbiterDAG`][orbiter.objects.dag.OrbiterDAG],\n in priority-order, stopping when the first rule returns a match.\n If no rule returns a match, the entry is filtered.\n 3. Apply the [`TranslationRuleset.task_filter_ruleset`][orbiter.rules.rulesets.TaskFilterRuleset]\n to filter down to entries in the DAG that can translate to a Task, in priority-order.\n - **For each:** Apply the [`TranslationRuleset.task_ruleset`][orbiter.rules.rulesets.TaskRuleset],\n to convert the object to a specific Task, in priority-order, stopping when the first rule returns a match.\n If no rule returns a match, the entry is filtered.\n 4. After the DAG and Tasks are mapped, the\n [`TranslationRuleset.task_dependency_ruleset`][orbiter.rules.rulesets.TaskDependencyRuleset]\n is applied in priority-order, stopping when the first rule returns a match,\n to create a list of\n [`OrbiterTaskDependency`][orbiter.objects.task.OrbiterTaskDependency],\n which are then added to each task in the\n [`OrbiterDAG`][orbiter.objects.dag.OrbiterDAG]\n 5. Apply the [`TranslationRuleset.post_processing_ruleset`][orbiter.rules.rulesets.PostProcessingRuleset],\n against the [`OrbiterProject`][orbiter.objects.project.OrbiterProject], which can make modifications after all\n other rules have been applied.\n 6. After translation - the [`OrbiterProject`][orbiter.objects.project.OrbiterProject]\n is rendered to the output folder.\n\n\n \"\"\"\n if not isinstance(translation_ruleset, TranslationRuleset):\n raise RuntimeError(\n f\"Error! type(translation_ruleset)=={type(translation_ruleset)}!=TranslationRuleset! Exiting!\"\n )\n\n # Create an initial OrbiterProject\n project = OrbiterProject()\n\n for i, (file, input_dict) in enumerate(\n translation_ruleset.get_files_with_extension(input_dir)\n ):\n logger.info(f\"Translating [File {i}]={file.resolve()}\")\n\n # DAG FILTER Ruleset - filter down to keys suspected of being translatable to a DAG, in priority order.\n dag_dicts = functools.reduce(\n add,\n translation_ruleset.dag_filter_ruleset.apply(val=input_dict),\n [],\n )\n logger.debug(f\"Found {len(dag_dicts)} DAG candidates in {file.resolve()}\")\n for dag_dict in dag_dicts:\n # DAG Ruleset - convert the object to an `OrbiterDAG` via `dag_ruleset`,\n # in priority-order, stopping when the first rule returns a match\n dag: OrbiterDAG | None = translation_ruleset.dag_ruleset.apply(\n val=dag_dict,\n take_first=True,\n )\n if dag is None:\n logger.warning(\n f\"Couldn't extract DAG from dag_dict={dag_dict} with dag_ruleset={translation_ruleset.dag_ruleset}\"\n )\n continue\n dag.orbiter_kwargs[\"file_path\"] = str(file.resolve())\n\n tasks = {}\n # TASK FILTER Ruleset - Many entries in dag_dict -> Many task_dict\n task_dicts = functools.reduce(\n add,\n translation_ruleset.task_filter_ruleset.apply(val=dag_dict),\n [],\n )\n logger.debug(\n f\"Found {len(task_dicts)} Task candidates in {dag.dag_id} in {file.resolve()}\"\n )\n for task_dict in task_dicts:\n # TASK Ruleset one -> one\n task: OrbiterOperator = translation_ruleset.task_ruleset.apply(\n val=task_dict, take_first=True\n )\n if task is None:\n logger.warning(\n f\"Couldn't extract task from expected task_dict={task_dict}\"\n )\n continue\n\n _add_task_deduped(task, tasks)\n logger.debug(f\"Adding {len(tasks)} tasks to DAG {dag.dag_id}\")\n dag.add_tasks(tasks.values())\n\n # Dag-Level TASK DEPENDENCY Ruleset\n task_dependencies: List[OrbiterTaskDependency] = (\n list(chain(*translation_ruleset.task_dependency_ruleset.apply(val=dag)))\n or []\n )\n if not len(task_dependencies):\n logger.warning(f\"Couldn't find task dependencies in dag={dag_dict}\")\n for task_dependency in task_dependencies:\n task_dependency: OrbiterTaskDependency\n if task_dependency.task_id not in dag.tasks:\n logger.warning(\n f\"Couldn't find task_id={task_dependency.task_id} in tasks={tasks} for dag_id={dag.dag_id}\"\n )\n continue\n else:\n dag.tasks[task_dependency.task_id].add_downstream(task_dependency)\n\n logger.debug(f\"Adding DAG {dag.dag_id} to project\")\n project.add_dags(dag)\n\n # POST PROCESSING Ruleset\n translation_ruleset.post_processing_ruleset.apply(val=project, take_first=False)\n\n return project\n
"},{"location":"Rules_and_Rulesets/rules/","title":"Rules","text":""},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.Rule","title":"orbiter.rules.Rule","text":"A Rule contains a python function that is evaluated and produces something (typically an Object) or nothing A Rule can be created from a decorator >>> @rule(priority=1)\n... def my_rule(val):\n... return 1\n>>> isinstance(my_rule, Rule)\nTrue\n>>> my_rule(val={})\n1\n
The function in a rule takes one parameter (val), and must always evaluate to something or nothing. >>> Rule(rule=lambda val: 4)({})\n4\n>>> Rule(rule=lambda val: None)({})\n
Tip If the returned value is an Orbiter Object, the passed kwargs are saved in a special orbiter_kwargs property >>> from orbiter.objects.dag import OrbiterDAG\n>>> @rule\n... def my_rule(foo):\n... return OrbiterDAG(dag_id=\"\", file_path=\"\")\n>>> my_rule(foo=\"bar\").orbiter_kwargs\n{'foo': 'bar'}\n
Note A Rule must have a rule property and extra properties cannot be passed >>> # noinspection Pydantic\n... Rule(rule=lambda: None, not_a_prop=\"???\")\n... # doctest: +ELLIPSIS\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description rule Callable[[dict | Any], Any | None] Python function to evaluate. Takes a single argument and returns something or nothing priority int, optional Higher priority rules are evaluated first, must be greater than 0. Default is 0 "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.DAGFilterRule","title":"orbiter.rules.DAGFilterRule","text":" Bases: Rule The @dag_filter_rule decorator creates a DAGFilterRule @dag_filter_rule\ndef foo(val: dict) -> List[dict]:\n return [{\"dag_id\": \"foo\"}]\n
Hint In addition to filtering, a DAGFilterRule can also map input to a more reasonable output for later processing "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.DAGRule","title":"orbiter.rules.DAGRule","text":" Bases: Rule A @dag_rule decorator creates a DAGRule @dag_rule\ndef foo(val: dict) -> OrbiterDAG | None:\n if 'id' in val:\n return OrbiterDAG(dag_id=val[\"id\"], file_path=f\"{val[\"id\"]}.py\")\n else:\n return None\n
"},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.TaskFilterRule","title":"orbiter.rules.TaskFilterRule","text":" Bases: Rule A @task_filter_rule decorator creates a TaskFilterRule @task_filter_rule\ndef foo(val: dict) -> List[dict] | None:\n return [{\"task_id\": \"foo\"}]\n
Hint In addition to filtering, a TaskFilterRule can also map input to a more reasonable output for later processing Parameters: Name Type Description val dict A dictionary of the task Returns: Type Description List[dict] | None A list of dictionaries of possible tasks or None "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.TaskRule","title":"orbiter.rules.TaskRule","text":" Bases: Rule A @task_rule decorator creates a TaskRule @task_rule\ndef foo(val: dict) -> OrbiterOperator | OrbiterTaskGroup:\n if 'id' in val and 'command' in val:\n return OrbiterBashOperator(task_id=val['id'], bash_command=val['command'])\n else:\n return None\n
Parameters: Name Type Description val dict A dictionary of the task Returns: Type Description OrbiterOperator | OrbiterTaskGroup | None A subclass of OrbiterOperator or OrbiterTaskGroup or None "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.TaskDependencyRule","title":"orbiter.rules.TaskDependencyRule","text":" Bases: Rule An @task_dependency_rule decorator creates a TaskDependencyRule, which takes an OrbiterDAG and returns a list[OrbiterTaskDependency] or None @task_dependency_rule\ndef foo(val: OrbiterDAG) -> OrbiterTaskDependency:\n return [OrbiterTaskDependency(task_id=\"upstream\", downstream=\"downstream\")]\n
Parameters: Name Type Description val OrbiterDAG An OrbiterDAG Returns: Type Description List[OrbiterTaskDependency] | None A list of OrbiterTaskDependency or None "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.PostProcessingRule","title":"orbiter.rules.PostProcessingRule","text":" Bases: Rule An @post_processing_rule decorator creates a PostProcessingRule, which takes an OrbiterProject, after all other rules have been applied, and modifies it in-place. @post_processing_rule\ndef foo(val: OrbiterProject) -> None:\n val.dags[\"foo\"].tasks[\"bar\"].doc = \"Hello World\"\n
Parameters: Name Type Description val OrbiterProject An OrbiterProject Returns: Type Description None None
"},{"location":"Rules_and_Rulesets/rulesets/","title":"Translation","text":""},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset","title":"orbiter.rules.rulesets.TranslationRuleset","text":"A Ruleset is a collection of Rules that are evaluated in priority order A TranslationRuleset is a container for Rulesets, which applies to a specific translation >>> TranslationRuleset(\n... file_type=FileType.JSON, # Has a file type\n... translate_fn=fake_translate, # and can have a callable\n... # translate_fn=\"orbiter.rules.translate.fake_translate\", # or a qualified name to a function\n... dag_filter_ruleset={\"ruleset\": [{\"rule\": lambda x: None}]}, # Rulesets can be dict within dicts\n... dag_ruleset=DAGRuleset(ruleset=[Rule(rule=lambda x: None)]), # or objects within objects\n... task_filter_ruleset=EMPTY_RULESET, # or a mix\n... task_ruleset=EMPTY_RULESET,\n... task_dependency_ruleset=EMPTY_RULESET, # Omitted for brevity\n... post_processing_ruleset=EMPTY_RULESET,\n... )\nTranslationRuleset(...)\n
Parameters: Name Type Description file_type FileType FileType to translate (.json, .xml, .yaml, etc.) dag_filter_ruleset DAGFilterRuleset | dict DAGFilterRuleset (of DAGFilterRule)
dag_ruleset DAGRuleset | dict DAGRuleset (of DAGRules)
task_filter_ruleset TaskFilterRuleset | dict TaskFilterRuleset (of TaskFilterRule)
task_ruleset TaskRuleset | dict TaskRuleset (of TaskRules)
task_dependency_ruleset TaskDependencyRuleset | dict TaskDependencyRuleset (of TaskDependencyRules)
post_processing_ruleset PostProcessingRuleset | dict PostProcessingRuleset (of PostProcessingRules)
translate_fn Callable[[TranslationRuleset, Path], OrbiterProject] | str | TranslateFn Either a qualified name to a function (e.g. path.to.file.function), or a function reference, with the signature: (translation_ruleset: Translation Ruleset, input_dir: Path) -> OrbiterProject "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.dumps","title":"dumps","text":"dumps(input_dict: dict) -> str\n
Convert Python dictionary back to source string form, useful for testing FileType Conversion Method XML xmltodict.unparse YAML yaml.safe_dump JSON json.dumps Parameters: Name Type Description input_dict dict The dictionary to convert to a string Returns: Type Description str The string representation of the input_dict, in the file_type format Source code in orbiter/rules/rulesets.py @validate_call\ndef dumps(self, input_dict: dict) -> str:\n \"\"\"\n Convert Python dictionary back to source string form, useful for testing\n\n | FileType | Conversion Method |\n |----------|---------------------|\n | `XML` | `xmltodict.unparse` |\n | `YAML` | `yaml.safe_dump` |\n | `JSON` | `json.dumps` |\n\n :param input_dict: The dictionary to convert to a string\n :type input_dict: dict\n :return str: The string representation of the input_dict, in the file_type format\n :rtype: str\n \"\"\"\n if self.file_type == FileType.JSON:\n import json\n\n return json.dumps(input_dict, indent=2)\n elif self.file_type == FileType.YAML:\n import yaml\n\n return yaml.safe_dump(input_dict)\n elif self.file_type == FileType.XML:\n import xmltodict\n\n return xmltodict.unparse(input_dict)\n else:\n raise NotImplementedError(f\"Cannot dump file_type={self.file_type}\")\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.get_files_with_extension","title":"get_files_with_extension","text":"get_files_with_extension(\n input_dir: Path,\n) -> Generator[Path, dict]\n
A generator that yields files with a specific extension(s) in a directory Parameters: Name Type Description input_dir Path The directory to search in Returns: Type Description Generator[Path, dict] Generator item of (Path, dict) for each file found Source code in orbiter/rules/rulesets.py def get_files_with_extension(self, input_dir: Path) -> Generator[Path, dict]:\n \"\"\"\n A generator that yields files with a specific extension(s) in a directory\n\n :param input_dir: The directory to search in\n :type input_dir: Path\n :return: Generator item of (Path, dict) for each file found\n :rtype: Generator[Path, dict]\n \"\"\"\n extension = f\".{self.file_type.value.lower()}\"\n extensions = [extension]\n\n # YAML and YML are both valid extensions\n extension_sub = {\n \"yaml\": \"yml\",\n }\n if other_extension := extension_sub.get(self.file_type.value.lower()):\n extensions.append(f\".{other_extension}\")\n\n logger.debug(f\"Finding files with extension={extensions} in {input_dir}\")\n\n def backport_walk(input_dir: Path):\n \"\"\"Path.walk() is only available in Python 3.12+, so, backport\"\"\"\n import os\n\n for result in os.walk(input_dir):\n yield Path(result[0]), result[1], result[2]\n\n for directory, _, files in (\n input_dir.walk() if hasattr(input_dir, \"walk\") else backport_walk(input_dir)\n ):\n logger.debug(f\"Checking directory={directory}\")\n for file in files:\n file = directory / file\n if file.suffix.lower() in extensions:\n logger.debug(f\"File={file} matches extension={extensions}\")\n yield (\n # Return the file path\n file,\n # and load the file and convert it into a python dict\n self.loads(file.read_text()),\n )\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.loads","title":"loads","text":"loads(input_str: str) -> dict\n
Converts all files of type into a Python dictionary \"intermediate representation\" form, prior to any rulesets being applied. FileType Conversion Method XML xmltodict_parse YAML yaml.safe_load JSON json.loads Parameters: Name Type Description input_str str The string to convert to a dictionary Returns: Type Description dict The dictionary representation of the input_str Source code in orbiter/rules/rulesets.py @validate_call\ndef loads(self, input_str: str) -> dict:\n \"\"\"\n Converts all files of type into a Python dictionary \"intermediate representation\" form,\n prior to any rulesets being applied.\n\n | FileType | Conversion Method |\n |----------|-------------------------------------------------------------|\n | `XML` | [`xmltodict_parse`][orbiter.rules.rulesets.xmltodict_parse] |\n | `YAML` | `yaml.safe_load` |\n | `JSON` | `json.loads` |\n\n :param input_str: The string to convert to a dictionary\n :type input_str: str\n :return: The dictionary representation of the input_str\n :rtype: dict\n \"\"\"\n\n if self.file_type == FileType.JSON:\n import json\n\n return json.loads(input_str)\n elif self.file_type == FileType.YAML:\n import yaml\n\n return yaml.safe_load(input_str)\n elif self.file_type == FileType.XML:\n return xmltodict_parse(input_str)\n else:\n raise NotImplementedError(f\"Cannot load file_type={self.file_type}\")\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.test","title":"test","text":"test(input_value: str | dict) -> OrbiterProject\n
Test an input against the whole ruleset. - 'input_dict' (a parsed python dict) - or 'input_str' (raw value) to test against the ruleset. Parameters: Name Type Description input_value str | dict The input to test can be either a dict (passed to translate_ruleset.dumps() before translate_ruleset.loads()) or a string (read directly by translate_ruleset.loads()) Returns: Type Description OrbiterProject OrbiterProject produced after applying the ruleset Source code in orbiter/rules/rulesets.py def test(self, input_value: str | dict) -> OrbiterProject:\n \"\"\"\n Test an input against the whole ruleset.\n - 'input_dict' (a parsed python dict)\n - or 'input_str' (raw value) to test against the ruleset.\n\n :param input_value: The input to test\n can be either a dict (passed to `translate_ruleset.dumps()` before `translate_ruleset.loads()`)\n or a string (read directly by `translate_ruleset.loads()`)\n :type input_value: str | dict\n :return: OrbiterProject produced after applying the ruleset\n :rtype: OrbiterProject\n \"\"\"\n with TemporaryDirectory() as tempdir:\n file = Path(tempdir) / f\"{uuid.uuid4()}.{self.file_type.value}\"\n file.write_text(\n self.dumps(input_value)\n if isinstance(input_value, dict)\n else input_value\n )\n return self.translate_fn(translation_ruleset=self, input_dir=file.parent)\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.xmltodict_parse","title":"orbiter.rules.rulesets.xmltodict_parse","text":"xmltodict_parse(input_str: str) -> Any\n
Calls xmltodict.parse and does post-processing fixes. Note The original xmltodict.parse method returns EITHER: - a dict (one child element of type)
- or a list of dict (many child element of type)
This behavior can be confusing, and is an issue with the original xml spec being referenced. This method deviates by standardizing to the latter case (always a list[dict]). All XML elements will be a list of dictionaries, even if there's only one element. >>> xmltodict_parse(\"\")\nTraceback (most recent call last):\nxml.parsers.expat.ExpatError: no element found: line 1, column 0\n>>> xmltodict_parse(\"<a></a>\")\n{'a': None}\n>>> xmltodict_parse(\"<a foo='bar'></a>\")\n{'a': [{'@foo': 'bar'}]}\n>>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'></foo></a>\") # Singleton - gets modified\n{'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz'}]}]}\n>>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'><bar><bop></bop></bar></foo></a>\") # Nested Singletons - modified\n{'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz', 'bar': [{'bop': None}]}]}]}\n>>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'></foo><foo bing='bop'></foo></a>\")\n{'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz'}, {'@bing': 'bop'}]}]}\n
Parameters: Name Type Description input_str str The XML string to parse Returns: Type Description dict The parsed XML Source code in orbiter/rules/rulesets.py def xmltodict_parse(input_str: str) -> Any:\n \"\"\"Calls `xmltodict.parse` and does post-processing fixes.\n\n !!! note\n\n The original [`xmltodict.parse`](https://pypi.org/project/xmltodict/) method returns EITHER:\n\n - a dict (one child element of type)\n - or a list of dict (many child element of type)\n\n This behavior can be confusing, and is an issue with the original xml spec being referenced.\n\n **This method deviates by standardizing to the latter case (always a `list[dict]`).**\n\n **All XML elements will be a list of dictionaries, even if there's only one element.**\n\n ```pycon\n >>> xmltodict_parse(\"\")\n Traceback (most recent call last):\n xml.parsers.expat.ExpatError: no element found: line 1, column 0\n >>> xmltodict_parse(\"<a></a>\")\n {'a': None}\n >>> xmltodict_parse(\"<a foo='bar'></a>\")\n {'a': [{'@foo': 'bar'}]}\n >>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'></foo></a>\") # Singleton - gets modified\n {'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz'}]}]}\n >>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'><bar><bop></bop></bar></foo></a>\") # Nested Singletons - modified\n {'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz', 'bar': [{'bop': None}]}]}]}\n >>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'></foo><foo bing='bop'></foo></a>\")\n {'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz'}, {'@bing': 'bop'}]}]}\n\n ```\n :param input_str: The XML string to parse\n :type input_str: str\n :return: The parsed XML\n :rtype: dict\n \"\"\"\n import xmltodict\n\n # noinspection t\n def _fix(d):\n \"\"\"fix the dict in place, recursively, standardizing on a list of dict even if there's only one entry.\"\"\"\n # if it's a dict, descend to fix\n if isinstance(d, dict):\n for k, v in d.items():\n # @keys are properties of elements, non-@keys are elements\n if not k.startswith(\"@\"):\n if isinstance(v, dict):\n # THE FIX\n # any non-@keys should be a list of dict, even if there's just one of the element\n d[k] = [v]\n _fix(v)\n else:\n _fix(v)\n # if it's a list, descend to fix\n if isinstance(d, list):\n for v in d:\n _fix(v)\n\n output = xmltodict.parse(input_str)\n _fix(output)\n return output\n
"},{"location":"Rules_and_Rulesets/rulesets/#rulesets","title":"Rulesets","text":""},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.Ruleset","title":"orbiter.rules.rulesets.Ruleset","text":"A list of rules, which are evaluated to generate different types of output You must pass a Rule (or dict with the schema of Rule) >>> from orbiter.rules import rule\n>>> @rule\n... def x(val):\n... return None\n>>> Ruleset(ruleset=[x, {\"rule\": lambda: None}])\n... # doctest: +ELLIPSIS\nRuleset(ruleset=[Rule(...), Rule(...)])\n
Note You can't pass non-Rules >>> # noinspection PyTypeChecker\n... Ruleset(ruleset=[None])\n... # doctest: +ELLIPSIS\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description ruleset List[Rule | Callable[[Any], Any | None]] List of Rule (or dict with the schema of Rule) "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.Ruleset.apply","title":"apply","text":"apply(\n take_first: bool = False, **kwargs\n) -> List[Any] | Any\n
Apply all rules in ruleset to a single item, in priority order, removing any None results. A ruleset with one rule can produce up to one result >>> from orbiter.rules import rule\n\n>>> @rule\n... def gt_4(val):\n... return str(val) if val > 4 else None\n>>> Ruleset(ruleset=[gt_4]).apply(val=5)\n['5']\n
Many rules can produce many results, one for each rule. >>> @rule\n... def gt_3(val):\n... return str(val) if val > 3 else None\n>>> Ruleset(ruleset=[gt_4, gt_3]).apply(val=5)\n['5', '5']\n
The take_first flag will evaluate rules in the ruleset and return the first match >>> Ruleset(ruleset=[gt_4, gt_3]).apply(val=5, take_first=True)\n'5'\n
If nothing matched, an empty list is returned >>> @rule\n... def always_none(val):\n... return None\n>>> @rule\n... def more_always_none(val):\n... return None\n>>> Ruleset(ruleset=[always_none, more_always_none]).apply(val=5)\n[]\n
If nothing matched, and take_first=True, None is returned >>> Ruleset(ruleset=[always_none, more_always_none]).apply(val=5, take_first=True)\n... # None\n
Tip If no input is given, an error is returned >>> Ruleset(ruleset=[always_none]).apply()\nTraceback (most recent call last):\nRuntimeError: No values provided! Supply at least one key=val pair as kwargs!\n
Parameters: Name Type Description take_first bool only take the first (if any) result from the ruleset application kwargs key=val pairs to pass to the evaluated rule function Returns: Type Description List[Any] | Any | None List of rules that evaluated to Any (in priority order), or an empty list, or Any (if take_first=True) Raises: Type Description RuntimeError if the Ruleset is empty or input_val is None RuntimeError if the Rule raises an exception Source code in orbiter/rules/rulesets.py @validate_call\ndef apply(self, take_first: bool = False, **kwargs) -> List[Any] | Any:\n \"\"\"\n Apply all rules in ruleset **to a single item**, in priority order, removing any `None` results.\n\n A ruleset with one rule can produce **up to one** result\n ```pycon\n >>> from orbiter.rules import rule\n\n >>> @rule\n ... def gt_4(val):\n ... return str(val) if val > 4 else None\n >>> Ruleset(ruleset=[gt_4]).apply(val=5)\n ['5']\n\n ```\n\n Many rules can produce many results, one for each rule.\n ```pycon\n >>> @rule\n ... def gt_3(val):\n ... return str(val) if val > 3 else None\n >>> Ruleset(ruleset=[gt_4, gt_3]).apply(val=5)\n ['5', '5']\n\n ```\n\n The `take_first` flag will evaluate rules in the ruleset and return the first match\n ```pycon\n >>> Ruleset(ruleset=[gt_4, gt_3]).apply(val=5, take_first=True)\n '5'\n\n ```\n\n If nothing matched, an empty list is returned\n ```pycon\n >>> @rule\n ... def always_none(val):\n ... return None\n >>> @rule\n ... def more_always_none(val):\n ... return None\n >>> Ruleset(ruleset=[always_none, more_always_none]).apply(val=5)\n []\n\n ```\n\n If nothing matched, and `take_first=True`, `None` is returned\n ```pycon\n >>> Ruleset(ruleset=[always_none, more_always_none]).apply(val=5, take_first=True)\n ... # None\n\n ```\n\n !!! tip\n\n If no input is given, an error is returned\n ```pycon\n >>> Ruleset(ruleset=[always_none]).apply()\n Traceback (most recent call last):\n RuntimeError: No values provided! Supply at least one key=val pair as kwargs!\n\n ```\n\n :param take_first: only take the first (if any) result from the ruleset application\n :type take_first: bool\n :param kwargs: key=val pairs to pass to the evaluated rule function\n :returns: List of rules that evaluated to `Any` (in priority order),\n or an empty list,\n or `Any` (if `take_first=True`)\n :rtype: List[Any] | Any | None\n :raises RuntimeError: if the Ruleset is empty or input_val is None\n :raises RuntimeError: if the Rule raises an exception\n \"\"\"\n if not len(kwargs):\n raise RuntimeError(\n \"No values provided! Supply at least one key=val pair as kwargs!\"\n )\n results = []\n for _rule in self._sorted():\n result = _rule(**kwargs)\n should_show_input = \"val\" in kwargs and not (\n isinstance(kwargs[\"val\"], OrbiterProject)\n or isinstance(kwargs[\"val\"], OrbiterDAG)\n )\n if result is not None:\n logger.debug(\n \"---------\\n\"\n f\"[RULESET MATCHED] '{self.__class__.__module__}.{self.__class__.__name__}'\\n\"\n f\"[RULE MATCHED] '{_rule.__name__}'\\n\"\n f\"[INPUT] {kwargs if should_show_input else '<Skipping...>'}\\n\"\n f\"[RETURN] {result}\\n\"\n f\"---------\"\n )\n results.append(result)\n if take_first:\n return result\n return None if take_first and not len(results) else results\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.Ruleset.apply_many","title":"apply_many","text":"apply_many(\n input_val: Collection[Any], take_first: bool = False\n) -> List[List[Any]] | List[Any]\n
Apply a ruleset to each item in collection (such as dict().items()) and return any results that are not None You can turn the output of apply_many into a dict, if the rule takes and returns a tuple >>> from itertools import chain\n>>> from orbiter.rules import rule\n\n>>> @rule\n... def filter_for_type_folder(val):\n... (key, val) = val\n... return (key, val) if val.get('Type', '') == 'Folder' else None\n>>> ruleset = Ruleset(ruleset=[filter_for_type_folder])\n>>> input_dict = {\n... \"a\": {\"Type\": \"Folder\"},\n... \"b\": {\"Type\": \"File\"},\n... \"c\": {\"Type\": \"Folder\"},\n... }\n>>> dict(chain(*chain(ruleset.apply_many(input_dict.items()))))\n... # use dict(chain(*chain(...))), if using `take_first=True`, to turn many results back into dict\n{'a': {'Type': 'Folder'}, 'c': {'Type': 'Folder'}}\n>>> dict(ruleset.apply_many(input_dict.items(), take_first=True))\n... # use dict(...) directly, if using `take_first=True`, to turn results back into dict\n{'a': {'Type': 'Folder'}, 'c': {'Type': 'Folder'}}\n
Tip You cannot pass input without length >>> ruleset.apply_many({})\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\nRuntimeError: Input is not Collection[Any] with length!\n
Parameters: Name Type Description input_val Collection[Any] List to evaluate ruleset over take_first bool Only take the first (if any) result from each ruleset application Returns: Type Description List[List[Any]] | List[Any] List of list with all non-null evaluations for each item or list of the first non-null evaluation for each item (if take_first=True) Raises: Type Description RuntimeError if the Ruleset or input_vals are empty RuntimeError if the Rule raises an exception Source code in orbiter/rules/rulesets.py def apply_many(\n self,\n input_val: Collection[Any],\n take_first: bool = False,\n) -> List[List[Any]] | List[Any]:\n \"\"\"\n Apply a ruleset to each item in collection (such as `dict().items()`)\n and return any results that are not `None`\n\n You can turn the output of `apply_many` into a dict, if the rule takes and returns a tuple\n ```pycon\n >>> from itertools import chain\n >>> from orbiter.rules import rule\n\n >>> @rule\n ... def filter_for_type_folder(val):\n ... (key, val) = val\n ... return (key, val) if val.get('Type', '') == 'Folder' else None\n >>> ruleset = Ruleset(ruleset=[filter_for_type_folder])\n >>> input_dict = {\n ... \"a\": {\"Type\": \"Folder\"},\n ... \"b\": {\"Type\": \"File\"},\n ... \"c\": {\"Type\": \"Folder\"},\n ... }\n >>> dict(chain(*chain(ruleset.apply_many(input_dict.items()))))\n ... # use dict(chain(*chain(...))), if using `take_first=True`, to turn many results back into dict\n {'a': {'Type': 'Folder'}, 'c': {'Type': 'Folder'}}\n >>> dict(ruleset.apply_many(input_dict.items(), take_first=True))\n ... # use dict(...) directly, if using `take_first=True`, to turn results back into dict\n {'a': {'Type': 'Folder'}, 'c': {'Type': 'Folder'}}\n\n ```\n !!! tip\n\n You cannot pass input without length\n ```pycon\n >>> ruleset.apply_many({})\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n RuntimeError: Input is not Collection[Any] with length!\n\n ```\n :param input_val: List to evaluate ruleset over\n :type input_val: Collection[Any]\n :param take_first: Only take the first (if any) result from each ruleset application\n :type take_first: bool\n :returns: List of list with all non-null evaluations for each item<br>\n or list of the first non-null evaluation for each item (if `take_first=True`)\n :rtype: List[List[Any]] | List[Any]\n :raises RuntimeError: if the Ruleset or input_vals are empty\n :raises RuntimeError: if the Rule raises an exception\n \"\"\"\n # Validate Input\n if not input_val or not len(input_val):\n raise RuntimeError(\"Input is not `Collection[Any]` with length!\")\n\n return [\n results[0] if take_first else results\n for item in input_val\n if (results := self.apply(take_first=False, val=item)) is not None\n and len(results)\n ]\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.DAGFilterRuleset","title":"orbiter.rules.rulesets.DAGFilterRuleset","text":" Bases: Ruleset Ruleset of DAGFilterRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.DAGRuleset","title":"orbiter.rules.rulesets.DAGRuleset","text":" Bases: Ruleset Ruleset of DAGRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TaskFilterRuleset","title":"orbiter.rules.rulesets.TaskFilterRuleset","text":" Bases: Ruleset Ruleset of TaskFilterRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TaskRuleset","title":"orbiter.rules.rulesets.TaskRuleset","text":" Bases: Ruleset Ruleset of TaskRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TaskDependencyRuleset","title":"orbiter.rules.rulesets.TaskDependencyRuleset","text":" Bases: Ruleset Ruleset of TaskDependencyRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.PostProcessingRuleset","title":"orbiter.rules.rulesets.PostProcessingRuleset","text":" Bases: Ruleset Ruleset of PostProcessingRule "},{"location":"Rules_and_Rulesets/template/","title":"Template","text":"The following template can be utilized for creating a new TranslationRuleset translation_template.pyfrom __future__ import annotations\nfrom orbiter import FileType\nfrom orbiter.objects.dag import OrbiterDAG\nfrom orbiter.objects.operators.empty import OrbiterEmptyOperator\nfrom orbiter.objects.project import OrbiterProject\nfrom orbiter.objects.task import OrbiterOperator\nfrom orbiter.objects.task_group import OrbiterTaskGroup\nfrom orbiter.rules import (\n dag_filter_rule,\n dag_rule,\n task_filter_rule,\n task_rule,\n task_dependency_rule,\n post_processing_rule,\n cannot_map_rule,\n)\nfrom orbiter.rules.rulesets import (\n DAGFilterRuleset,\n DAGRuleset,\n TaskFilterRuleset,\n TaskRuleset,\n TaskDependencyRuleset,\n PostProcessingRuleset,\n TranslationRuleset,\n)\n\n\n@dag_filter_rule\ndef basic_dag_filter(val: dict) -> list | None:\n \"\"\"Filter input down to a list of dictionaries that can be processed by the `@dag_rules`\"\"\"\n for k, v in val.items():\n pass\n return []\n\n\n@dag_rule\ndef basic_dag_rule(val: dict) -> OrbiterDAG | None:\n \"\"\"Translate input into an `OrbiterDAG`\"\"\"\n if \"dag_id\" in val:\n return OrbiterDAG(dag_id=val[\"dag_id\"], file_path=\"file.py\")\n else:\n return None\n\n\n@task_filter_rule\ndef basic_task_filter(val: dict) -> list | None:\n \"\"\"Filter input down to a list of dictionaries that can be processed by the `@task_rules`\"\"\"\n for k, v in val.items():\n pass\n return []\n\n\n@task_rule(priority=2)\ndef basic_task_rule(val: dict) -> OrbiterOperator | OrbiterTaskGroup | None:\n \"\"\"Translate input into an Operator (e.g. `OrbiterBashOperator`). will be applied first, with a higher priority\"\"\"\n if \"task_id\" in val:\n return OrbiterEmptyOperator(task_id=val[\"task_id\"])\n else:\n return None\n\n\n@task_dependency_rule\ndef basic_task_dependency_rule(val: OrbiterDAG) -> list | None:\n \"\"\"Translate input into a list of task dependencies\"\"\"\n for task_dependency in val.orbiter_kwargs[\"task_dependencies\"]:\n pass\n return []\n\n\n@post_processing_rule\ndef basic_post_processing_rule(val: OrbiterProject) -> None:\n \"\"\"Modify the project in-place, after all other rules have applied\"\"\"\n for dag_id, dag in val.dags.items():\n for task_id, task in dag.tasks.items():\n pass\n\n\ntranslation_ruleset = TranslationRuleset(\n file_type=FileType.JSON,\n dag_filter_ruleset=DAGFilterRuleset(ruleset=[basic_dag_filter]),\n dag_ruleset=DAGRuleset(ruleset=[basic_dag_rule]),\n task_filter_ruleset=TaskFilterRuleset(ruleset=[basic_task_filter]),\n task_ruleset=TaskRuleset(ruleset=[basic_task_rule, cannot_map_rule]),\n task_dependency_ruleset=TaskDependencyRuleset(ruleset=[basic_task_dependency_rule]),\n post_processing_ruleset=PostProcessingRuleset(ruleset=[basic_post_processing_rule]),\n)\n
"},{"location":"objects/","title":"Overview","text":"Objects are returned from Rules during a translation, and are rendered to produce an Apache Airflow Project An OrbiterProject holds everything necessary to render an Airflow Project. It is generated by a TranslationRuleset.translate_fn. Workflows are represented by a OrbiterDAG which is a Directed Acyclic Graph (of Tasks). OrbiterOperators represent Airflow Tasks, which are units of work. An Operator is a pre-defined task with specific functionality.
"},{"location":"objects/#orbiter.objects.OrbiterBase","title":"orbiter.objects.OrbiterBase","text":"AbstractBaseClass for Orbiter objects, provides a number of properties Parameters: Name Type Description imports List[OrbiterRequirement] List of OrbiterRequirement objects orbiter_kwargs dict, optional Optional dictionary of keyword arguments, to preserve what was originally parsed by a rule orbiter_conns Set[OrbiterConnection], optional Optional set of OrbiterConnection objects orbiter_env_vars Set[OrbiterEnvVar], optional Optional set of OrbiterEnvVar objects orbiter_includes Set[OrbiterInclude], optional Optional set of OrbiterInclude objects orbiter_vars Set[OrbiterVariable], optional Optional set of OrbiterVariable objects "},{"location":"objects/project/","title":"Project","text":"An OrbiterProject holds everything necessary to render an Airflow Project. It is generated by a TranslationRuleset.translate_fn. "},{"location":"objects/project/#diagram","title":"Diagram","text":"classDiagram\n direction LR\n\n OrbiterProject --> \"many\" OrbiterConnection\n OrbiterProject --> \"many\" OrbiterDAG\n OrbiterProject --> \"many\" OrbiterEnvVar\n OrbiterProject --> \"many\" OrbiterInclude\n OrbiterProject --> \"many\" OrbiterPool\n OrbiterProject --> \"many\" OrbiterRequirement\n OrbiterProject --> \"many\" OrbiterVariable\n class OrbiterProject[\"orbiter.objects.project.OrbiterProject\"] {\n connections: Dict[str, OrbiterConnection]\n dags: Dict[str, OrbiterDAG]\n env_vars: Dict[str, OrbiterEnvVar]\n includes: Dict[str, OrbiterInclude]\n pools: Dict[str, OrbiterPool]\n requirements: Set[OrbiterRequirement]\n variables: Dict[str, OrbiterVariable]\n }\n click OrbiterProject href \"#orbiter.objects.project.OrbiterProject\" \"OrbiterProject Documentation\"\n\n class OrbiterConnection[\"orbiter.objects.connection.OrbiterConnection\"] {\n conn_id: str\n conn_type: str\n **kwargs\n }\n click OrbiterConnection href \"#orbiter.objects.connection.OrbiterConnection\" \"OrbiterConnection Documentation\"\n\n OrbiterDAG --> \"many\" OrbiterInclude\n OrbiterDAG --> \"many\" OrbiterConnection\n OrbiterDAG --> \"many\" OrbiterEnvVar\n OrbiterDAG --> \"many\" OrbiterRequirement\n OrbiterDAG --> \"many\" OrbiterVariable\n class OrbiterDAG[\"orbiter.objects.dag.OrbiterDAG\"] {\n imports: List[OrbiterRequirement]\n file_path: str\n dag_id: str\n schedule: str | OrbiterTimetable | None\n catchup: bool\n start_date: DateTime\n tags: List[str]\n default_args: Dict[str, Any]\n params: Dict[str, Any]\n doc_md: str | None\n tasks: Dict[str, OrbiterOperator]\n kwargs: dict\n orbiter_kwargs: dict\n orbiter_conns: Set[OrbiterConnection]\n orbiter_vars: Set[OrbiterVariable]\n orbiter_env_vars: Set[OrbiterEnvVar]\n orbiter_includes: Set[OrbiterInclude]\n }\n\n class OrbiterEnvVar[\"orbiter.objects.env_var.OrbiterEnvVar\"] {\n key: str\n value: str\n }\n click OrbiterEnvVar href \"#orbiter.objects.env_var.OrbiterEnvVar\" \"OrbiterEnvVar Documentation\"\n\n class OrbiterInclude[\"orbiter.objects.include.OrbiterInclude\"] {\n filepath: str\n contents: str\n }\n click OrbiterInclude href \"#orbiter.objects.include.OrbiterInclude\" \"OrbiterInclude Documentation\"\n\n class OrbiterPool[\"orbiter.objects.pool.OrbiterPool\"] {\n name: str\n description: str | None\n slots: int | None\n }\n click OrbiterPool href \"#orbiter.objects.pool.OrbiterPool\" \"OrbiterPool Documentation\"\n\n class OrbiterRequirement[\"orbiter.objects.requirement.OrbiterRequirement\"] {\n package: str | None\n module: str | None\n names: List[str] | None\n sys_package: str | None\n }\n click OrbiterRequirement href \"#orbiter.objects.requirement.OrbiterRequirement\" \"OrbiterRequirement Documentation\"\n\n class OrbiterVariable[\"orbiter.objects.variable.OrbiterVariable\"] {\n key: str\n value: str\n }\n click OrbiterVariable href \"#orbiter.objects.variable.OrbiterVariable\" \"OrbiterVariable Documentation\"\n
"},{"location":"objects/project/#orbiter.objects.connection.OrbiterConnection","title":"orbiter.objects.connection.OrbiterConnection","text":"An Airflow Connection, rendered to an airflow_settings.yaml file. See also other Connection documentation. >>> OrbiterConnection(\n... conn_id=\"my_conn_id\", conn_type=\"mysql\", host=\"localhost\", port=3306, login=\"root\"\n... ).render()\n{'conn_id': 'my_conn_id', 'conn_type': 'mysql', 'conn_host': 'localhost', 'conn_port': 3306, 'conn_login': 'root'}\n
Note Use the utility conn_id function to generate both an OrbiterConnection and connection property for an operator from orbiter.objects import conn_id\n\nOrbiterTask(\n ... ,\n **conn_id(\"my_conn_id\", conn_type=\"mysql\"),\n)\n
Parameters: Name Type Description conn_id str The ID of the connection conn_type str, optional The type of the connection, always lowercase. Defaults to 'generic' **kwargs Additional properties for the connection "},{"location":"objects/project/#orbiter.objects.env_var.OrbiterEnvVar","title":"orbiter.objects.env_var.OrbiterEnvVar","text":"Represents an Environmental Variable, renders to a line in .env file >>> OrbiterEnvVar(key=\"foo\", value=\"bar\").render()\n'foo=bar'\n
Parameters: Name Type Description key str The key of the environment variable value str The value of the environment variable "},{"location":"objects/project/#orbiter.objects.include.OrbiterInclude","title":"orbiter.objects.include.OrbiterInclude","text":"Represents an included file in an /include directory Parameters: Name Type Description filepath str The relative path (from the output directory) to write the file to contents str The contents of the file "},{"location":"objects/project/#orbiter.objects.pool.OrbiterPool","title":"orbiter.objects.pool.OrbiterPool","text":"An Airflow Pool, rendered to an airflow_settings.yaml file. >>> OrbiterPool(name=\"foo\", description=\"bar\", slots=5).render()\n{'pool_name': 'foo', 'pool_description': 'bar', 'pool_slot': 5}\n
Note Use the utility pool function to easily generate both an OrbiterPool and pool property for an operator from orbiter.objects import pool\n\nOrbiterTask(\n ... ,\n **pool(\"my_pool\"),\n)\n
Parameters: Name Type Description name str The name of the pool description str, optional The description of the pool slots int, optional The number of slots in the pool. Defaults to 128 "},{"location":"objects/project/#orbiter.objects.requirement.OrbiterRequirement","title":"orbiter.objects.requirement.OrbiterRequirement","text":"A requirement for a project (e.g. apache-airflow-providers-google), and it's representation in the DAG file. Renders via the DAG File (as an import statement), requirements.txt, and packages.txt Tip If a given requirement has multiple packages required, it can be defined as multiple OrbiterRequirement objects. Example: OrbiterTask(\n ...,\n imports=[\n OrbiterRequirement(package=\"apache-airflow-providers-google\", ...),\n OrbiterRequirement(package=\"bigquery\", sys_package=\"mysql\", ...),\n ],\n)\n
Parameters: Name Type Description package str, optional e.g. \"apache-airflow-providers-google\" module str, optional e.g. \"airflow.providers.google.cloud.operators.bigquery\", defaults to None names List[str], optional e.g. [\"BigQueryCreateEmptyDatasetOperator\"], defaults to [] sys_package Set[str], optional e.g. \"mysql\" - represents a Debian system package "},{"location":"objects/project/#orbiter.objects.variable.OrbiterVariable","title":"orbiter.objects.variable.OrbiterVariable","text":"An Airflow Variable, rendered to an airflow_settings.yaml file. >>> OrbiterVariable(key=\"foo\", value=\"bar\").render()\n{'variable_value': 'bar', 'variable_name': 'foo'}\n
Parameters: Name Type Description key str The key of the variable value str The value of the variable "},{"location":"objects/project/#orbiter.objects.project.OrbiterProject","title":"orbiter.objects.project.OrbiterProject","text":"OrbiterProject()\n
Holds everything necessary to render an Airflow Project. This is generated by a TranslationRuleset.translate_fn. Tip They can be added together >>> OrbiterProject() + OrbiterProject()\nOrbiterProject(dags=[], requirements=[], pools=[], connections=[], variables=[], env_vars=[])\n
And compared >>> OrbiterProject() == OrbiterProject()\nTrue\n
Parameters: Name Type Description connections Dict[str, OrbiterConnection] A dictionary of OrbiterConnections dags Dict[str, OrbiterDAG] A dictionary of OrbiterDAGs env_vars Dict[str, OrbiterEnvVar] A dictionary of OrbiterEnvVars includes Dict[str, OrbiterInclude] A dictionary of OrbiterIncludes pools Dict[str, OrbiterPool] A dictionary of OrbiterPools requirements Set[OrbiterRequirement] A set of OrbiterRequirements variables Dict[str, OrbiterVariable] A dictionary of OrbiterVariables Source code in orbiter/objects/project.py def __init__(self):\n self.dags: Dict[str, OrbiterDAG] = dict()\n self.requirements: Set[OrbiterRequirement] = set()\n self.pools: Dict[str, OrbiterPool] = dict()\n self.connections: Dict[str, OrbiterConnection] = dict()\n self.variables: Dict[str, OrbiterVariable] = dict()\n self.env_vars: Dict[str, OrbiterEnvVar] = dict()\n self.includes: Dict[str, OrbiterInclude] = dict()\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_connections","title":"add_connections","text":"add_connections(\n connections: (\n OrbiterConnection | Iterable[OrbiterConnection]\n ),\n) -> \"OrbiterProject\"\n
Add OrbiterConnections to the Project or override an existing connection with new properties >>> OrbiterProject().add_connections(OrbiterConnection(conn_id='foo')).connections\n{'foo': OrbiterConnection(conn_id=foo, conn_type=generic)}\n\n>>> OrbiterProject().add_connections(\n... [OrbiterConnection(conn_id='foo'), OrbiterConnection(conn_id='bar')]\n... ).connections\n{'foo': OrbiterConnection(conn_id=foo, conn_type=generic), 'bar': OrbiterConnection(conn_id=bar, conn_type=generic)}\n
Tip Validation requires an OrbiterConnection to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_connections('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n>>> OrbiterProject().add_connections(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description connections OrbiterConnection | Iterable[OrbiterConnection] List of OrbiterConnections Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_connections(\n self, connections: OrbiterConnection | Iterable[OrbiterConnection]\n) -> \"OrbiterProject\":\n \"\"\"Add [`OrbiterConnections`][orbiter.objects.connection.OrbiterConnection] to the Project\n or override an existing connection with new properties\n\n ```pycon\n >>> OrbiterProject().add_connections(OrbiterConnection(conn_id='foo')).connections\n {'foo': OrbiterConnection(conn_id=foo, conn_type=generic)}\n\n >>> OrbiterProject().add_connections(\n ... [OrbiterConnection(conn_id='foo'), OrbiterConnection(conn_id='bar')]\n ... ).connections\n {'foo': OrbiterConnection(conn_id=foo, conn_type=generic), 'bar': OrbiterConnection(conn_id=bar, conn_type=generic)}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterConnection` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_connections('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n >>> OrbiterProject().add_connections(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n\n :param connections: List of [`OrbiterConnections`][orbiter.objects.connection.OrbiterConnection]\n :type connections: List[OrbiterConnection] | OrbiterConnection\n :return: self\n :rtype: OrbiterProject\n \"\"\" # noqa: E501\n for connection in (\n [connections] if isinstance(connections, OrbiterConnection) else connections\n ):\n self.connections[connection.conn_id] = connection\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_dags","title":"add_dags","text":"add_dags(\n dags: OrbiterDAG | Iterable[OrbiterDAG],\n) -> \"OrbiterProject\"\n
Add OrbiterDAGs (and any OrbiterRequirements, OrbiterConns, OrbiterVars, OrbiterPools, OrbiterEnvVars, etc.) to the Project. >>> OrbiterProject().add_dags(OrbiterDAG(dag_id='foo', file_path=\"\")).dags['foo'].repr()\n'OrbiterDAG(dag_id=foo, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)'\n\n>>> dags = OrbiterProject().add_dags(\n... [OrbiterDAG(dag_id='foo', file_path=\"\"), OrbiterDAG(dag_id='bar', file_path=\"\")]\n... ).dags; dags['foo'].repr(), dags['bar'].repr()\n... # doctest: +NORMALIZE_WHITESPACE\n('OrbiterDAG(dag_id=foo, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)', 'OrbiterDAG(dag_id=bar, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)')\n\n>>> # An example adding a little of everything, including deeply nested things\n... from orbiter.objects.operators.bash import OrbiterBashOperator\n>>> from orbiter.objects.timetables.multi_cron_timetable import OrbiterMultiCronTimetable\n>>> from orbiter.objects.callbacks.smtp import OrbiterSmtpNotifierCallback\n>>> OrbiterProject().add_dags(OrbiterDAG(\n... dag_id='foo', file_path=\"\",\n... orbiter_env_vars={OrbiterEnvVar(key=\"foo\", value=\"bar\")},\n... orbiter_includes={OrbiterInclude(filepath='foo.txt', contents=\"Hello, World!\")},\n... schedule=OrbiterMultiCronTimetable(cron_defs=[\"0 */5 * * *\", \"0 */3 * * *\"]),\n... tasks={'foo': OrbiterTaskGroup(task_group_id=\"foo\",\n... tasks=[OrbiterBashOperator(\n... task_id='foo', bash_command='echo \"Hello, World!\"',\n... orbiter_pool=OrbiterPool(name='foo', slots=1),\n... orbiter_vars={OrbiterVariable(key='foo', value='bar')},\n... orbiter_conns={OrbiterConnection(conn_id='foo')},\n... orbiter_env_vars={OrbiterEnvVar(key='foo', value='bar')},\n... on_success_callback=OrbiterSmtpNotifierCallback(\n... to=\"foo@bar.com\",\n... smtp_conn_id=\"SMTP\",\n... orbiter_conns={OrbiterConnection(conn_id=\"SMTP\", conn_type=\"smtp\")}\n... )\n... )]\n... )}\n... ))\n... # doctest: +NORMALIZE_WHITESPACE\nOrbiterProject(dags=[foo],\nrequirements=[OrbiterRequirements(names=[DAG], package=apache-airflow, module=airflow, sys_package=None),\nOrbiterRequirements(names=[BashOperator], package=apache-airflow, module=airflow.operators.bash, sys_package=None),\nOrbiterRequirements(names=[send_smtp_notification], package=apache-airflow-providers-smtp, module=airflow.providers.smtp.notifications.smtp, sys_package=None),\nOrbiterRequirements(names=[TaskGroup], package=apache-airflow, module=airflow.utils.task_group, sys_package=None),\nOrbiterRequirements(names=[MultiCronTimetable], package=croniter, module=multi_cron_timetable, sys_package=None),\nOrbiterRequirements(names=[DateTime,Timezone], package=pendulum, module=pendulum, sys_package=None)],\npools=['foo'],\nconnections=['SMTP', 'foo'],\nvariables=['foo'],\nenv_vars=['foo'])\n
Tip Validation requires an OrbiterDAG to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_dags('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n>>> OrbiterProject().add_dags(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description dags OrbiterDAG | Iterable[OrbiterDAG] List of OrbiterDAGs Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_dags(self, dags: OrbiterDAG | Iterable[OrbiterDAG]) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterDAGs][orbiter.objects.dag.OrbiterDAG]\n (and any [OrbiterRequirements][orbiter.objects.requirement.OrbiterRequirement],\n [OrbiterConns][orbiter.objects.connection.OrbiterConnection],\n [OrbiterVars][orbiter.objects.variable.OrbiterVariable],\n [OrbiterPools][orbiter.objects.pool.OrbiterPool],\n [OrbiterEnvVars][orbiter.objects.env_var.OrbiterEnvVar], etc.)\n to the Project.\n\n ```pycon\n >>> OrbiterProject().add_dags(OrbiterDAG(dag_id='foo', file_path=\"\")).dags['foo'].repr()\n 'OrbiterDAG(dag_id=foo, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)'\n\n >>> dags = OrbiterProject().add_dags(\n ... [OrbiterDAG(dag_id='foo', file_path=\"\"), OrbiterDAG(dag_id='bar', file_path=\"\")]\n ... ).dags; dags['foo'].repr(), dags['bar'].repr()\n ... # doctest: +NORMALIZE_WHITESPACE\n ('OrbiterDAG(dag_id=foo, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)', 'OrbiterDAG(dag_id=bar, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)')\n\n >>> # An example adding a little of everything, including deeply nested things\n ... from orbiter.objects.operators.bash import OrbiterBashOperator\n >>> from orbiter.objects.timetables.multi_cron_timetable import OrbiterMultiCronTimetable\n >>> from orbiter.objects.callbacks.smtp import OrbiterSmtpNotifierCallback\n >>> OrbiterProject().add_dags(OrbiterDAG(\n ... dag_id='foo', file_path=\"\",\n ... orbiter_env_vars={OrbiterEnvVar(key=\"foo\", value=\"bar\")},\n ... orbiter_includes={OrbiterInclude(filepath='foo.txt', contents=\"Hello, World!\")},\n ... schedule=OrbiterMultiCronTimetable(cron_defs=[\"0 */5 * * *\", \"0 */3 * * *\"]),\n ... tasks={'foo': OrbiterTaskGroup(task_group_id=\"foo\",\n ... tasks=[OrbiterBashOperator(\n ... task_id='foo', bash_command='echo \"Hello, World!\"',\n ... orbiter_pool=OrbiterPool(name='foo', slots=1),\n ... orbiter_vars={OrbiterVariable(key='foo', value='bar')},\n ... orbiter_conns={OrbiterConnection(conn_id='foo')},\n ... orbiter_env_vars={OrbiterEnvVar(key='foo', value='bar')},\n ... on_success_callback=OrbiterSmtpNotifierCallback(\n ... to=\"foo@bar.com\",\n ... smtp_conn_id=\"SMTP\",\n ... orbiter_conns={OrbiterConnection(conn_id=\"SMTP\", conn_type=\"smtp\")}\n ... )\n ... )]\n ... )}\n ... ))\n ... # doctest: +NORMALIZE_WHITESPACE\n OrbiterProject(dags=[foo],\n requirements=[OrbiterRequirements(names=[DAG], package=apache-airflow, module=airflow, sys_package=None),\n OrbiterRequirements(names=[BashOperator], package=apache-airflow, module=airflow.operators.bash, sys_package=None),\n OrbiterRequirements(names=[send_smtp_notification], package=apache-airflow-providers-smtp, module=airflow.providers.smtp.notifications.smtp, sys_package=None),\n OrbiterRequirements(names=[TaskGroup], package=apache-airflow, module=airflow.utils.task_group, sys_package=None),\n OrbiterRequirements(names=[MultiCronTimetable], package=croniter, module=multi_cron_timetable, sys_package=None),\n OrbiterRequirements(names=[DateTime,Timezone], package=pendulum, module=pendulum, sys_package=None)],\n pools=['foo'],\n connections=['SMTP', 'foo'],\n variables=['foo'],\n env_vars=['foo'])\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterDAG` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_dags('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n >>> OrbiterProject().add_dags(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n\n :param dags: List of [OrbiterDAGs][orbiter.objects.dag.OrbiterDAG]\n :type dags: List[OrbiterDAG] | OrbiterDAG\n :return: self\n :rtype: OrbiterProject\n \"\"\" # noqa: E501\n\n # noinspection t\n def _add_recursively(\n things: Iterable[\n OrbiterOperator\n | OrbiterTaskGroup\n | OrbiterCallback\n | OrbiterTimetable\n | OrbiterDAG\n ],\n ):\n for thing in things:\n if isinstance(thing, str):\n continue\n if hasattr(thing, \"orbiter_pool\") and (pool := thing.orbiter_pool):\n self.add_pools(pool)\n if hasattr(thing, \"orbiter_conns\") and (conns := thing.orbiter_conns):\n self.add_connections(conns)\n if hasattr(thing, \"orbiter_vars\") and (variables := thing.orbiter_vars):\n self.add_variables(variables)\n if hasattr(thing, \"orbiter_env_vars\") and (\n env_vars := thing.orbiter_env_vars\n ):\n self.add_env_vars(env_vars)\n if hasattr(thing, \"orbiter_includes\") and (\n includes := thing.orbiter_includes\n ):\n self.add_includes(includes)\n if hasattr(thing, \"imports\") and (imports := thing.imports):\n self.add_requirements(imports)\n if isinstance(thing, OrbiterTaskGroup) and (tasks := thing.tasks):\n _add_recursively(tasks)\n if hasattr(thing, \"__dict__\") or hasattr(thing, \"model_extra\"):\n # If it's a pydantic model or dict, check its properties for more things to add\n _add_recursively(\n (\n (getattr(thing, \"__dict__\", {}) or dict())\n | (getattr(thing, \"model_extra\", {}) or dict())\n ).values()\n )\n\n for dag in [dags] if isinstance(dags, OrbiterDAG) else dags:\n dag_id = dag.dag_id\n\n # Add or update the DAG\n if dag_id in self.dags:\n self.dags[dag_id] += dag\n else:\n self.dags[dag_id] = dag\n\n # Add anything that might be in the tasks of the DAG - such as imports, Connections, etc\n _add_recursively((dag.tasks or {}).values())\n\n # Add anything that might be in the `dag.schedule` - such as Includes, Timetables, Connections, etc\n _add_recursively([dag])\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_env_vars","title":"add_env_vars","text":"add_env_vars(\n env_vars: OrbiterEnvVar | Iterable[OrbiterEnvVar],\n) -> \"OrbiterProject\"\n
Add OrbiterEnvVars to the Project or override an existing env var with new properties >>> OrbiterProject().add_env_vars(OrbiterEnvVar(key=\"foo\", value=\"bar\")).env_vars\n{'foo': OrbiterEnvVar(key='foo', value='bar')}\n\n>>> OrbiterProject().add_env_vars([OrbiterEnvVar(key=\"foo\", value=\"bar\")]).env_vars\n{'foo': OrbiterEnvVar(key='foo', value='bar')}\n
Tip Validation requires an OrbiterEnvVar to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_env_vars('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n\n>>> # noinspection PyTypeChecker\n... OrbiterProject().add_env_vars(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description env_vars OrbiterEnvVar | Iterable[OrbiterEnvVar] List of OrbiterEnvVar Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_env_vars(\n self, env_vars: OrbiterEnvVar | Iterable[OrbiterEnvVar]\n) -> \"OrbiterProject\":\n \"\"\"\n Add [OrbiterEnvVars][orbiter.objects.env_var.OrbiterEnvVar] to the Project\n or override an existing env var with new properties\n\n ```pycon\n >>> OrbiterProject().add_env_vars(OrbiterEnvVar(key=\"foo\", value=\"bar\")).env_vars\n {'foo': OrbiterEnvVar(key='foo', value='bar')}\n\n >>> OrbiterProject().add_env_vars([OrbiterEnvVar(key=\"foo\", value=\"bar\")]).env_vars\n {'foo': OrbiterEnvVar(key='foo', value='bar')}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterEnvVar` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_env_vars('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_env_vars(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n\n :param env_vars: List of [OrbiterEnvVar][orbiter.objects.env_var.OrbiterEnvVar]\n :type env_vars: List[OrbiterEnvVar] | OrbiterEnvVar\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for env_var in [env_vars] if isinstance(env_vars, OrbiterEnvVar) else env_vars:\n self.env_vars[env_var.key] = env_var\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_includes","title":"add_includes","text":"add_includes(\n includes: OrbiterInclude | Iterable[OrbiterInclude],\n) -> \"OrbiterProject\"\n
Add OrbiterIncludes to the Project or override an existing OrbiterInclude with new properties >>> OrbiterProject().add_includes(OrbiterInclude(filepath=\"foo\", contents=\"bar\")).includes\n{'foo': OrbiterInclude(filepath='foo', contents='bar')}\n\n>>> OrbiterProject().add_includes([OrbiterInclude(filepath=\"foo\", contents=\"bar\")]).includes\n{'foo': OrbiterInclude(filepath='foo', contents='bar')}\n
Tip Validation requires an OrbiterInclude to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_includes('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n\n>>> # noinspection PyTypeChecker\n... OrbiterProject().add_includes(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description includes OrbiterInclude | Iterable[OrbiterInclude] List of OrbiterIncludes Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_includes(\n self, includes: OrbiterInclude | Iterable[OrbiterInclude]\n) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterIncludes][orbiter.objects.include.OrbiterInclude] to the Project\n or override an existing [OrbiterInclude][orbiter.objects.include.OrbiterInclude] with new properties\n\n ```pycon\n >>> OrbiterProject().add_includes(OrbiterInclude(filepath=\"foo\", contents=\"bar\")).includes\n {'foo': OrbiterInclude(filepath='foo', contents='bar')}\n\n >>> OrbiterProject().add_includes([OrbiterInclude(filepath=\"foo\", contents=\"bar\")]).includes\n {'foo': OrbiterInclude(filepath='foo', contents='bar')}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterInclude` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_includes('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_includes(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param includes: List of [OrbiterIncludes][orbiter.objects.include.OrbiterInclude]\n :type includes: List[OrbiterInclude]\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for include in [includes] if isinstance(includes, OrbiterInclude) else includes:\n self.includes[include.filepath] = include\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_pools","title":"add_pools","text":"add_pools(\n pools: OrbiterPool | Iterable[OrbiterPool],\n) -> \"OrbiterProject\"\n
Add OrbiterPool to the Project or override existing pools with new properties >>> OrbiterProject().add_pools(OrbiterPool(name=\"foo\", slots=1)).pools\n{'foo': OrbiterPool(name='foo', description='', slots=1)}\n\n>>> ( OrbiterProject()\n... .add_pools([OrbiterPool(name=\"foo\", slots=1)])\n... .add_pools([OrbiterPool(name=\"foo\", slots=2)])\n... .pools\n... )\n{'foo': OrbiterPool(name='foo', description='', slots=2)}\n
Tip Validation requires an OrbiterPool to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_pools('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n... OrbiterProject().add_pools(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description pools OrbiterPool | Iterable[OrbiterPool] List of OrbiterPools Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_pools(self, pools: OrbiterPool | Iterable[OrbiterPool]) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterPool][orbiter.objects.pool.OrbiterPool] to the Project\n or override existing pools with new properties\n\n ```pycon\n >>> OrbiterProject().add_pools(OrbiterPool(name=\"foo\", slots=1)).pools\n {'foo': OrbiterPool(name='foo', description='', slots=1)}\n\n >>> ( OrbiterProject()\n ... .add_pools([OrbiterPool(name=\"foo\", slots=1)])\n ... .add_pools([OrbiterPool(name=\"foo\", slots=2)])\n ... .pools\n ... )\n {'foo': OrbiterPool(name='foo', description='', slots=2)}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterPool` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_pools('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_pools(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param pools: List of [OrbiterPools][orbiter.objects.pool.OrbiterPool]\n :type pools: List[OrbiterPool] | OrbiterPool\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for pool in [pools] if isinstance(pools, OrbiterPool) else pools:\n if pool.name in self.pools:\n self.pools[pool.name] += pool\n else:\n self.pools[pool.name] = pool\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_requirements","title":"add_requirements","text":"add_requirements(\n requirements: (\n OrbiterRequirement | Iterable[OrbiterRequirement]\n ),\n) -> \"OrbiterProject\"\n
Add OrbiterRequirements to the Project or override an existing requirement with new properties >>> OrbiterProject().add_requirements(\n... OrbiterRequirement(package=\"apache-airflow\", names=['foo'], module='bar'),\n... ).requirements\n{OrbiterRequirements(names=[foo], package=apache-airflow, module=bar, sys_package=None)}\n\n>>> OrbiterProject().add_requirements(\n... [OrbiterRequirement(package=\"apache-airflow\", names=['foo'], module='bar')],\n... ).requirements\n{OrbiterRequirements(names=[foo], package=apache-airflow, module=bar, sys_package=None)}\n
Tip Validation requires an OrbiterRequirement to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_requirements('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n>>> OrbiterProject().add_requirements(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description requirements OrbiterRequirement | Iterable[OrbiterRequirement] List of OrbiterRequirements Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_requirements(\n self, requirements: OrbiterRequirement | Iterable[OrbiterRequirement]\n) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterRequirements][orbiter.objects.requirement.OrbiterRequirement] to the Project\n or override an existing requirement with new properties\n\n ```pycon\n >>> OrbiterProject().add_requirements(\n ... OrbiterRequirement(package=\"apache-airflow\", names=['foo'], module='bar'),\n ... ).requirements\n {OrbiterRequirements(names=[foo], package=apache-airflow, module=bar, sys_package=None)}\n\n >>> OrbiterProject().add_requirements(\n ... [OrbiterRequirement(package=\"apache-airflow\", names=['foo'], module='bar')],\n ... ).requirements\n {OrbiterRequirements(names=[foo], package=apache-airflow, module=bar, sys_package=None)}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterRequirement` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_requirements('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n >>> OrbiterProject().add_requirements(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param requirements: List of [OrbiterRequirements][orbiter.objects.requirement.OrbiterRequirement]\n :type requirements: List[OrbiterRequirement] | OrbiterRequirement\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for requirement in (\n [requirements]\n if isinstance(requirements, OrbiterRequirement)\n else requirements\n ):\n self.requirements.add(requirement)\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_variables","title":"add_variables","text":"add_variables(\n variables: OrbiterVariable | Iterable[OrbiterVariable],\n) -> \"OrbiterProject\"\n
Add OrbiterVariables to the Project or override an existing variable with new properties >>> OrbiterProject().add_variables(OrbiterVariable(key=\"foo\", value=\"bar\")).variables\n{'foo': OrbiterVariable(key='foo', value='bar')}\n\n>>> OrbiterProject().add_variables([OrbiterVariable(key=\"foo\", value=\"bar\")]).variables\n{'foo': OrbiterVariable(key='foo', value='bar')}\n
Tip Validation requires an OrbiterVariable to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_variables('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n... OrbiterProject().add_variables(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description variables OrbiterVariable | Iterable[OrbiterVariable] List of OrbiterVariable Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_variables(\n self, variables: OrbiterVariable | Iterable[OrbiterVariable]\n) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterVariables][orbiter.objects.variable.OrbiterVariable] to the Project\n or override an existing variable with new properties\n\n ```pycon\n >>> OrbiterProject().add_variables(OrbiterVariable(key=\"foo\", value=\"bar\")).variables\n {'foo': OrbiterVariable(key='foo', value='bar')}\n\n >>> OrbiterProject().add_variables([OrbiterVariable(key=\"foo\", value=\"bar\")]).variables\n {'foo': OrbiterVariable(key='foo', value='bar')}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterVariable` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_variables('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_variables(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param variables: List of [OrbiterVariable][orbiter.objects.variable.OrbiterVariable]\n :type variables: List[OrbiterVariable] | OrbiterVariable\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for variable in (\n [variables] if isinstance(variables, OrbiterVariable) else variables\n ):\n self.variables[variable.key] = variable\n return self\n
"},{"location":"objects/workflow/","title":"Workflow","text":"Airflow workflows are represented by a DAG which is a Directed Acyclic Graph (of Tasks). "},{"location":"objects/workflow/#diagram","title":"Diagram","text":"classDiagram\n direction LR\n class OrbiterRequirement[\"orbiter.objects.requirement.OrbiterRequirement\"] {\n package: str | None\n module: str | None\n names: List[str] | None\n sys_package: str | None\n }\n\n OrbiterDAG \"via schedule\" --> OrbiterTimetable\n OrbiterDAG --> \"many\" OrbiterOperator\n OrbiterDAG --> \"many\" OrbiterTaskGroup\n OrbiterDAG --> \"many\" OrbiterRequirement\n class OrbiterDAG[\"orbiter.objects.dag.OrbiterDAG\"] {\n imports: List[OrbiterRequirement]\n file_path: str\n dag_id: str\n schedule: str | OrbiterTimetable | None\n catchup: bool\n start_date: DateTime\n tags: List[str]\n default_args: Dict[str, Any]\n params: Dict[str, Any]\n doc_md: str | None\n tasks: Dict[str, OrbiterOperator]\n kwargs: dict\n orbiter_kwargs: dict\n orbiter_conns: Set[OrbiterConnection]\n orbiter_vars: Set[OrbiterVariable]\n orbiter_env_vars: Set[OrbiterEnvVar]\n orbiter_includes: Set[OrbiterInclude]\n }\n click OrbiterDAG href \"#orbiter.objects.dag.OrbiterDAG\" \"OrbiterDAG Documentation\"\n\n OrbiterTaskGroup --> \"many\" OrbiterRequirement\n class OrbiterTaskGroup[\"orbiter.objects.task.OrbiterTaskGroup\"] {\n task_group_id: str\n tasks: List[OrbiterOperator | OrbiterTaskGroup]\n add_downstream(str | List[str] | OrbiterTaskDependency)\n }\n\n OrbiterOperator --> \"many\" OrbiterRequirement\n OrbiterOperator --> \"one\" OrbiterPool\n OrbiterOperator --> \"many\" OrbiterConnection\n OrbiterOperator --> \"many\" OrbiterVariable\n OrbiterOperator --> \"many\" OrbiterEnvVar\n class OrbiterOperator[\"orbiter.objects.task.OrbiterOperator\"] {\n imports: List[OrbiterRequirement]\n operator: str\n task_id: str\n pool: str | None\n pool_slots: int | None\n trigger_rule: str | None\n downstream: Set[str]\n add_downstream(str | List[str] | OrbiterTaskDependency)\n }\n\n class OrbiterTimetable[\"orbiter.objects.timetables.OrbiterTimetable\"] {\n imports: List[OrbiterRequirements]\n orbiter_includes: Set[OrbiterIncludes]\n **kwargs: dict\n }\n click OrbiterTimetable href \"#orbiter.objects.timetables.OrbiterTimetable\" \"OrbiterTimetable Documentation\"\n\n class OrbiterConnection[\"orbiter.objects.connection.OrbiterConnection\"] {\n conn_id: str\n conn_type: str\n **kwargs\n }\n\n class OrbiterEnvVar[\"orbiter.objects.env_var.OrbiterEnvVar\"] {\n key: str\n value: str\n }\n\n class OrbiterPool[\"orbiter.objects.pool.OrbiterPool\"] {\n name: str\n description: str | None\n slots: int | None\n }\n\n class OrbiterRequirement[\"orbiter.objects.requirement.OrbiterRequirement\"] {\n package: str | None\n module: str | None\n names: List[str] | None\n sys_package: str | None\n }\n\n class OrbiterVariable[\"orbiter.objects.variable.OrbiterVariable\"] {\n key: str\n value: str\n }
"},{"location":"objects/workflow/#orbiter.objects.dag.OrbiterDAG","title":"orbiter.objects.dag.OrbiterDAG","text":"Represents an Airflow DAG, with its tasks and dependencies. Renders to a .py file in the /dags folder Parameters: Name Type Description file_path str File path of the DAG, relative to the /dags folder (filepath=my_dag.py would render to dags/my_dag.py) dag_id str The dag_id. Must be unique and snake_case. Good practice is to set dag_id == file_path schedule str | OrbiterTimetable, optional The schedule for the DAG. Defaults to None (only runs when manually triggered) catchup bool, optional Whether to catchup runs from the start_date to now, on first run. Defaults to False start_date DateTime, optional The start date for the DAG. Defaults to Unix Epoch tags List[str], optional Tags for the DAG, used for sorting and filtering in the Airflow UI default_args Dict[str, Any], optional Default arguments for any tasks in the DAG params Dict[str, Any], optional Params for the DAG doc_md str, optional Documentation for the DAG with markdown support kwargs dict, optional Additional keyword arguments to pass to the DAG **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/workflow/#orbiter.objects.dag.OrbiterDAG.add_tasks","title":"add_tasks","text":"add_tasks(\n tasks: (\n OrbiterOperator\n | OrbiterTaskGroup\n | Iterable[OrbiterOperator | OrbiterTaskGroup]\n ),\n) -> \"OrbiterDAG\"\n
Add one or more OrbiterOperators to the DAG >>> from orbiter.objects.operators.empty import OrbiterEmptyOperator\n>>> OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks(OrbiterEmptyOperator(task_id=\"bar\")).tasks\n{'bar': bar_task = EmptyOperator(task_id='bar')}\n\n>>> OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks([OrbiterEmptyOperator(task_id=\"bar\")]).tasks\n{'bar': bar_task = EmptyOperator(task_id='bar')}\n
Tip Validation requires a OrbiterTaskGroup, OrbiterOperator (or subclass), or list of either to be passed >>> # noinspection PyTypeChecker\n... OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks(\"bar\")\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n... OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks([\"bar\"])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description tasks OrbiterOperator | OrbiterTaskGroup | Iterable[OrbiterOperator | OrbiterTaskGroup] List of OrbiterOperator, or OrbiterTaskGroup or subclass Returns: Type Description OrbiterProject self Source code in orbiter/objects/dag.py def add_tasks(\n self,\n tasks: (\n OrbiterOperator\n | OrbiterTaskGroup\n | Iterable[OrbiterOperator | OrbiterTaskGroup]\n ),\n) -> \"OrbiterDAG\":\n \"\"\"\n Add one or more [`OrbiterOperators`][orbiter.objects.task.OrbiterOperator] to the DAG\n\n ```pycon\n >>> from orbiter.objects.operators.empty import OrbiterEmptyOperator\n >>> OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks(OrbiterEmptyOperator(task_id=\"bar\")).tasks\n {'bar': bar_task = EmptyOperator(task_id='bar')}\n\n >>> OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks([OrbiterEmptyOperator(task_id=\"bar\")]).tasks\n {'bar': bar_task = EmptyOperator(task_id='bar')}\n\n ```\n\n !!! tip\n\n Validation requires a `OrbiterTaskGroup`, `OrbiterOperator` (or subclass), or list of either to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks(\"bar\")\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n ... OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks([\"bar\"])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param tasks: List of [OrbiterOperator][orbiter.objects.task.OrbiterOperator], or OrbiterTaskGroup or subclass\n :type tasks: OrbiterOperator | OrbiterTaskGroup | Iterable[OrbiterOperator | OrbiterTaskGroup]\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n if (\n isinstance(tasks, OrbiterOperator)\n or isinstance(tasks, OrbiterTaskGroup)\n or issubclass(type(tasks), OrbiterOperator)\n ):\n tasks = [tasks]\n\n for task in tasks:\n try:\n task_id = getattr(task, \"task_id\", None) or getattr(\n task, \"task_group_id\"\n )\n except AttributeError:\n raise AttributeError(\n f\"Task {task} does not have a task_id or task_group_id attribute\"\n )\n self.tasks[task_id] = task\n return self\n
"},{"location":"objects/workflow/#timetables","title":"Timetables","text":""},{"location":"objects/workflow/#orbiter.objects.timetables.OrbiterTimetable","title":"orbiter.objects.timetables.OrbiterTimetable","text":"An Airflow Timetable reference. Utilizes OrbiterInclude to add a file to a /plugins folder to register the timetable. Parameters: Name Type Description **kwargs any other kwargs to provide to Timetable "},{"location":"objects/workflow/#orbiter.objects.timetables.multi_cron_timetable","title":"orbiter.objects.timetables.multi_cron_timetable","text":""},{"location":"objects/workflow/#orbiter.objects.timetables.multi_cron_timetable.OrbiterMultiCronTimetable","title":"orbiter.objects.timetables.multi_cron_timetable.OrbiterMultiCronTimetable","text":"An Airflow Timetable that can be supplied with multiple cron strings. >>> OrbiterMultiCronTimetable(cron_defs=[\"*/5 * * * *\", \"*/7 * * * *\"])\nMultiCronTimetable(cron_defs=['*/5 * * * *', '*/7 * * * *'])\n
Parameters: Name Type Description cron_defs List[str] A list of cron strings timezone str The timezone to use for the timetable period_length int The length of the period period_unit str The unit of the period "},{"location":"objects/Tasks/","title":"Overview","text":"Airflow Tasks are units of work. An Operator is a pre-defined task with specific functionality. Operators can be looked up in the Astronomer Registry. The easiest way to create an operator in a translation to use an existing subclass of OrbiterOperator (e.g. OrbiterBashOperator). If an OrbiterOperator subclass doesn't exist for your use case, you can: 1) Utilize OrbiterTask from orbiter.objects.requirement import OrbiterRequirement\nfrom orbiter.objects.task import OrbiterTask\nfrom orbiter.rules import task_rule\n\n@task_rule\ndef my_rule(val: dict):\n return OrbiterTask(\n task_id=\"my_task\",\n imports=[OrbiterRequirement(\n package=\"apache-airflow\",\n module=\"airflow.operators.trigger_dagrun\",\n names=[\"TriggerDagRunOperator\"],\n )],\n ...\n )\n
2) Create a new subclass of OrbiterOperator, which can be beneficial if you are using it frequently in separate @task_rules from orbiter.objects.task import OrbiterOperator\nfrom orbiter.objects.requirement import OrbiterRequirement\nfrom orbiter.rules import task_rule\n\nclass OrbiterTriggerDagRunOperator(OrbiterOperator):\n # Define the imports required for the operator, and the operator name\n imports = [\n OrbiterRequirement(\n package=\"apache-airflow\",\n module=\"airflow.operators.trigger_dagrun\",\n names=[\"TriggerDagRunOperator\"],\n )\n ]\n operator: str = \"PythonOperator\"\n\n # Add fields should be rendered in the output\n render_attributes = OrbiterOperator.render_attributes + [\n ...\n ]\n\n # Add the fields that are required for the operator here, with their types\n # Not all Airflow Operator fields are required, just the ones you will use.\n trigger_dag_id: str\n ...\n\n@task_rule\ndef my_rule(val: dict):\n return OrbiterTriggerDagRunOperator(...)\n
"},{"location":"objects/Tasks/#diagram","title":"Diagram","text":"classDiagram\n direction LR\n OrbiterOperator \"implements\" <|-- OrbiterTask\n OrbiterOperator --> \"many\" OrbiterCallback\n class OrbiterOperator[\"orbiter.objects.task.OrbiterOperator\"] {\n imports: List[OrbiterRequirement]\n operator: str\n task_id: str\n pool: str | None\n pool_slots: int | None\n trigger_rule: str | None\n downstream: Set[str]\n add_downstream(str | List[str] | OrbiterTaskDependency)\n }\n click OrbiterOperator href \"#orbiter.objects.task.OrbiterOperator\" \"OrbiterOperator Documentation\"\n\n class OrbiterTask[\"orbiter.objects.task.OrbiterTask\"] {\n <<OrbiterOperator>>\n <<OrbiterOperator>>\n imports: List[OrbiterRequirement]\n task_id: str\n **kwargs\n }\n click OrbiterTask href \"#orbiter.objects.task.OrbiterTask\" \"OrbiterTask Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterBashOperator\n class OrbiterBashOperator[\"orbiter.objects.operators.bash.OrbiterBashOperator\"] {\n <<OrbiterOperator>>\n operator = \"BashOperator\"\n task_id: str\n bash_command: str\n }\n click OrbiterBashOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.bash.OrbiterBashOperator\" \"OrbiterBashOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterEmailOperator\n class OrbiterEmailOperator[\"orbiter.objects.operators.smtp.OrbiterEmailOperator\"] {\n <<OrbiterOperator>>\n operator = \"EmailOperator\"\n task_id: str\n to: str | list[str]\n subject: str\n html_content: str\n files: list | None\n conn_id: str\n }\n click OrbiterEmailOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.smtp.OrbiterEmailOperator\" \"OrbiterEmailOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterEmptyOperator\n class OrbiterEmptyOperator[\"orbiter.objects.operators.empty.OrbiterEmptyOperator\"] {\n <<OrbiterOperator>>\n operator = \"BashOperator\"\n task_id: str\n }\n click OrbiterEmptyOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.empty.OrbiterEmptyOperator\" \"OrbiterEmptyOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterPythonOperator\n class OrbiterPythonOperator[\"orbiter.objects.operators.python.OrbiterPythonOperator\"] {\n <<OrbiterOperator>>\n operator = \"PythonOperator\"\n task_id: str\n python_callable: Callable\n op_args: list | None\n op_kwargs: dict | None\n }\n click OrbiterPythonOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.python.OrbiterPythonOperator\" \"OrbiterPythonOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterSQLExecuteQueryOperator\n class OrbiterSQLExecuteQueryOperator[\"orbiter.objects.operators.sql.OrbiterSQLExecuteQueryOperator\"] {\n <<OrbiterOperator>>\n operator = \"SQLExecuteQueryOperator\"\n task_id: str\n conn_id: str\n sql: str\n }\n click OrbiterSQLExecuteQueryOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.sql.OrbiterSQLExecuteQueryOperator\" \"OrbiterSQLExecuteQueryOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterSSHOperator\n class OrbiterSSHOperator[\"orbiter.objects.operators.ssh.OrbiterSSHOperator\"] {\n <<OrbiterOperator>>\n operator = \"SSHOperator\"\n task_id: str\n ssh_conn_id: str\n command: str\n environment: Dict[str, str] | None\n }\n click OrbiterSSHOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.ssh.OrbiterSSHOperator\" \"OrbiterSSHOperator Documentation\"\n\n class OrbiterCallback[\"orbiter.objects.callbacks.OrbiterCallback\"] {\n function: str\n }\n click OrbiterCallback href \"Operators_and_Callbacks/callbacks#orbiter.objects.callbacks.OrbiterCallback\" \"OrbiterCallback Documentation\"\n\n OrbiterCallback \"implements\" <|-- OrbiterSmtpNotifierCallback\n class OrbiterSmtpNotifierCallback[\"orbiter.objects.callbacks.smtp.OrbiterSmtpNotifierCallback\"] {\n <<OrbiterCallback>>\n to: str\n from_email: str\n smtp_conn_id: str\n subject: str\n html_content: str\n cc: str | Iterable[str]\n }\n click OrbiterSmtpNotifierCallback href \"Operators_and_Callbacks/callbacks#orbiter.objects.callbacks.smtp.OrbiterSmtpNotifierCallback\" \"OrbiterSmtpNotifierCallback Documentation\"
"},{"location":"objects/Tasks/#orbiter.objects.task.OrbiterOperator","title":"orbiter.objects.task.OrbiterOperator","text":"Abstract class representing a Task in Airflow. Must be subclassed (such as OrbiterBashOperator, or OrbiterTask). Subclassing Example: >>> from orbiter.objects import OrbiterRequirement\n>>> class OrbiterMyOperator(OrbiterOperator):\n... imports: ImportList = [OrbiterRequirement(package=\"apache-airflow\")]\n... operator: str = \"MyOperator\"\n\n>>> foo = OrbiterMyOperator(task_id=\"task_id\"); foo\ntask_id_task = MyOperator(task_id='task_id')\n
Adding single downstream tasks: >>> from orbiter.ast_helper import render_ast\n>>> render_ast(foo.add_downstream(\"downstream\")._downstream_to_ast())\n'task_id_task >> downstream_task'\n
Adding multiple downstream tasks: >>> render_ast(foo.add_downstream([\"a\", \"b\"])._downstream_to_ast())\n'task_id_task >> [a_task, b_task, downstream_task]'\n
Note Validation - task_id in OrbiterTaskDependency must match this task_id >>> foo.add_downstream(OrbiterTaskDependency(task_id=\"other\", downstream=\"bar\")).downstream\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\nValueError: Task dependency ... has a different task_id than task_id\n
Parameters: Name Type Description imports List[OrbiterRequirement] List of requirements for the operator task_id str The task_id for the operator, must be unique and snake_case trigger_rule str, optional Conditions under which to start the task (docs) pool str, optional Name of the pool to use pool_slots int, optional Slots for this task to occupy operator str, optional Operator name downstream Set[str], optional Downstream tasks, defaults to set() **kwargs Other properties that may be passed to operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/#orbiter.objects.task.OrbiterTaskDependency","title":"orbiter.objects.task.OrbiterTaskDependency","text":"Represents a task dependency, which is added to either an OrbiterOperator or an OrbiterTaskGroup. Parameters: Name Type Description task_id str The task_id for the operator downstream str | List[str] downstream task(s) "},{"location":"objects/Tasks/#orbiter.objects.task.OrbiterTask","title":"orbiter.objects.task.OrbiterTask","text":"A generic version of OrbiterOperator that can be instantiated directly. The operator that is instantiated is inferred from the imports field, via the first *Operator or *Sensor import. View info for specific operators at the Astronomer Registry. >>> from orbiter.objects.requirement import OrbiterRequirement\n>>> OrbiterTask(task_id=\"foo\", bash_command=\"echo 'hello world'\", other=1, imports=[\n... OrbiterRequirement(package=\"apache-airflow\", module=\"airflow.operators.bash\", names=[\"BashOperator\"])\n... ])\nfoo_task = BashOperator(task_id='foo', bash_command=\"echo 'hello world'\", other=1)\n\n>>> def foo():\n... pass\n>>> OrbiterTask(task_id=\"foo\", python_callable=foo, other=1, imports=[\n... OrbiterRequirement(package=\"apache-airflow\", module=\"airflow.sensors.python\", names=[\"PythonSensor\"])\n... ])\ndef foo():\n pass\nfoo_task = PythonSensor(task_id='foo', other=1, python_callable=foo)\n
Parameters: Name Type Description task_id str The task_id for the operator. Must be unique and snake_case imports List[OrbiterRequirement] List of requirements for the operator. The Operator is inferred from first *Operator or *Sensor imported. **kwargs Any other keyword arguments to be passed to the operator "},{"location":"objects/Tasks/#orbiter.objects.task_group.OrbiterTaskGroup","title":"orbiter.objects.task_group.OrbiterTaskGroup","text":"Represents a TaskGroup in Airflow, which contains multiple tasks >>> from orbiter.objects.operators.bash import OrbiterBashOperator\n>>> from orbiter.ast_helper import render_ast\n>>> OrbiterTaskGroup(task_group_id=\"foo\", tasks=[\n... OrbiterBashOperator(task_id=\"b\", bash_command=\"b\"),\n... OrbiterBashOperator(task_id=\"a\", bash_command=\"a\").add_downstream(\"b\"),\n... ], downstream={\"c\"})\nwith TaskGroup(group_id='foo') as foo:\n b_task = BashOperator(task_id='b', bash_command='b')\n a_task = BashOperator(task_id='a', bash_command='a')\n a_task >> b_task\n\n>>> render_ast(OrbiterTaskGroup(task_group_id=\"foo\", tasks=[], downstream={\"c\"})._downstream_to_ast())\n'foo >> c_task'\n
Parameters: Name Type Description task_group_id str The id of the TaskGroup tasks List[OrbiterOperator | OrbiterTaskGroup] The tasks in the TaskGroup **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/callbacks/","title":"Callbacks","text":"Airflow callback functions are often used to send emails, slack messages, or other notifications when a task fails, succeeds, or is retried. They can also run any general Python function. "},{"location":"objects/Tasks/Operators_and_Callbacks/callbacks/#orbiter.objects.callbacks.OrbiterCallback","title":"orbiter.objects.callbacks.OrbiterCallback","text":"Represents an Airflow callback function, which might be used in DAG.on_failure_callback, or Task.on_success_callback, or etc. Can be instantiated directly as a bare callback function (with no arguments): >>> from orbiter.objects.dag import OrbiterDAG\n>>> from orbiter.objects.include import OrbiterInclude\n>>> my_callback = OrbiterCallback(\n... function=\"my_callback\",\n... imports=[OrbiterRequirement(module=\"my_callback\", names=[\"my_callback\"])],\n... orbiter_includes={OrbiterInclude(filepath=\"my_callback.py\", contents=\"...\")}\n... )\n>>> OrbiterDAG(dag_id='', file_path='', on_failure_callback=my_callback)\n... # doctest: +ELLIPSIS\nfrom airflow import DAG\nfrom my_callback import my_callback\n...\nwith DAG(... on_failure_callback=my_callback):\n
or be subclassed: >>> class OrbiterMyCallback(OrbiterCallback):\n... function: str = \"my_callback\"\n... foo: str\n... bar: str\n... render_attributes: RenderAttributes = [\"foo\", \"bar\"]\n>>> OrbiterMyCallback(foo=\"fop\", bar=\"bop\")\nmy_callback(foo='fop', bar='bop')\n
Parameters: Name Type Description function str The name of the function to call **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/callbacks/#orbiter.objects.callbacks.smtp","title":"orbiter.objects.callbacks.smtp","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/callbacks/#orbiter.objects.callbacks.smtp.OrbiterSmtpNotifierCallback","title":"orbiter.objects.callbacks.smtp.OrbiterSmtpNotifierCallback","text":"An Airflow SMTP Callback Note Use smtp_conn_id and reference an SMTP Connection. You can use the **conn_id(\"SMTP\", conn_type=\"smtp\") utility function to set both properties at once. >>> [_import] = OrbiterSmtpNotifierCallback(to=\"foo@test.com\").imports; _import\nOrbiterRequirements(names=[send_smtp_notification], package=apache-airflow-providers-smtp, module=airflow.providers.smtp.notifications.smtp, sys_package=None)\n\n>>> OrbiterSmtpNotifierCallback(to=\"foo@test.com\", from_email=\"bar@test.com\", subject=\"Hello\", html_content=\"World\")\nsend_smtp_notification(to='foo@test.com', from_email='bar@test.com', smtp_conn_id='SMTP', subject='Hello', html_content='World')\n
Parameters: Name Type Description to str | Iterable[str] The email address to send to from_email str, optional The email address to send from smtp_conn_id str, optional The connection id to use (Note: use the **conn_id(...) utility function). Defaults to \"SMTP\" subject str, optional The subject of the email html_content str, optional The content of the email cc str | Iterable[str], optional The email address to cc **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/","title":"Operators","text":"Note These operators are included and are intended to represent some of the most common Airflow Operators, but not all Airflow Operators. Additional Operators can be created by subclassing OrbiterOperator or using OrbiterTask directly. Review the Astronomer Registry to find additional Airflow Operators. "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.bash","title":"orbiter.objects.operators.bash","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.bash.OrbiterBashOperator","title":"orbiter.objects.operators.bash.OrbiterBashOperator","text":" Bases: OrbiterOperator An Airflow BashOperator. Used to run shell commands. >>> OrbiterBashOperator(task_id=\"foo\", bash_command=\"echo 'hello world'\")\nfoo_task = BashOperator(task_id='foo', bash_command=\"echo 'hello world'\")\n
Parameters: Name Type Description task_id str The task_id for the operator bash_command str The shell command to execute **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.empty","title":"orbiter.objects.operators.empty","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.empty.OrbiterEmptyOperator","title":"orbiter.objects.operators.empty.OrbiterEmptyOperator","text":" Bases: OrbiterOperator An Airflow EmptyOperator. Does nothing. >>> OrbiterEmptyOperator(task_id=\"foo\")\nfoo_task = EmptyOperator(task_id='foo')\n
Parameters: Name Type Description task_id str The task_id for the operator **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.python","title":"orbiter.objects.operators.python","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.python.OrbiterPythonOperator","title":"orbiter.objects.operators.python.OrbiterPythonOperator","text":" Bases: OrbiterOperator An Airflow PythonOperator. Used to execute any Python Function. >>> def foo(a, b):\n... print(a + b)\n>>> OrbiterPythonOperator(task_id=\"foo\", python_callable=foo)\ndef foo(a, b):\n print(a + b)\nfoo_task = PythonOperator(task_id='foo', python_callable=foo)\n
Parameters: Name Type Description task_id str The task_id for the operator python_callable Callable The python function to execute op_args list | None, optional The arguments to pass to the python function, defaults to None op_kwargs dict | None, optional The keyword arguments to pass to the python function, defaults to None **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.smtp","title":"orbiter.objects.operators.smtp","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.smtp.OrbiterEmailOperator","title":"orbiter.objects.operators.smtp.OrbiterEmailOperator","text":" Bases: OrbiterOperator An Airflow EmailOperator. Used to send emails. >>> OrbiterEmailOperator(\n... task_id=\"foo\", to=\"humans@astronomer.io\", subject=\"Hello\", html_content=\"World!\"\n... )\nfoo_task = EmailOperator(task_id='foo', to='humans@astronomer.io', subject='Hello', html_content='World!', conn_id='SMTP')\n
Parameters: Name Type Description task_id str The task_id for the operator to str | list[str] The recipient of the email subject str The subject of the email html_content str The content of the email files list, optional The files to attach to the email, defaults to None conn_id str, optional The SMTP connection to use. Defaults to \"SMTP\" and sets orbiter_conns property **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.sql","title":"orbiter.objects.operators.sql","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.sql.OrbiterSQLExecuteQueryOperator","title":"orbiter.objects.operators.sql.OrbiterSQLExecuteQueryOperator","text":" Bases: OrbiterOperator An Airflow Generic SQL Operator. Used to run SQL against any Database. >>> OrbiterSQLExecuteQueryOperator(\n... task_id=\"foo\", conn_id='sql', sql=\"select 1;\"\n... )\nfoo_task = SQLExecuteQueryOperator(task_id='foo', conn_id='sql', sql='select 1;')\n
Parameters: Name Type Description task_id str The task_id for the operator conn_id str The SQL connection to utilize. (Note: use the **conn_id(...) utility function) sql str The SQL to execute **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.ssh","title":"orbiter.objects.operators.ssh","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.ssh.OrbiterSSHOperator","title":"orbiter.objects.operators.ssh.OrbiterSSHOperator","text":" Bases: OrbiterOperator An Airflow SSHOperator. Used to run shell commands over SSH. >>> OrbiterSSHOperator(task_id=\"foo\", ssh_conn_id=\"SSH\", command=\"echo 'hello world'\")\nfoo_task = SSHOperator(task_id='foo', ssh_conn_id='SSH', command=\"echo 'hello world'\")\n
Parameters: Name Type Description task_id str The task_id for the operator ssh_conn_id str The SSH connection to use. (Note: use the **conn_id(...) utility function) command str The command to execute environment dict, optional The environment variables to set, defaults to None **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":" Astronomer Orbiter can land legacy workloads safely down in a new home on Apache Airflow! "},{"location":"#what-is-orbiter","title":"What is Orbiter?","text":"Orbiter is both a CLI and Framework for converting workflows from other orchestration tools to Apache Airflow. Generally it can be thoughts of as: flowchart LR\n origin{{ XML/JSON/YAML/Etc Workflows }}\n origin -->| \u2728 Translations \u2728 | airflow{{ Apache Airflow Project }}
The framework is a set of Rules and Objects that can translate workflows from an Origin system to an Airflow project."},{"location":"#installation","title":"Installation","text":"Install the orbiter CLI, if you have Python >= 3.10 installed via pip: pip install astronomer-orbiter\n
If you do not have a compatible Python environment, pre-built binary executables of the orbiter CLI are available for download on the Releases page."},{"location":"#translate","title":"Translate","text":"Utilize the orbiter CLI with existing translations to convert workflows from other systems to an Airflow project. - Set up a new folder, and create a
workflow/ folder. Add your workflows files to it .\n\u2514\u2500\u2500 workflow/\n \u251c\u2500\u2500 workflow_a.json\n \u251c\u2500\u2500 workflow_b.json\n \u2514\u2500\u2500 ...\n
- Determine the specific translation ruleset via:
- the Origins documentation
- the
orbiter list-rulesets command - or by creating a translation ruleset, if one does not exist
- Install the translation ruleset via the
orbiter install command (substituting <REPOSITORY> with the value in the last step) orbiter install --repo=<REPOSITORY>\n
- Use the
orbiter translate command with the <RULESET> determined in the last step This will produce output to an output/ folder: orbiter translate workflow/ --ruleset <RULESET> output/\n
- Review the contents of the
output/ folder. If extensions or customizations are required, review how to extend a translation ruleset - (optional) Utilize the
astro CLI to run Airflow instance with your migrated workloads - (optional) Deploy to Astro to run your translated workflows in production! \ud83d\ude80
"},{"location":"#authoring-rulesets-customization","title":"Authoring Rulesets & Customization","text":"Orbiter can be extended to fit specific needs, patterns, or to support additional origins. Read more specifics about how to use the framework at Rules and Objects "},{"location":"#extend-or-customize","title":"Extend or Customize","text":"To extend or customize an existing ruleset, you can easily modify it with simple Python code. - Set up your workspace as described in steps 1+2 of the Translate instructions
- Create a Python script, named
override.py .\n\u251c\u2500\u2500 override.py\n\u2514\u2500\u2500 workflow/\n \u251c\u2500\u2500 workflow_a.json\n \u251c\u2500\u2500 workflow_b.json\n \u2514\u2500\u2500 ...\n
-
Add contents to override.py: ```python title=\"override.py\" linenums=\"1\" from orbiter_community_translations.dag_factory import translation_ruleset # (1)! from orbiter.objects.operators.ssh import OrbiterSSHOperator # (2)! from orbiter.rules import task_rule # (3)! @task_rule(priority=99) # (4)! def ssh_rule(val: dict): \"\"\"Demonstration of overriding rulesets, by switching DAG Factory BashOperators to SSHOperators\"\"\" if val.pop(\"operator\", \"\") == \"BashOperator\": # (5)! return OrbiterSSHOperator( # (6)! command=val.pop(\"bash_command\"), doc=\"Hello World!\", **{k: v for k, v in val if k != \"dependencies\"}, ) else: return None translation_ruleset.task_ruleset.ruleset.append(ssh_rule) # (7)! `` 1. Importing specific translation ruleset, determined via the [Origins](origins.md) page 2. Importing required [Objects](./objects/index.md) 3. Importing required [Rule](./Rules_and_Rulesets/index.md) types 4. Create one or more@rulefunctions, as required. A higher priority means this rule will be applied first. [@task_ruleReference](./Rules_and_Rulesets/rules.md#orbiter.rules.TaskRule) 5.Ruleshave anif/elsestatement - they must always return a **single** thing or **nothing** 6. [OrbiterSSHOperatorReference](./objects/Tasks/Operators_and_Callbacks/operators.md#orbiter.objects.operators.ssh.OrbiterSSHOperator) 7. Append the new [Rule](./Rules_and_Rulesets/index.md) to the [translation_ruleset`](./Rules_and_Rulesets/rulesets.md#orbiter.rules.rulesets.TranslationRuleset) -
Invoke the orbiter CLI, pointing it at your customized ruleset, and writing output to an output/ folder: orbiter translate workflow/ output/ --ruleset override.translation_ruleset\n
- Follow the remaining steps of the Translate instructions
"},{"location":"#authoring-a-new-ruleset","title":"Authoring a new Ruleset","text":"You can utilize the TranslationRuleset Template to create a new TranslationRuleset. "},{"location":"#faq","title":"FAQ","text":" -
Can this tool convert my workflows from tool X to Airflow? If you don't see your tool listed in Supported Origins, contact us for services to create translations, create an issue in the orbiter-community-translations repository, or write a TranslationRuleset and submit a pull request to share your translations with the community. -
Are the results of this tool under any guarantee of correctness? No. This tool is provided as-is, with no guarantee of correctness. It is your responsibility to verify the results. We accept Pull Requests to improve parsing, and strive to make the tool easily configurable to handle your specific use-case. Artwork Orbiter logo by Ivan Colic used with permission from The Noun Project under Creative Commons. "},{"location":"cli/","title":"CLI","text":""},{"location":"cli/#orbiter","title":"orbiter","text":"Orbiter is a CLI that converts other workflows to Airflow Projects. Usage: orbiter [OPTIONS] COMMAND [ARGS]...\n
Options: Name Type Description Default --version boolean Show the version and exit. False --help boolean Show this message and exit. False"},{"location":"cli/#analyze","title":"analyze","text":"Analyze workflows in an INPUT_DIR Provide a specific ruleset with the --ruleset flag. Run orbiter list-rulesets to see available rulesets. INPUT_DIR defaults to $CWD/workflow.
Usage: orbiter analyze [OPTIONS] INPUT_DIR\n
Options: Name Type Description Default -r, --ruleset text Qualified name of a TranslationRuleset _required --format choice (json | csv | md) [optional] format for analysis output md -o, --output-file filename File to write to, defaults to '-' (stdout) - --help boolean Show this message and exit. False"},{"location":"cli/#install","title":"install","text":"Install a new Translation Ruleset from a repository Usage: orbiter install [OPTIONS]\n
Options: Name Type Description Default -r, --repo choice (astronomer-orbiter-translations | orbiter-community-translations) Choose a repository to install (will prompt, if not given) None -k, --key text [Optional] License Key to use for the translation ruleset. Should look like AAAA-BBBB-1111-2222-3333-XXXX-YYYY-ZZZZ None --help boolean Show this message and exit. False"},{"location":"cli/#list-rulesets","title":"list-rulesets","text":"List available Translation Rulesets Usage: orbiter list-rulesets [OPTIONS]\n
Options: Name Type Description Default --help boolean Show this message and exit. False"},{"location":"cli/#translate","title":"translate","text":"Translate workflows in an INPUT_DIR to an OUTPUT_DIR Airflow Project. Provide a specific ruleset with the --ruleset flag. Run orbiter list-rulesets to see available rulesets. INPUT_DIR defaults to $CWD/workflow.
OUTPUT_DIR defaults to $CWD/output
Formats output with Ruff (https://astral.sh/ruff), by default. Usage: orbiter translate [OPTIONS] INPUT_DIR OUTPUT_DIR\n
Options: Name Type Description Default -r, --ruleset text Qualified name of a TranslationRuleset _required --format / --no-format boolean [optional] format the output with Ruff True --help boolean Show this message and exit. False"},{"location":"cli/#logging","title":"Logging","text":"You can alter the verbosity of the CLI by setting the LOG_LEVEL environment variable. The default is INFO. export LOG_LEVEL=DEBUG\n
"},{"location":"origins/","title":"Origins","text":"An Origin is a source system that contains workflows that can be translated to an Apache Airflow project. "},{"location":"origins/#supported-origins","title":"Supported Origins","text":"Origin Maintainer Repository Ruleset DAG Equivalent Task Equivalent DAG Factory Community orbiter-community-translations orbiter_translations.dag_factory.yaml_base.translation_ruleset DAG Task Control M Astronomer astronomer-orbiter-translations orbiter_translations.control_m.json_base.translation_ruleset Folder Job \u2800\u2800 \u2800 \u2800 orbiter_translations.control_m.json_ssh.translation_ruleset \u2800 \u2800 \u2800 \u2800 \u2800 orbiter_translations.control_m.xml_base.translation_ruleset \u2800 \u2800 \u2800 \u2800 \u2800 orbiter_translations.control_m.xml_ssh.translation_ruleset \u2800 \u2800 \u2800 \u2800 orbiter-community-translations orbiter_translations.control_m.xml_demo.translation_ruleset \u2800 \u2800 Automic Astronomer astronomer-orbiter-translations WIP Job Plan Job Autosys Astronomer astronomer-orbiter-translations WIP \u2800 \u2800 JAMS Astronomer astronomer-orbiter-translations WIP Folder Job SSIS Astronomer astronomer-orbiter-translations orbiter_translations.ssis.xml_base.translation_ruleset Pipeline Component \u2800 \u2800 orbiter-community-translations orbiter_translations.oozie.xml_demo.translation_ruleset \u2800 \u2800 Oozie Astronomer astronomer-orbiter-translations orbiter_translations.oozie.xml_base.translation_ruleset Workflow Node \u2800 \u2800\u2800 orbiter-community-translations orbiter_translations.oozie.xml_demo.translation_ruleset \u2800 \u2800 & more! \u2800 \u2800 \u2800 \u2800 \u2800 For Astronomer maintained Translation Rulesets, please contact us for access to the most up-to-date versions. If you don't see your Origin system listed, please either: - contact us for services to create translations
- create an issue in our
orbiter-community-translations repository - write a
TranslationRuleset Template and submit a pull request to share your translations with the community "},{"location":"Rules_and_Rulesets/","title":"Overview","text":"The brain of the Orbiter framework is in it's Rules and the Rulesets that contain them. - A
Rule is a python function that is evaluated and produces something (typically an Object) or nothing - A
Ruleset is a collection of Rules that are evaluated in priority order - A
TranslationRuleset is a collection of Rulesets, relating to an Origin and FileType, with a translate_fn which determines how to apply the rulesets. Different Rules are applied in different scenarios, such as: - converting input to an Airflow DAG (
@dag_rule), - converting input to a specific Airflow Operator (
@task_rule), - filtering entries from the input data (
@dag_filter_rule, @task_filter_rule). Tip To map the following input {\n \"id\": \"my_task\",\n \"command\": \"echo 'hi'\"\n}\n
to an Airflow BashOperator, a Rule could parse it as follows: @task_rule\ndef my_rule(val):\n if 'command' in val:\n return OrbiterBashOperator(task_id=val['id'], bash_command=val['command'])\n else:\n return None\n
This returns a OrbiterBashOperator, which will become an Airflow BashOperator when the translation completes. "},{"location":"Rules_and_Rulesets/#orbiter.rules.rulesets.translate","title":"orbiter.rules.rulesets.translate","text":"translate(\n translation_ruleset, input_dir: Path\n) -> OrbiterProject\n
Orbiter expects a folder containing text files which may have a structure like: {\"<workflow name>\": { ...<workflow properties>, \"<task name>\": { ...<task properties>} }}\n
The standard translation function performs the following steps: - Find all files with the expected
TranslationRuleset.file_type (.json, .xml, .yaml, etc.) in the input folder. - Load each file and turn it into a Python Dictionary.
- For each file: Apply the
TranslationRuleset.dag_filter_ruleset to filter down to entries that can translate to a DAG, in priority order. The file name is added under a __file key. - For each dictionary: Apply the
TranslationRuleset.dag_ruleset, to convert the object to an OrbiterDAG, in priority-order, stopping when the first rule returns a match. If no rule returns a match, the entry is filtered. - Apply the
TranslationRuleset.task_filter_ruleset to filter down to entries in the DAG that can translate to a Task, in priority-order. - For each: Apply the
TranslationRuleset.task_ruleset, to convert the object to a specific Task, in priority-order, stopping when the first rule returns a match. If no rule returns a match, the entry is filtered. - After the DAG and Tasks are mapped, the
TranslationRuleset.task_dependency_ruleset is applied in priority-order, stopping when the first rule returns a match, to create a list of OrbiterTaskDependency, which are then added to each task in the OrbiterDAG - Apply the
TranslationRuleset.post_processing_ruleset, against the OrbiterProject, which can make modifications after all other rules have been applied. - After translation - the
OrbiterProject is rendered to the output folder. Source code in orbiter/rules/rulesets.py @validate_call\ndef translate(translation_ruleset, input_dir: Path) -> OrbiterProject:\n \"\"\"\n Orbiter expects a folder containing text files which may have a structure like:\n ```json\n {\"<workflow name>\": { ...<workflow properties>, \"<task name>\": { ...<task properties>} }}\n ```\n\n The standard translation function performs the following steps:\n\n \n\n 1. [**Find all files**][orbiter.rules.rulesets.TranslationRuleset.get_files_with_extension] with the expected\n [`TranslationRuleset.file_type`][orbiter.rules.rulesets.TranslationRuleset]\n (`.json`, `.xml`, `.yaml`, etc.) in the input folder.\n - [**Load each file**][orbiter.rules.rulesets.TranslationRuleset.loads] and turn it into a Python Dictionary.\n 2. **For each file:** Apply the [`TranslationRuleset.dag_filter_ruleset`][orbiter.rules.rulesets.DAGFilterRuleset]\n to filter down to entries that can translate to a DAG, in priority order.\n The file name is added under a `__file` key.\n - **For each dictionary**: Apply the [`TranslationRuleset.dag_ruleset`][orbiter.rules.rulesets.DAGRuleset],\n to convert the object to an [`OrbiterDAG`][orbiter.objects.dag.OrbiterDAG],\n in priority-order, stopping when the first rule returns a match.\n If no rule returns a match, the entry is filtered.\n 3. Apply the [`TranslationRuleset.task_filter_ruleset`][orbiter.rules.rulesets.TaskFilterRuleset]\n to filter down to entries in the DAG that can translate to a Task, in priority-order.\n - **For each:** Apply the [`TranslationRuleset.task_ruleset`][orbiter.rules.rulesets.TaskRuleset],\n to convert the object to a specific Task, in priority-order, stopping when the first rule returns a match.\n If no rule returns a match, the entry is filtered.\n 4. After the DAG and Tasks are mapped, the\n [`TranslationRuleset.task_dependency_ruleset`][orbiter.rules.rulesets.TaskDependencyRuleset]\n is applied in priority-order, stopping when the first rule returns a match,\n to create a list of\n [`OrbiterTaskDependency`][orbiter.objects.task.OrbiterTaskDependency],\n which are then added to each task in the\n [`OrbiterDAG`][orbiter.objects.dag.OrbiterDAG]\n 5. Apply the [`TranslationRuleset.post_processing_ruleset`][orbiter.rules.rulesets.PostProcessingRuleset],\n against the [`OrbiterProject`][orbiter.objects.project.OrbiterProject], which can make modifications after all\n other rules have been applied.\n 6. After translation - the [`OrbiterProject`][orbiter.objects.project.OrbiterProject]\n is rendered to the output folder.\n \"\"\"\n if not isinstance(translation_ruleset, TranslationRuleset):\n raise RuntimeError(\n f\"Error! type(translation_ruleset)=={type(translation_ruleset)}!=TranslationRuleset! Exiting!\"\n )\n\n # Create an initial OrbiterProject\n project = OrbiterProject()\n\n for i, (file, input_dict) in enumerate(\n translation_ruleset.get_files_with_extension(input_dir)\n ):\n logger.info(f\"Translating [File {i}]={file.resolve()}\")\n\n # DAG FILTER Ruleset - filter down to keys suspected of being translatable to a DAG, in priority order.\n dag_dicts: List[dict] = functools.reduce(\n add,\n translation_ruleset.dag_filter_ruleset.apply(val=input_dict),\n [],\n )\n # Add __file as a key to each DAG dict\n dag_dicts = [\n {\"__file\": file.relative_to(input_dir)} | dag_dict for dag_dict in dag_dicts\n ]\n logger.debug(f\"Found {len(dag_dicts)} DAG candidates in {file.resolve()}\")\n for dag_dict in dag_dicts:\n # DAG Ruleset - convert the object to an `OrbiterDAG` via `dag_ruleset`,\n # in priority-order, stopping when the first rule returns a match\n dag: OrbiterDAG | None = translation_ruleset.dag_ruleset.apply(\n val=dag_dict,\n take_first=True,\n )\n if dag is None:\n logger.warning(\n f\"Couldn't extract DAG from dag_dict={dag_dict} with dag_ruleset={translation_ruleset.dag_ruleset}\"\n )\n continue\n dag.orbiter_kwargs[\"file_path\"] = file.relative_to(input_dir)\n\n tasks = {}\n # TASK FILTER Ruleset - Many entries in dag_dict -> Many task_dict\n task_dicts = functools.reduce(\n add,\n translation_ruleset.task_filter_ruleset.apply(val=dag_dict),\n [],\n )\n logger.debug(\n f\"Found {len(task_dicts)} Task candidates in {dag.dag_id} in {file.resolve()}\"\n )\n for task_dict in task_dicts:\n # TASK Ruleset one -> one\n task: OrbiterOperator = translation_ruleset.task_ruleset.apply(\n val=task_dict, take_first=True\n )\n if task is None:\n logger.warning(\n f\"Couldn't extract task from expected task_dict={task_dict}\"\n )\n continue\n\n _add_task_deduped(task, tasks)\n logger.debug(f\"Adding {len(tasks)} tasks to DAG {dag.dag_id}\")\n dag.add_tasks(tasks.values())\n\n # Dag-Level TASK DEPENDENCY Ruleset\n task_dependencies: List[OrbiterTaskDependency] = (\n list(chain(*translation_ruleset.task_dependency_ruleset.apply(val=dag)))\n or []\n )\n if not len(task_dependencies):\n logger.warning(\n f\"Couldn't find task dependencies in \" f\"dag={trim_dict(dag_dict)}\"\n )\n for task_dependency in task_dependencies:\n task_dependency: OrbiterTaskDependency\n if task_dependency.task_id not in dag.tasks:\n logger.warning(\n f\"Couldn't find task_id={task_dependency.task_id} in tasks={tasks} for dag_id={dag.dag_id}\"\n )\n continue\n else:\n dag.tasks[task_dependency.task_id].add_downstream(task_dependency)\n\n logger.debug(f\"Adding DAG {dag.dag_id} to project\")\n project.add_dags(dag)\n\n # POST PROCESSING Ruleset\n translation_ruleset.post_processing_ruleset.apply(val=project, take_first=False)\n\n return project\n
"},{"location":"Rules_and_Rulesets/rules/","title":"Rules","text":""},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.Rule","title":"orbiter.rules.Rule","text":"A Rule contains a python function that is evaluated and produces something (typically an Object) or nothing A Rule can be created from a decorator >>> @rule(priority=1)\n... def my_rule(val):\n... return 1\n>>> isinstance(my_rule, Rule)\nTrue\n>>> my_rule(val={})\n1\n
The function in a rule takes one parameter (val), and must always evaluate to something or nothing. >>> Rule(rule=lambda val: 4)({})\n4\n>>> Rule(rule=lambda val: None)({})\n
Tip If the returned value is an Orbiter Object, the passed kwargs are saved in a special orbiter_kwargs property >>> from orbiter.objects.dag import OrbiterDAG\n>>> @rule\n... def my_rule(foo):\n... return OrbiterDAG(dag_id=\"\", file_path=\"\")\n>>> my_rule(foo=\"bar\").orbiter_kwargs\n{'foo': 'bar'}\n
Note A Rule must have a rule property and extra properties cannot be passed >>> # noinspection Pydantic\n... Rule(rule=lambda: None, not_a_prop=\"???\")\n... # doctest: +ELLIPSIS\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description rule Callable[[dict | Any], Any | None] Python function to evaluate. Takes a single argument and returns something or nothing priority int, optional Higher priority rules are evaluated first, must be greater than 0. Default is 0 "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.DAGFilterRule","title":"orbiter.rules.DAGFilterRule","text":" Bases: Rule The @dag_filter_rule decorator creates a DAGFilterRule @dag_filter_rule\ndef foo(val: dict) -> List[dict]:\n return [{\"dag_id\": \"foo\"}]\n
Hint In addition to filtering, a DAGFilterRule can also map input to a more reasonable output for later processing "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.DAGRule","title":"orbiter.rules.DAGRule","text":" Bases: Rule A @dag_rule decorator creates a DAGRule Tip A __file key is added to the original input, which is the file path of the input. @dag_rule\ndef foo(val: dict) -> OrbiterDAG | None:\n if 'id' in val:\n return OrbiterDAG(dag_id=val[\"id\"], file_path=f\"{val[\"id\"]}.py\")\n else:\n return None\n
"},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.TaskFilterRule","title":"orbiter.rules.TaskFilterRule","text":" Bases: Rule A @task_filter_rule decorator creates a TaskFilterRule @task_filter_rule\ndef foo(val: dict) -> List[dict] | None:\n return [{\"task_id\": \"foo\"}]\n
Hint In addition to filtering, a TaskFilterRule can also map input to a more reasonable output for later processing Parameters: Name Type Description val dict A dictionary of the task Returns: Type Description List[dict] | None A list of dictionaries of possible tasks or None "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.TaskRule","title":"orbiter.rules.TaskRule","text":" Bases: Rule A @task_rule decorator creates a TaskRule @task_rule\ndef foo(val: dict) -> OrbiterOperator | OrbiterTaskGroup:\n if 'id' in val and 'command' in val:\n return OrbiterBashOperator(task_id=val['id'], bash_command=val['command'])\n else:\n return None\n
Parameters: Name Type Description val dict A dictionary of the task Returns: Type Description OrbiterOperator | OrbiterTaskGroup | None A subclass of OrbiterOperator or OrbiterTaskGroup or None "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.TaskDependencyRule","title":"orbiter.rules.TaskDependencyRule","text":" Bases: Rule An @task_dependency_rule decorator creates a TaskDependencyRule, which takes an OrbiterDAG and returns a list[OrbiterTaskDependency] or None @task_dependency_rule\ndef foo(val: OrbiterDAG) -> OrbiterTaskDependency:\n return [OrbiterTaskDependency(task_id=\"upstream\", downstream=\"downstream\")]\n
Parameters: Name Type Description val OrbiterDAG An OrbiterDAG Returns: Type Description List[OrbiterTaskDependency] | None A list of OrbiterTaskDependency or None "},{"location":"Rules_and_Rulesets/rules/#orbiter.rules.PostProcessingRule","title":"orbiter.rules.PostProcessingRule","text":" Bases: Rule An @post_processing_rule decorator creates a PostProcessingRule, which takes an OrbiterProject, after all other rules have been applied, and modifies it in-place. @post_processing_rule\ndef foo(val: OrbiterProject) -> None:\n val.dags[\"foo\"].tasks[\"bar\"].doc = \"Hello World\"\n
Parameters: Name Type Description val OrbiterProject An OrbiterProject Returns: Type Description None None
"},{"location":"Rules_and_Rulesets/rulesets/","title":"Translation","text":""},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset","title":"orbiter.rules.rulesets.TranslationRuleset","text":"A Ruleset is a collection of Rules that are evaluated in priority order A TranslationRuleset is a container for Rulesets, which applies to a specific translation >>> TranslationRuleset(\n... file_type={FileTypeJSON}, # Has a file type\n... translate_fn=fake_translate, # and can have a callable\n... # translate_fn=\"orbiter.rules.translate.fake_translate\", # or a qualified name to a function\n... dag_filter_ruleset={\"ruleset\": [{\"rule\": lambda x: None}]}, # Rulesets can be dict within dicts\n... dag_ruleset=DAGRuleset(ruleset=[Rule(rule=lambda x: None)]), # or objects within objects\n... task_filter_ruleset=EMPTY_RULESET, # or a mix\n... task_ruleset=EMPTY_RULESET,\n... task_dependency_ruleset=EMPTY_RULESET, # Omitted for brevity\n... post_processing_ruleset=EMPTY_RULESET,\n... )\nTranslationRuleset(...)\n
Parameters: Name Type Description file_type Set[Type[FileType]] FileType to translate dag_filter_ruleset DAGFilterRuleset | dict DAGFilterRuleset (of DAGFilterRule)
dag_ruleset DAGRuleset | dict DAGRuleset (of DAGRules)
task_filter_ruleset TaskFilterRuleset | dict TaskFilterRuleset (of TaskFilterRule)
task_ruleset TaskRuleset | dict TaskRuleset (of TaskRules)
task_dependency_ruleset TaskDependencyRuleset | dict TaskDependencyRuleset (of TaskDependencyRules)
post_processing_ruleset PostProcessingRuleset | dict PostProcessingRuleset (of PostProcessingRules)
translate_fn Callable[[TranslationRuleset, Path], OrbiterProject] | str | TranslateFn Either a qualified name to a function (e.g. path.to.file.function), or a function reference, with the signature: (translation_ruleset: Translation Ruleset, input_dir: Path) -> OrbiterProject "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.dumps","title":"dumps","text":"dumps(input_dict: dict, ext: str | None = None) -> str\n
Convert Python dictionary back to source string form, useful for testing Parameters: Name Type Description input_dict dict The dictionary to convert to a string ext str | None The file type extension to dump as, defaults to first 'file_type' in the set Returns: Type Description str The string representation of the input_dict, in the file_type format Source code in orbiter/rules/rulesets.py @validate_call\ndef dumps(self, input_dict: dict, ext: str | None = None) -> str:\n \"\"\"\n Convert Python dictionary back to source string form, useful for testing\n\n :param input_dict: The dictionary to convert to a string\n :type input_dict: dict\n :param ext: The file type extension to dump as, defaults to first 'file_type' in the set\n :type ext: str | None\n :return str: The string representation of the input_dict, in the file_type format\n :rtype: str\n \"\"\"\n for file_type in self.file_type:\n if ext is None or ext.lower() in file_type.extension:\n return file_type.dump_fn(input_dict)\n raise TypeError(f\"Invalid file_type={ext}\")\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.get_ext","title":"get_ext","text":"get_ext() -> str\n
Get the first file extension for this ruleset >>> EMPTY_TRANSLATION_RULESET.get_ext()\n'JSON'\n
Source code in orbiter/rules/rulesets.py def get_ext(self) -> str:\n \"\"\"\n Get the first file extension for this ruleset\n\n ```pycon\n >>> EMPTY_TRANSLATION_RULESET.get_ext()\n 'JSON'\n\n ```\n \"\"\"\n return next(iter(next(iter(self.file_type)).extension))\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.get_files_with_extension","title":"get_files_with_extension","text":"get_files_with_extension(\n input_dir: Path,\n) -> Generator[Path, dict]\n
A generator that yields files with a specific extension(s) in a directory Parameters: Name Type Description input_dir Path The directory to search in Returns: Type Description Generator[Path, dict] Generator item of (Path, dict) for each file found Source code in orbiter/rules/rulesets.py def get_files_with_extension(self, input_dir: Path) -> Generator[Path, dict]:\n \"\"\"\n A generator that yields files with a specific extension(s) in a directory\n\n :param input_dir: The directory to search in\n :type input_dir: Path\n :return: Generator item of (Path, dict) for each file found\n :rtype: Generator[Path, dict]\n \"\"\"\n for directory, _, files in (\n input_dir.walk()\n if hasattr(input_dir, \"walk\")\n else _backport_walk(input_dir)\n ):\n logger.debug(f\"Checking directory={directory}\")\n for file in files:\n file = directory / file\n # noinspection PyBroadException\n try:\n yield (\n # Return the file path\n file,\n # and load the file and convert it into a python dict\n self.loads(file),\n )\n except TypeError:\n logger.debug(f\"File={file} not of correct type, skipping...\")\n continue\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.loads","title":"loads","text":"loads(file: Path) -> dict\n
Converts all files of type into a Python dictionary \"intermediate representation\" form, prior to any rulesets being applied. Parameters: Name Type Description file Path The file to load Returns: Type Description dict The dictionary representation of the input_str Source code in orbiter/rules/rulesets.py @validate_call\ndef loads(self, file: Path) -> dict:\n \"\"\"\n Converts all files of type into a Python dictionary \"intermediate representation\" form,\n prior to any rulesets being applied.\n\n :param file: The file to load\n :type file: Path\n :return: The dictionary representation of the input_str\n :rtype: dict\n \"\"\"\n for file_type in self.file_type:\n if file.suffix.lower() in {\n f\".{ext.lower()}\" for ext in file_type.extension\n }:\n try:\n return file_type.load_fn(file.read_text())\n except Exception as e:\n logger.error(f\"Error loading file={file}! Skipping!\\n{e}\")\n continue\n raise TypeError(\n f\"Invalid file_type={file.suffix}, does not match file_type={self.file_type}\"\n )\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset.test","title":"test","text":"test(input_value: str | dict) -> OrbiterProject\n
Test an input against the whole ruleset. - 'input_dict' (a parsed python dict) - or 'input_str' (raw value) to test against the ruleset. Parameters: Name Type Description input_value str | dict The input to test can be either a dict (passed to translate_ruleset.dumps() before translate_ruleset.loads()) or a string (read directly by translate_ruleset.loads()) Returns: Type Description OrbiterProject OrbiterProject produced after applying the ruleset Source code in orbiter/rules/rulesets.py def test(self, input_value: str | dict) -> OrbiterProject:\n \"\"\"\n Test an input against the whole ruleset.\n - 'input_dict' (a parsed python dict)\n - or 'input_str' (raw value) to test against the ruleset.\n\n :param input_value: The input to test\n can be either a dict (passed to `translate_ruleset.dumps()` before `translate_ruleset.loads()`)\n or a string (read directly by `translate_ruleset.loads()`)\n :type input_value: str | dict\n :return: OrbiterProject produced after applying the ruleset\n :rtype: OrbiterProject\n \"\"\"\n with TemporaryDirectory() as tempdir:\n file = Path(tempdir) / f\"{uuid.uuid4()}.{self.get_ext()}\"\n file.write_text(\n self.dumps(input_value)\n if isinstance(input_value, dict)\n else input_value\n )\n return self.translate_fn(translation_ruleset=self, input_dir=file.parent)\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.file_types","title":"orbiter.file_types","text":""},{"location":"Rules_and_Rulesets/rulesets/#orbiter.file_types.FileType","title":"FileType","text":"Abstract Base File Type Parameters: Name Type Description extension Set[str] The file extension(s) for this file type load_fn Callable[[str], dict] The function to load the file into a dictionary for this file type dump_fn Callable[[dict], str] The function to dump a dictionary to a string for this file type "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.file_types.FileTypeJSON","title":"FileTypeJSON","text":"JSON File Type >>> out = FileTypeJSON.dump_fn({'a': 1}); out\n'{\"a\": 1}'\n>>> FileTypeJSON.load_fn(out)\n{'a': 1}\n
Parameters: Name Type Description extension Set[str] JSON load_fn Callable[[str], dict] json.loads dump_fn Callable[[dict], str] json.dumps "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.file_types.FileTypeXML","title":"FileTypeXML","text":"XML File Type Note This class uses a custom xmltodict_parse method to standardize the output to a list of dictionaries >>> out = FileTypeXML.dump_fn({'a': 1}); out\n'<?xml version=\"1.0\" encoding=\"utf-8\"?>\\n<a>1</a>'\n>>> FileTypeXML.load_fn(out)\n{'a': '1'}\n
Parameters: Name Type Description extension Set[str] XML load_fn Callable[[str], dict] xmltodict_parse dump_fn Callable[[dict], str] xmltodict.unparse "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.file_types.FileTypeYAML","title":"FileTypeYAML","text":"YAML File Type >>> out = FileTypeYAML.dump_fn({'a': 1}); out\n'a: 1\\n'\n>>> FileTypeYAML.load_fn(out)\n{'a': 1}\n
Parameters: Name Type Description extension Set[str] YAML, YML load_fn Callable[[str], dict] yaml.safe_load dump_fn Callable[[dict], str] yaml.safe_dump "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.file_types.xmltodict_parse","title":"xmltodict_parse","text":"xmltodict_parse(input_str: str) -> Any\n
Calls xmltodict.parse and does post-processing fixes. Note The original xmltodict.parse method returns EITHER: - a dict (one child element of type)
- or a list of dict (many child element of type)
This behavior can be confusing, and is an issue with the original xml spec being referenced. This method deviates by standardizing to the latter case (always a list[dict]). All XML elements will be a list of dictionaries, even if there's only one element. >>> xmltodict_parse(\"\")\nTraceback (most recent call last):\nxml.parsers.expat.ExpatError: no element found: line 1, column 0\n>>> xmltodict_parse(\"<a></a>\")\n{'a': None}\n>>> xmltodict_parse(\"<a foo='bar'></a>\")\n{'a': [{'@foo': 'bar'}]}\n>>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'></foo></a>\") # Singleton - gets modified\n{'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz'}]}]}\n>>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'><bar><bop></bop></bar></foo></a>\") # Nested Singletons - modified\n{'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz', 'bar': [{'bop': None}]}]}]}\n>>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'></foo><foo bing='bop'></foo></a>\")\n{'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz'}, {'@bing': 'bop'}]}]}\n
Parameters: Name Type Description input_str str The XML string to parse Returns: Type Description dict The parsed XML Source code in orbiter/file_types.py def xmltodict_parse(input_str: str) -> Any:\n \"\"\"Calls `xmltodict.parse` and does post-processing fixes.\n\n !!! note\n\n The original [`xmltodict.parse`](https://pypi.org/project/xmltodict/) method returns EITHER:\n\n - a dict (one child element of type)\n - or a list of dict (many child element of type)\n\n This behavior can be confusing, and is an issue with the original xml spec being referenced.\n\n **This method deviates by standardizing to the latter case (always a `list[dict]`).**\n\n **All XML elements will be a list of dictionaries, even if there's only one element.**\n\n ```pycon\n >>> xmltodict_parse(\"\")\n Traceback (most recent call last):\n xml.parsers.expat.ExpatError: no element found: line 1, column 0\n >>> xmltodict_parse(\"<a></a>\")\n {'a': None}\n >>> xmltodict_parse(\"<a foo='bar'></a>\")\n {'a': [{'@foo': 'bar'}]}\n >>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'></foo></a>\") # Singleton - gets modified\n {'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz'}]}]}\n >>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'><bar><bop></bop></bar></foo></a>\") # Nested Singletons - modified\n {'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz', 'bar': [{'bop': None}]}]}]}\n >>> xmltodict_parse(\"<a foo='bar'><foo bar='baz'></foo><foo bing='bop'></foo></a>\")\n {'a': [{'@foo': 'bar', 'foo': [{'@bar': 'baz'}, {'@bing': 'bop'}]}]}\n\n ```\n :param input_str: The XML string to parse\n :type input_str: str\n :return: The parsed XML\n :rtype: dict\n \"\"\"\n\n # noinspection t\n def _fix(d):\n \"\"\"fix the dict in place, recursively, standardizing on a list of dict even if there's only one entry.\"\"\"\n # if it's a dict, descend to fix\n if isinstance(d, dict):\n for k, v in d.items():\n # @keys are properties of elements, non-@keys are elements\n if not k.startswith(\"@\"):\n if isinstance(v, dict):\n # THE FIX\n # any non-@keys should be a list of dict, even if there's just one of the element\n d[k] = [v]\n _fix(v)\n else:\n _fix(v)\n # if it's a list, descend to fix\n if isinstance(d, list):\n for v in d:\n _fix(v)\n\n output = xmltodict.parse(input_str)\n _fix(output)\n return output\n
"},{"location":"Rules_and_Rulesets/rulesets/#rulesets","title":"Rulesets","text":""},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.Ruleset","title":"orbiter.rules.rulesets.Ruleset","text":"A list of rules, which are evaluated to generate different types of output You must pass a Rule (or dict with the schema of Rule) >>> from orbiter.rules import rule\n>>> @rule\n... def x(val):\n... return None\n>>> Ruleset(ruleset=[x, {\"rule\": lambda: None}])\n... # doctest: +ELLIPSIS\nRuleset(ruleset=[Rule(...), Rule(...)])\n
Note You can't pass non-Rules >>> # noinspection PyTypeChecker\n... Ruleset(ruleset=[None])\n... # doctest: +ELLIPSIS\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description ruleset List[Rule | Callable[[Any], Any | None]] List of Rule (or dict with the schema of Rule) "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.Ruleset.apply","title":"apply","text":"apply(\n take_first: bool = False, **kwargs\n) -> List[Any] | Any\n
Apply all rules in ruleset to a single item, in priority order, removing any None results. A ruleset with one rule can produce up to one result >>> from orbiter.rules import rule\n\n>>> @rule\n... def gt_4(val):\n... return str(val) if val > 4 else None\n>>> Ruleset(ruleset=[gt_4]).apply(val=5)\n['5']\n
Many rules can produce many results, one for each rule. >>> @rule\n... def gt_3(val):\n... return str(val) if val > 3 else None\n>>> Ruleset(ruleset=[gt_4, gt_3]).apply(val=5)\n['5', '5']\n
The take_first flag will evaluate rules in the ruleset and return the first match >>> Ruleset(ruleset=[gt_4, gt_3]).apply(val=5, take_first=True)\n'5'\n
If nothing matched, an empty list is returned >>> @rule\n... def always_none(val):\n... return None\n>>> @rule\n... def more_always_none(val):\n... return None\n>>> Ruleset(ruleset=[always_none, more_always_none]).apply(val=5)\n[]\n
If nothing matched, and take_first=True, None is returned >>> Ruleset(ruleset=[always_none, more_always_none]).apply(val=5, take_first=True)\n... # None\n
Tip If no input is given, an error is returned >>> Ruleset(ruleset=[always_none]).apply()\nTraceback (most recent call last):\nRuntimeError: No values provided! Supply at least one key=val pair as kwargs!\n
Parameters: Name Type Description take_first bool only take the first (if any) result from the ruleset application kwargs key=val pairs to pass to the evaluated rule function Returns: Type Description List[Any] | Any | None List of rules that evaluated to Any (in priority order), or an empty list, or Any (if take_first=True) Raises: Type Description RuntimeError if the Ruleset is empty or input_val is None RuntimeError if the Rule raises an exception Source code in orbiter/rules/rulesets.py @validate_call\ndef apply(self, take_first: bool = False, **kwargs) -> List[Any] | Any:\n \"\"\"\n Apply all rules in ruleset **to a single item**, in priority order, removing any `None` results.\n\n A ruleset with one rule can produce **up to one** result\n ```pycon\n >>> from orbiter.rules import rule\n\n >>> @rule\n ... def gt_4(val):\n ... return str(val) if val > 4 else None\n >>> Ruleset(ruleset=[gt_4]).apply(val=5)\n ['5']\n\n ```\n\n Many rules can produce many results, one for each rule.\n ```pycon\n >>> @rule\n ... def gt_3(val):\n ... return str(val) if val > 3 else None\n >>> Ruleset(ruleset=[gt_4, gt_3]).apply(val=5)\n ['5', '5']\n\n ```\n\n The `take_first` flag will evaluate rules in the ruleset and return the first match\n ```pycon\n >>> Ruleset(ruleset=[gt_4, gt_3]).apply(val=5, take_first=True)\n '5'\n\n ```\n\n If nothing matched, an empty list is returned\n ```pycon\n >>> @rule\n ... def always_none(val):\n ... return None\n >>> @rule\n ... def more_always_none(val):\n ... return None\n >>> Ruleset(ruleset=[always_none, more_always_none]).apply(val=5)\n []\n\n ```\n\n If nothing matched, and `take_first=True`, `None` is returned\n ```pycon\n >>> Ruleset(ruleset=[always_none, more_always_none]).apply(val=5, take_first=True)\n ... # None\n\n ```\n\n !!! tip\n\n If no input is given, an error is returned\n ```pycon\n >>> Ruleset(ruleset=[always_none]).apply()\n Traceback (most recent call last):\n RuntimeError: No values provided! Supply at least one key=val pair as kwargs!\n\n ```\n\n :param take_first: only take the first (if any) result from the ruleset application\n :type take_first: bool\n :param kwargs: key=val pairs to pass to the evaluated rule function\n :returns: List of rules that evaluated to `Any` (in priority order),\n or an empty list,\n or `Any` (if `take_first=True`)\n :rtype: List[Any] | Any | None\n :raises RuntimeError: if the Ruleset is empty or input_val is None\n :raises RuntimeError: if the Rule raises an exception\n \"\"\"\n if not len(kwargs):\n raise RuntimeError(\n \"No values provided! Supply at least one key=val pair as kwargs!\"\n )\n results = []\n for _rule in self._sorted():\n result = _rule(**kwargs)\n should_show_input = \"val\" in kwargs and not (\n isinstance(kwargs[\"val\"], OrbiterProject)\n or isinstance(kwargs[\"val\"], OrbiterDAG)\n )\n if result is not None:\n logger.debug(\n \"---------\\n\"\n f\"[RULESET MATCHED] '{self.__class__.__module__}.{self.__class__.__name__}'\\n\"\n f\"[RULE MATCHED] '{_rule.__name__}'\\n\"\n f\"[INPUT] {trim_dict(kwargs) if should_show_input else '<Skipping...>'}\\n\"\n f\"[RETURN] {trim_dict(result)}\\n\"\n f\"---------\"\n )\n results.append(result)\n if take_first:\n return result\n return None if take_first and not len(results) else results\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.Ruleset.apply_many","title":"apply_many","text":"apply_many(\n input_val: Collection[Any], take_first: bool = False\n) -> List[List[Any]] | List[Any]\n
Apply a ruleset to each item in collection (such as dict().items()) and return any results that are not None You can turn the output of apply_many into a dict, if the rule takes and returns a tuple >>> from itertools import chain\n>>> from orbiter.rules import rule\n\n>>> @rule\n... def filter_for_type_folder(val):\n... (key, val) = val\n... return (key, val) if val.get('Type', '') == 'Folder' else None\n>>> ruleset = Ruleset(ruleset=[filter_for_type_folder])\n>>> input_dict = {\n... \"a\": {\"Type\": \"Folder\"},\n... \"b\": {\"Type\": \"File\"},\n... \"c\": {\"Type\": \"Folder\"},\n... }\n>>> dict(chain(*chain(ruleset.apply_many(input_dict.items()))))\n... # use dict(chain(*chain(...))), if using `take_first=True`, to turn many results back into dict\n{'a': {'Type': 'Folder'}, 'c': {'Type': 'Folder'}}\n>>> dict(ruleset.apply_many(input_dict.items(), take_first=True))\n... # use dict(...) directly, if using `take_first=True`, to turn results back into dict\n{'a': {'Type': 'Folder'}, 'c': {'Type': 'Folder'}}\n
Tip You cannot pass input without length >>> ruleset.apply_many({})\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\nRuntimeError: Input is not Collection[Any] with length!\n
Parameters: Name Type Description input_val Collection[Any] List to evaluate ruleset over take_first bool Only take the first (if any) result from each ruleset application Returns: Type Description List[List[Any]] | List[Any] List of list with all non-null evaluations for each item or list of the first non-null evaluation for each item (if take_first=True) Raises: Type Description RuntimeError if the Ruleset or input_vals are empty RuntimeError if the Rule raises an exception Source code in orbiter/rules/rulesets.py def apply_many(\n self,\n input_val: Collection[Any],\n take_first: bool = False,\n) -> List[List[Any]] | List[Any]:\n \"\"\"\n Apply a ruleset to each item in collection (such as `dict().items()`)\n and return any results that are not `None`\n\n You can turn the output of `apply_many` into a dict, if the rule takes and returns a tuple\n ```pycon\n >>> from itertools import chain\n >>> from orbiter.rules import rule\n\n >>> @rule\n ... def filter_for_type_folder(val):\n ... (key, val) = val\n ... return (key, val) if val.get('Type', '') == 'Folder' else None\n >>> ruleset = Ruleset(ruleset=[filter_for_type_folder])\n >>> input_dict = {\n ... \"a\": {\"Type\": \"Folder\"},\n ... \"b\": {\"Type\": \"File\"},\n ... \"c\": {\"Type\": \"Folder\"},\n ... }\n >>> dict(chain(*chain(ruleset.apply_many(input_dict.items()))))\n ... # use dict(chain(*chain(...))), if using `take_first=True`, to turn many results back into dict\n {'a': {'Type': 'Folder'}, 'c': {'Type': 'Folder'}}\n >>> dict(ruleset.apply_many(input_dict.items(), take_first=True))\n ... # use dict(...) directly, if using `take_first=True`, to turn results back into dict\n {'a': {'Type': 'Folder'}, 'c': {'Type': 'Folder'}}\n\n ```\n !!! tip\n\n You cannot pass input without length\n ```pycon\n >>> ruleset.apply_many({})\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n RuntimeError: Input is not Collection[Any] with length!\n\n ```\n :param input_val: List to evaluate ruleset over\n :type input_val: Collection[Any]\n :param take_first: Only take the first (if any) result from each ruleset application\n :type take_first: bool\n :returns: List of list with all non-null evaluations for each item<br>\n or list of the first non-null evaluation for each item (if `take_first=True`)\n :rtype: List[List[Any]] | List[Any]\n :raises RuntimeError: if the Ruleset or input_vals are empty\n :raises RuntimeError: if the Rule raises an exception\n \"\"\"\n # Validate Input\n if not input_val or not len(input_val):\n raise RuntimeError(\"Input is not `Collection[Any]` with length!\")\n\n return [\n results[0] if take_first else results\n for item in input_val\n if (results := self.apply(take_first=False, val=item)) is not None\n and len(results)\n ]\n
"},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.DAGFilterRuleset","title":"orbiter.rules.rulesets.DAGFilterRuleset","text":" Bases: Ruleset Ruleset of DAGFilterRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.DAGRuleset","title":"orbiter.rules.rulesets.DAGRuleset","text":" Bases: Ruleset Ruleset of DAGRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TaskFilterRuleset","title":"orbiter.rules.rulesets.TaskFilterRuleset","text":" Bases: Ruleset Ruleset of TaskFilterRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TaskRuleset","title":"orbiter.rules.rulesets.TaskRuleset","text":" Bases: Ruleset Ruleset of TaskRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TaskDependencyRuleset","title":"orbiter.rules.rulesets.TaskDependencyRuleset","text":" Bases: Ruleset Ruleset of TaskDependencyRule "},{"location":"Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.PostProcessingRuleset","title":"orbiter.rules.rulesets.PostProcessingRuleset","text":" Bases: Ruleset Ruleset of PostProcessingRule "},{"location":"Rules_and_Rulesets/template/","title":"Template","text":"The following template can be utilized for creating a new TranslationRuleset translation_template.pyfrom __future__ import annotations\nfrom orbiter.file_types import FileTypeJSON\nfrom orbiter.objects.dag import OrbiterDAG\nfrom orbiter.objects.operators.empty import OrbiterEmptyOperator\nfrom orbiter.objects.project import OrbiterProject\nfrom orbiter.objects.task import OrbiterOperator\nfrom orbiter.objects.task_group import OrbiterTaskGroup\nfrom orbiter.rules import (\n dag_filter_rule,\n dag_rule,\n task_filter_rule,\n task_rule,\n task_dependency_rule,\n post_processing_rule,\n cannot_map_rule,\n)\nfrom orbiter.rules.rulesets import (\n DAGFilterRuleset,\n DAGRuleset,\n TaskFilterRuleset,\n TaskRuleset,\n TaskDependencyRuleset,\n PostProcessingRuleset,\n TranslationRuleset,\n)\n\n\n@dag_filter_rule\ndef basic_dag_filter(val: dict) -> list | None:\n \"\"\"Filter input down to a list of dictionaries that can be processed by the `@dag_rules`\"\"\"\n if ...:\n for k, v in val.items():\n pass\n return []\n else:\n return None\n\n\n@dag_rule\ndef basic_dag_rule(val: dict) -> OrbiterDAG | None:\n \"\"\"Translate input into an `OrbiterDAG`\"\"\"\n if \"dag_id\" in val:\n return OrbiterDAG(dag_id=val[\"dag_id\"], file_path=\"file.py\")\n else:\n return None\n\n\n@task_filter_rule\ndef basic_task_filter(val: dict) -> list | None:\n \"\"\"Filter input down to a list of dictionaries that can be processed by the `@task_rules`\"\"\"\n if ...:\n for k, v in val.items():\n pass\n return []\n else:\n return None\n\n\n@task_rule(priority=2)\ndef basic_task_rule(val: dict) -> OrbiterOperator | OrbiterTaskGroup | None:\n \"\"\"Translate input into an Operator (e.g. `OrbiterBashOperator`). will be applied first, with a higher priority\"\"\"\n if \"task_id\" in val:\n return OrbiterEmptyOperator(task_id=val[\"task_id\"])\n else:\n return None\n\n\n@task_dependency_rule\ndef basic_task_dependency_rule(val: OrbiterDAG) -> list | None:\n \"\"\"Translate input into a list of task dependencies\"\"\"\n if ...:\n for task in val.tasks.values():\n original_task_kwargs = task.orbiter_kwargs[\"val\"]\n for task_dependency in original_task_kwargs.get(\"task_dependencies\", []):\n pass\n return []\n else:\n return None\n\n\n@post_processing_rule\ndef basic_post_processing_rule(val: OrbiterProject) -> None:\n \"\"\"Modify the project in-place, after all other rules have applied\"\"\"\n if ...:\n for dag_id, dag in val.dags.items():\n for task_id, task in dag.tasks.items():\n pass\n\n\ntranslation_ruleset = TranslationRuleset(\n file_type={FileTypeJSON},\n dag_filter_ruleset=DAGFilterRuleset(ruleset=[basic_dag_filter]),\n dag_ruleset=DAGRuleset(ruleset=[basic_dag_rule]),\n task_filter_ruleset=TaskFilterRuleset(ruleset=[basic_task_filter]),\n task_ruleset=TaskRuleset(ruleset=[basic_task_rule, cannot_map_rule]),\n task_dependency_ruleset=TaskDependencyRuleset(ruleset=[basic_task_dependency_rule]),\n post_processing_ruleset=PostProcessingRuleset(ruleset=[basic_post_processing_rule]),\n)\n
"},{"location":"objects/","title":"Overview","text":"Objects are returned from Rules during a translation, and are rendered to produce an Apache Airflow Project An OrbiterProject holds everything necessary to render an Airflow Project. It is generated by a TranslationRuleset.translate_fn. Workflows are represented by a OrbiterDAG which is a Directed Acyclic Graph (of Tasks). OrbiterOperators represent Airflow Tasks, which are units of work. An Operator is a pre-defined task with specific functionality.
"},{"location":"objects/#orbiter.objects.OrbiterBase","title":"orbiter.objects.OrbiterBase","text":"AbstractBaseClass for Orbiter objects, provides a number of properties Parameters: Name Type Description imports List[OrbiterRequirement] List of OrbiterRequirement objects orbiter_kwargs dict, optional Optional dictionary of keyword arguments, to preserve what was originally parsed by a rule orbiter_conns Set[OrbiterConnection], optional Optional set of OrbiterConnection objects orbiter_env_vars Set[OrbiterEnvVar], optional Optional set of OrbiterEnvVar objects orbiter_includes Set[OrbiterInclude], optional Optional set of OrbiterInclude objects orbiter_vars Set[OrbiterVariable], optional Optional set of OrbiterVariable objects "},{"location":"objects/project/","title":"Project","text":"An OrbiterProject holds everything necessary to render an Airflow Project. It is generated by a TranslationRuleset.translate_fn. "},{"location":"objects/project/#diagram","title":"Diagram","text":"classDiagram\n direction LR\n\n OrbiterProject --> \"many\" OrbiterConnection\n OrbiterProject --> \"many\" OrbiterDAG\n OrbiterProject --> \"many\" OrbiterEnvVar\n OrbiterProject --> \"many\" OrbiterInclude\n OrbiterProject --> \"many\" OrbiterPool\n OrbiterProject --> \"many\" OrbiterRequirement\n OrbiterProject --> \"many\" OrbiterVariable\n class OrbiterProject[\"orbiter.objects.project.OrbiterProject\"] {\n connections: Dict[str, OrbiterConnection]\n dags: Dict[str, OrbiterDAG]\n env_vars: Dict[str, OrbiterEnvVar]\n includes: Dict[str, OrbiterInclude]\n pools: Dict[str, OrbiterPool]\n requirements: Set[OrbiterRequirement]\n variables: Dict[str, OrbiterVariable]\n }\n click OrbiterProject href \"#orbiter.objects.project.OrbiterProject\" \"OrbiterProject Documentation\"\n\n class OrbiterConnection[\"orbiter.objects.connection.OrbiterConnection\"] {\n conn_id: str\n conn_type: str\n **kwargs\n }\n click OrbiterConnection href \"#orbiter.objects.connection.OrbiterConnection\" \"OrbiterConnection Documentation\"\n\n OrbiterDAG --> \"many\" OrbiterInclude\n OrbiterDAG --> \"many\" OrbiterConnection\n OrbiterDAG --> \"many\" OrbiterEnvVar\n OrbiterDAG --> \"many\" OrbiterRequirement\n OrbiterDAG --> \"many\" OrbiterVariable\n class OrbiterDAG[\"orbiter.objects.dag.OrbiterDAG\"] {\n imports: List[OrbiterRequirement]\n file_path: str\n dag_id: str\n schedule: str | OrbiterTimetable | None\n catchup: bool\n start_date: DateTime\n tags: List[str]\n default_args: Dict[str, Any]\n params: Dict[str, Any]\n doc_md: str | None\n tasks: Dict[str, OrbiterOperator]\n kwargs: dict\n orbiter_kwargs: dict\n orbiter_conns: Set[OrbiterConnection]\n orbiter_vars: Set[OrbiterVariable]\n orbiter_env_vars: Set[OrbiterEnvVar]\n orbiter_includes: Set[OrbiterInclude]\n }\n\n class OrbiterEnvVar[\"orbiter.objects.env_var.OrbiterEnvVar\"] {\n key: str\n value: str\n }\n click OrbiterEnvVar href \"#orbiter.objects.env_var.OrbiterEnvVar\" \"OrbiterEnvVar Documentation\"\n\n class OrbiterInclude[\"orbiter.objects.include.OrbiterInclude\"] {\n filepath: str\n contents: str\n }\n click OrbiterInclude href \"#orbiter.objects.include.OrbiterInclude\" \"OrbiterInclude Documentation\"\n\n class OrbiterPool[\"orbiter.objects.pool.OrbiterPool\"] {\n name: str\n description: str | None\n slots: int | None\n }\n click OrbiterPool href \"#orbiter.objects.pool.OrbiterPool\" \"OrbiterPool Documentation\"\n\n class OrbiterRequirement[\"orbiter.objects.requirement.OrbiterRequirement\"] {\n package: str | None\n module: str | None\n names: List[str] | None\n sys_package: str | None\n }\n click OrbiterRequirement href \"#orbiter.objects.requirement.OrbiterRequirement\" \"OrbiterRequirement Documentation\"\n\n class OrbiterVariable[\"orbiter.objects.variable.OrbiterVariable\"] {\n key: str\n value: str\n }\n click OrbiterVariable href \"#orbiter.objects.variable.OrbiterVariable\" \"OrbiterVariable Documentation\"\n
"},{"location":"objects/project/#orbiter.objects.connection.OrbiterConnection","title":"orbiter.objects.connection.OrbiterConnection","text":"An Airflow Connection, rendered to an airflow_settings.yaml file. See also other Connection documentation. >>> OrbiterConnection(\n... conn_id=\"my_conn_id\", conn_type=\"mysql\", host=\"localhost\", port=3306, login=\"root\"\n... ).render()\n{'conn_id': 'my_conn_id', 'conn_type': 'mysql', 'conn_host': 'localhost', 'conn_port': 3306, 'conn_login': 'root'}\n
Note Use the utility conn_id function to generate both an OrbiterConnection and connection property for an operator from orbiter.objects import conn_id\n\nOrbiterTask(\n ... ,\n **conn_id(\"my_conn_id\", conn_type=\"mysql\"),\n)\n
Parameters: Name Type Description conn_id str The ID of the connection conn_type str, optional The type of the connection, always lowercase. Defaults to 'generic' **kwargs Additional properties for the connection "},{"location":"objects/project/#orbiter.objects.env_var.OrbiterEnvVar","title":"orbiter.objects.env_var.OrbiterEnvVar","text":"Represents an Environmental Variable, renders to a line in .env file >>> OrbiterEnvVar(key=\"foo\", value=\"bar\").render()\n'foo=bar'\n
Parameters: Name Type Description key str The key of the environment variable value str The value of the environment variable "},{"location":"objects/project/#orbiter.objects.include.OrbiterInclude","title":"orbiter.objects.include.OrbiterInclude","text":"Represents an included file in an /include directory Parameters: Name Type Description filepath str The relative path (from the output directory) to write the file to contents str The contents of the file "},{"location":"objects/project/#orbiter.objects.pool.OrbiterPool","title":"orbiter.objects.pool.OrbiterPool","text":"An Airflow Pool, rendered to an airflow_settings.yaml file. >>> OrbiterPool(name=\"foo\", description=\"bar\", slots=5).render()\n{'pool_name': 'foo', 'pool_description': 'bar', 'pool_slot': 5}\n
Note Use the utility pool function to easily generate both an OrbiterPool and pool property for an operator from orbiter.objects import pool\n\nOrbiterTask(\n ... ,\n **pool(\"my_pool\"),\n)\n
Parameters: Name Type Description name str The name of the pool description str, optional The description of the pool slots int, optional The number of slots in the pool. Defaults to 128 "},{"location":"objects/project/#orbiter.objects.requirement.OrbiterRequirement","title":"orbiter.objects.requirement.OrbiterRequirement","text":"A requirement for a project (e.g. apache-airflow-providers-google), and it's representation in the DAG file. Renders via the DAG File (as an import statement), requirements.txt, and packages.txt Tip If a given requirement has multiple packages required, it can be defined as multiple OrbiterRequirement objects. Example: OrbiterTask(\n ...,\n imports=[\n OrbiterRequirement(package=\"apache-airflow-providers-google\", ...),\n OrbiterRequirement(package=\"bigquery\", sys_package=\"mysql\", ...),\n ],\n)\n
Parameters: Name Type Description package str, optional e.g. \"apache-airflow-providers-google\" module str, optional e.g. \"airflow.providers.google.cloud.operators.bigquery\", defaults to None names List[str], optional e.g. [\"BigQueryCreateEmptyDatasetOperator\"], defaults to [] sys_package Set[str], optional e.g. \"mysql\" - represents a Debian system package "},{"location":"objects/project/#orbiter.objects.variable.OrbiterVariable","title":"orbiter.objects.variable.OrbiterVariable","text":"An Airflow Variable, rendered to an airflow_settings.yaml file. >>> OrbiterVariable(key=\"foo\", value=\"bar\").render()\n{'variable_value': 'bar', 'variable_name': 'foo'}\n
Parameters: Name Type Description key str The key of the variable value str The value of the variable "},{"location":"objects/project/#orbiter.objects.project.OrbiterProject","title":"orbiter.objects.project.OrbiterProject","text":"OrbiterProject()\n
Holds everything necessary to render an Airflow Project. This is generated by a TranslationRuleset.translate_fn. Tip They can be added together >>> OrbiterProject() + OrbiterProject()\nOrbiterProject(dags=[], requirements=[], pools=[], connections=[], variables=[], env_vars=[])\n
And compared >>> OrbiterProject() == OrbiterProject()\nTrue\n
Parameters: Name Type Description connections Dict[str, OrbiterConnection] A dictionary of OrbiterConnections dags Dict[str, OrbiterDAG] A dictionary of OrbiterDAGs env_vars Dict[str, OrbiterEnvVar] A dictionary of OrbiterEnvVars includes Dict[str, OrbiterInclude] A dictionary of OrbiterIncludes pools Dict[str, OrbiterPool] A dictionary of OrbiterPools requirements Set[OrbiterRequirement] A set of OrbiterRequirements variables Dict[str, OrbiterVariable] A dictionary of OrbiterVariables Source code in orbiter/objects/project.py def __init__(self):\n self.dags: Dict[str, OrbiterDAG] = dict()\n self.requirements: Set[OrbiterRequirement] = set()\n self.pools: Dict[str, OrbiterPool] = dict()\n self.connections: Dict[str, OrbiterConnection] = dict()\n self.variables: Dict[str, OrbiterVariable] = dict()\n self.env_vars: Dict[str, OrbiterEnvVar] = dict()\n self.includes: Dict[str, OrbiterInclude] = dict()\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_connections","title":"add_connections","text":"add_connections(\n connections: (\n OrbiterConnection | Iterable[OrbiterConnection]\n ),\n) -> \"OrbiterProject\"\n
Add OrbiterConnections to the Project or override an existing connection with new properties >>> OrbiterProject().add_connections(OrbiterConnection(conn_id='foo')).connections\n{'foo': OrbiterConnection(conn_id=foo, conn_type=generic)}\n\n>>> OrbiterProject().add_connections(\n... [OrbiterConnection(conn_id='foo'), OrbiterConnection(conn_id='bar')]\n... ).connections\n{'foo': OrbiterConnection(conn_id=foo, conn_type=generic), 'bar': OrbiterConnection(conn_id=bar, conn_type=generic)}\n
Tip Validation requires an OrbiterConnection to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_connections('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n>>> OrbiterProject().add_connections(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description connections OrbiterConnection | Iterable[OrbiterConnection] List of OrbiterConnections Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_connections(\n self, connections: OrbiterConnection | Iterable[OrbiterConnection]\n) -> \"OrbiterProject\":\n \"\"\"Add [`OrbiterConnections`][orbiter.objects.connection.OrbiterConnection] to the Project\n or override an existing connection with new properties\n\n ```pycon\n >>> OrbiterProject().add_connections(OrbiterConnection(conn_id='foo')).connections\n {'foo': OrbiterConnection(conn_id=foo, conn_type=generic)}\n\n >>> OrbiterProject().add_connections(\n ... [OrbiterConnection(conn_id='foo'), OrbiterConnection(conn_id='bar')]\n ... ).connections\n {'foo': OrbiterConnection(conn_id=foo, conn_type=generic), 'bar': OrbiterConnection(conn_id=bar, conn_type=generic)}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterConnection` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_connections('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n >>> OrbiterProject().add_connections(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n\n :param connections: List of [`OrbiterConnections`][orbiter.objects.connection.OrbiterConnection]\n :type connections: List[OrbiterConnection] | OrbiterConnection\n :return: self\n :rtype: OrbiterProject\n \"\"\" # noqa: E501\n for connection in (\n [connections] if isinstance(connections, OrbiterConnection) else connections\n ):\n self.connections[connection.conn_id] = connection\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_dags","title":"add_dags","text":"add_dags(\n dags: OrbiterDAG | Iterable[OrbiterDAG],\n) -> \"OrbiterProject\"\n
Add OrbiterDAGs (and any OrbiterRequirements, OrbiterConns, OrbiterVars, OrbiterPools, OrbiterEnvVars, etc.) to the Project. >>> OrbiterProject().add_dags(OrbiterDAG(dag_id='foo', file_path=\"\")).dags['foo'].repr()\n'OrbiterDAG(dag_id=foo, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)'\n\n>>> dags = OrbiterProject().add_dags(\n... [OrbiterDAG(dag_id='foo', file_path=\"\"), OrbiterDAG(dag_id='bar', file_path=\"\")]\n... ).dags; dags['foo'].repr(), dags['bar'].repr()\n... # doctest: +NORMALIZE_WHITESPACE\n('OrbiterDAG(dag_id=foo, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)', 'OrbiterDAG(dag_id=bar, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)')\n\n>>> # An example adding a little of everything, including deeply nested things\n... from orbiter.objects.operators.bash import OrbiterBashOperator\n>>> from orbiter.objects.timetables.multi_cron_timetable import OrbiterMultiCronTimetable\n>>> from orbiter.objects.callbacks.smtp import OrbiterSmtpNotifierCallback\n>>> OrbiterProject().add_dags(OrbiterDAG(\n... dag_id='foo', file_path=\"\",\n... orbiter_env_vars={OrbiterEnvVar(key=\"foo\", value=\"bar\")},\n... orbiter_includes={OrbiterInclude(filepath='foo.txt', contents=\"Hello, World!\")},\n... schedule=OrbiterMultiCronTimetable(cron_defs=[\"0 */5 * * *\", \"0 */3 * * *\"]),\n... tasks={'foo': OrbiterTaskGroup(task_group_id=\"foo\",\n... tasks=[OrbiterBashOperator(\n... task_id='foo', bash_command='echo \"Hello, World!\"',\n... orbiter_pool=OrbiterPool(name='foo', slots=1),\n... orbiter_vars={OrbiterVariable(key='foo', value='bar')},\n... orbiter_conns={OrbiterConnection(conn_id='foo')},\n... orbiter_env_vars={OrbiterEnvVar(key='foo', value='bar')},\n... on_success_callback=OrbiterSmtpNotifierCallback(\n... to=\"foo@bar.com\",\n... smtp_conn_id=\"SMTP\",\n... orbiter_conns={OrbiterConnection(conn_id=\"SMTP\", conn_type=\"smtp\")}\n... )\n... )]\n... )}\n... ))\n... # doctest: +NORMALIZE_WHITESPACE\nOrbiterProject(dags=[foo],\nrequirements=[OrbiterRequirements(names=[DAG], package=apache-airflow, module=airflow, sys_package=None),\nOrbiterRequirements(names=[BashOperator], package=apache-airflow, module=airflow.operators.bash, sys_package=None),\nOrbiterRequirements(names=[send_smtp_notification], package=apache-airflow-providers-smtp, module=airflow.providers.smtp.notifications.smtp, sys_package=None),\nOrbiterRequirements(names=[TaskGroup], package=apache-airflow, module=airflow.utils.task_group, sys_package=None),\nOrbiterRequirements(names=[MultiCronTimetable], package=croniter, module=multi_cron_timetable, sys_package=None),\nOrbiterRequirements(names=[DateTime,Timezone], package=pendulum, module=pendulum, sys_package=None)],\npools=['foo'],\nconnections=['SMTP', 'foo'],\nvariables=['foo'],\nenv_vars=['foo'])\n
Tip Validation requires an OrbiterDAG to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_dags('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n>>> OrbiterProject().add_dags(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description dags OrbiterDAG | Iterable[OrbiterDAG] List of OrbiterDAGs Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_dags(self, dags: OrbiterDAG | Iterable[OrbiterDAG]) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterDAGs][orbiter.objects.dag.OrbiterDAG]\n (and any [OrbiterRequirements][orbiter.objects.requirement.OrbiterRequirement],\n [OrbiterConns][orbiter.objects.connection.OrbiterConnection],\n [OrbiterVars][orbiter.objects.variable.OrbiterVariable],\n [OrbiterPools][orbiter.objects.pool.OrbiterPool],\n [OrbiterEnvVars][orbiter.objects.env_var.OrbiterEnvVar], etc.)\n to the Project.\n\n ```pycon\n >>> OrbiterProject().add_dags(OrbiterDAG(dag_id='foo', file_path=\"\")).dags['foo'].repr()\n 'OrbiterDAG(dag_id=foo, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)'\n\n >>> dags = OrbiterProject().add_dags(\n ... [OrbiterDAG(dag_id='foo', file_path=\"\"), OrbiterDAG(dag_id='bar', file_path=\"\")]\n ... ).dags; dags['foo'].repr(), dags['bar'].repr()\n ... # doctest: +NORMALIZE_WHITESPACE\n ('OrbiterDAG(dag_id=foo, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)', 'OrbiterDAG(dag_id=bar, schedule=None, start_date=1970-01-01 00:00:00, catchup=False)')\n\n >>> # An example adding a little of everything, including deeply nested things\n ... from orbiter.objects.operators.bash import OrbiterBashOperator\n >>> from orbiter.objects.timetables.multi_cron_timetable import OrbiterMultiCronTimetable\n >>> from orbiter.objects.callbacks.smtp import OrbiterSmtpNotifierCallback\n >>> OrbiterProject().add_dags(OrbiterDAG(\n ... dag_id='foo', file_path=\"\",\n ... orbiter_env_vars={OrbiterEnvVar(key=\"foo\", value=\"bar\")},\n ... orbiter_includes={OrbiterInclude(filepath='foo.txt', contents=\"Hello, World!\")},\n ... schedule=OrbiterMultiCronTimetable(cron_defs=[\"0 */5 * * *\", \"0 */3 * * *\"]),\n ... tasks={'foo': OrbiterTaskGroup(task_group_id=\"foo\",\n ... tasks=[OrbiterBashOperator(\n ... task_id='foo', bash_command='echo \"Hello, World!\"',\n ... orbiter_pool=OrbiterPool(name='foo', slots=1),\n ... orbiter_vars={OrbiterVariable(key='foo', value='bar')},\n ... orbiter_conns={OrbiterConnection(conn_id='foo')},\n ... orbiter_env_vars={OrbiterEnvVar(key='foo', value='bar')},\n ... on_success_callback=OrbiterSmtpNotifierCallback(\n ... to=\"foo@bar.com\",\n ... smtp_conn_id=\"SMTP\",\n ... orbiter_conns={OrbiterConnection(conn_id=\"SMTP\", conn_type=\"smtp\")}\n ... )\n ... )]\n ... )}\n ... ))\n ... # doctest: +NORMALIZE_WHITESPACE\n OrbiterProject(dags=[foo],\n requirements=[OrbiterRequirements(names=[DAG], package=apache-airflow, module=airflow, sys_package=None),\n OrbiterRequirements(names=[BashOperator], package=apache-airflow, module=airflow.operators.bash, sys_package=None),\n OrbiterRequirements(names=[send_smtp_notification], package=apache-airflow-providers-smtp, module=airflow.providers.smtp.notifications.smtp, sys_package=None),\n OrbiterRequirements(names=[TaskGroup], package=apache-airflow, module=airflow.utils.task_group, sys_package=None),\n OrbiterRequirements(names=[MultiCronTimetable], package=croniter, module=multi_cron_timetable, sys_package=None),\n OrbiterRequirements(names=[DateTime,Timezone], package=pendulum, module=pendulum, sys_package=None)],\n pools=['foo'],\n connections=['SMTP', 'foo'],\n variables=['foo'],\n env_vars=['foo'])\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterDAG` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_dags('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n >>> OrbiterProject().add_dags(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n\n :param dags: List of [OrbiterDAGs][orbiter.objects.dag.OrbiterDAG]\n :type dags: List[OrbiterDAG] | OrbiterDAG\n :return: self\n :rtype: OrbiterProject\n \"\"\" # noqa: E501\n\n # noinspection t\n def _add_recursively(\n things: Iterable[\n OrbiterOperator\n | OrbiterTaskGroup\n | OrbiterCallback\n | OrbiterTimetable\n | OrbiterDAG\n ],\n ):\n for thing in things:\n if isinstance(thing, str):\n continue\n if hasattr(thing, \"orbiter_pool\") and (pool := thing.orbiter_pool):\n self.add_pools(pool)\n if hasattr(thing, \"orbiter_conns\") and (conns := thing.orbiter_conns):\n self.add_connections(conns)\n if hasattr(thing, \"orbiter_vars\") and (variables := thing.orbiter_vars):\n self.add_variables(variables)\n if hasattr(thing, \"orbiter_env_vars\") and (\n env_vars := thing.orbiter_env_vars\n ):\n self.add_env_vars(env_vars)\n if hasattr(thing, \"orbiter_includes\") and (\n includes := thing.orbiter_includes\n ):\n self.add_includes(includes)\n if hasattr(thing, \"imports\") and (imports := thing.imports):\n self.add_requirements(imports)\n if isinstance(thing, OrbiterTaskGroup) and (tasks := thing.tasks):\n _add_recursively(tasks)\n if hasattr(thing, \"__dict__\") or hasattr(thing, \"model_extra\"):\n # If it's a pydantic model or dict, check its properties for more things to add\n _add_recursively(\n (\n (getattr(thing, \"__dict__\", {}) or dict())\n | (getattr(thing, \"model_extra\", {}) or dict())\n ).values()\n )\n\n for dag in [dags] if isinstance(dags, OrbiterDAG) else dags:\n dag_id = dag.dag_id\n\n # Add or update the DAG\n if dag_id in self.dags:\n self.dags[dag_id] += dag\n else:\n self.dags[dag_id] = dag\n\n # Add anything that might be in the tasks of the DAG - such as imports, Connections, etc\n _add_recursively((dag.tasks or {}).values())\n\n # Add anything that might be in the `dag.schedule` - such as Includes, Timetables, Connections, etc\n _add_recursively([dag])\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_env_vars","title":"add_env_vars","text":"add_env_vars(\n env_vars: OrbiterEnvVar | Iterable[OrbiterEnvVar],\n) -> \"OrbiterProject\"\n
Add OrbiterEnvVars to the Project or override an existing env var with new properties >>> OrbiterProject().add_env_vars(OrbiterEnvVar(key=\"foo\", value=\"bar\")).env_vars\n{'foo': OrbiterEnvVar(key='foo', value='bar')}\n\n>>> OrbiterProject().add_env_vars([OrbiterEnvVar(key=\"foo\", value=\"bar\")]).env_vars\n{'foo': OrbiterEnvVar(key='foo', value='bar')}\n
Tip Validation requires an OrbiterEnvVar to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_env_vars('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n\n>>> # noinspection PyTypeChecker\n... OrbiterProject().add_env_vars(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description env_vars OrbiterEnvVar | Iterable[OrbiterEnvVar] List of OrbiterEnvVar Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_env_vars(\n self, env_vars: OrbiterEnvVar | Iterable[OrbiterEnvVar]\n) -> \"OrbiterProject\":\n \"\"\"\n Add [OrbiterEnvVars][orbiter.objects.env_var.OrbiterEnvVar] to the Project\n or override an existing env var with new properties\n\n ```pycon\n >>> OrbiterProject().add_env_vars(OrbiterEnvVar(key=\"foo\", value=\"bar\")).env_vars\n {'foo': OrbiterEnvVar(key='foo', value='bar')}\n\n >>> OrbiterProject().add_env_vars([OrbiterEnvVar(key=\"foo\", value=\"bar\")]).env_vars\n {'foo': OrbiterEnvVar(key='foo', value='bar')}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterEnvVar` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_env_vars('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_env_vars(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n\n :param env_vars: List of [OrbiterEnvVar][orbiter.objects.env_var.OrbiterEnvVar]\n :type env_vars: List[OrbiterEnvVar] | OrbiterEnvVar\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for env_var in [env_vars] if isinstance(env_vars, OrbiterEnvVar) else env_vars:\n self.env_vars[env_var.key] = env_var\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_includes","title":"add_includes","text":"add_includes(\n includes: OrbiterInclude | Iterable[OrbiterInclude],\n) -> \"OrbiterProject\"\n
Add OrbiterIncludes to the Project or override an existing OrbiterInclude with new properties >>> OrbiterProject().add_includes(OrbiterInclude(filepath=\"foo\", contents=\"bar\")).includes\n{'foo': OrbiterInclude(filepath='foo', contents='bar')}\n\n>>> OrbiterProject().add_includes([OrbiterInclude(filepath=\"foo\", contents=\"bar\")]).includes\n{'foo': OrbiterInclude(filepath='foo', contents='bar')}\n
Tip Validation requires an OrbiterInclude to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_includes('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n\n>>> # noinspection PyTypeChecker\n... OrbiterProject().add_includes(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description includes OrbiterInclude | Iterable[OrbiterInclude] List of OrbiterIncludes Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_includes(\n self, includes: OrbiterInclude | Iterable[OrbiterInclude]\n) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterIncludes][orbiter.objects.include.OrbiterInclude] to the Project\n or override an existing [OrbiterInclude][orbiter.objects.include.OrbiterInclude] with new properties\n\n ```pycon\n >>> OrbiterProject().add_includes(OrbiterInclude(filepath=\"foo\", contents=\"bar\")).includes\n {'foo': OrbiterInclude(filepath='foo', contents='bar')}\n\n >>> OrbiterProject().add_includes([OrbiterInclude(filepath=\"foo\", contents=\"bar\")]).includes\n {'foo': OrbiterInclude(filepath='foo', contents='bar')}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterInclude` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_includes('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_includes(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param includes: List of [OrbiterIncludes][orbiter.objects.include.OrbiterInclude]\n :type includes: List[OrbiterInclude]\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for include in [includes] if isinstance(includes, OrbiterInclude) else includes:\n self.includes[include.filepath] = include\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_pools","title":"add_pools","text":"add_pools(\n pools: OrbiterPool | Iterable[OrbiterPool],\n) -> \"OrbiterProject\"\n
Add OrbiterPool to the Project or override existing pools with new properties >>> OrbiterProject().add_pools(OrbiterPool(name=\"foo\", slots=1)).pools\n{'foo': OrbiterPool(name='foo', description='', slots=1)}\n\n>>> ( OrbiterProject()\n... .add_pools([OrbiterPool(name=\"foo\", slots=1)])\n... .add_pools([OrbiterPool(name=\"foo\", slots=2)])\n... .pools\n... )\n{'foo': OrbiterPool(name='foo', description='', slots=2)}\n
Tip Validation requires an OrbiterPool to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_pools('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n... OrbiterProject().add_pools(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description pools OrbiterPool | Iterable[OrbiterPool] List of OrbiterPools Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_pools(self, pools: OrbiterPool | Iterable[OrbiterPool]) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterPool][orbiter.objects.pool.OrbiterPool] to the Project\n or override existing pools with new properties\n\n ```pycon\n >>> OrbiterProject().add_pools(OrbiterPool(name=\"foo\", slots=1)).pools\n {'foo': OrbiterPool(name='foo', description='', slots=1)}\n\n >>> ( OrbiterProject()\n ... .add_pools([OrbiterPool(name=\"foo\", slots=1)])\n ... .add_pools([OrbiterPool(name=\"foo\", slots=2)])\n ... .pools\n ... )\n {'foo': OrbiterPool(name='foo', description='', slots=2)}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterPool` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_pools('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_pools(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param pools: List of [OrbiterPools][orbiter.objects.pool.OrbiterPool]\n :type pools: List[OrbiterPool] | OrbiterPool\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for pool in [pools] if isinstance(pools, OrbiterPool) else pools:\n if pool.name in self.pools:\n self.pools[pool.name] += pool\n else:\n self.pools[pool.name] = pool\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_requirements","title":"add_requirements","text":"add_requirements(\n requirements: (\n OrbiterRequirement | Iterable[OrbiterRequirement]\n ),\n) -> \"OrbiterProject\"\n
Add OrbiterRequirements to the Project or override an existing requirement with new properties >>> OrbiterProject().add_requirements(\n... OrbiterRequirement(package=\"apache-airflow\", names=['foo'], module='bar'),\n... ).requirements\n{OrbiterRequirements(names=[foo], package=apache-airflow, module=bar, sys_package=None)}\n\n>>> OrbiterProject().add_requirements(\n... [OrbiterRequirement(package=\"apache-airflow\", names=['foo'], module='bar')],\n... ).requirements\n{OrbiterRequirements(names=[foo], package=apache-airflow, module=bar, sys_package=None)}\n
Tip Validation requires an OrbiterRequirement to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_requirements('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n>>> OrbiterProject().add_requirements(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description requirements OrbiterRequirement | Iterable[OrbiterRequirement] List of OrbiterRequirements Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_requirements(\n self, requirements: OrbiterRequirement | Iterable[OrbiterRequirement]\n) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterRequirements][orbiter.objects.requirement.OrbiterRequirement] to the Project\n or override an existing requirement with new properties\n\n ```pycon\n >>> OrbiterProject().add_requirements(\n ... OrbiterRequirement(package=\"apache-airflow\", names=['foo'], module='bar'),\n ... ).requirements\n {OrbiterRequirements(names=[foo], package=apache-airflow, module=bar, sys_package=None)}\n\n >>> OrbiterProject().add_requirements(\n ... [OrbiterRequirement(package=\"apache-airflow\", names=['foo'], module='bar')],\n ... ).requirements\n {OrbiterRequirements(names=[foo], package=apache-airflow, module=bar, sys_package=None)}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterRequirement` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_requirements('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n >>> OrbiterProject().add_requirements(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param requirements: List of [OrbiterRequirements][orbiter.objects.requirement.OrbiterRequirement]\n :type requirements: List[OrbiterRequirement] | OrbiterRequirement\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for requirement in (\n [requirements]\n if isinstance(requirements, OrbiterRequirement)\n else requirements\n ):\n self.requirements.add(requirement)\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.add_variables","title":"add_variables","text":"add_variables(\n variables: OrbiterVariable | Iterable[OrbiterVariable],\n) -> \"OrbiterProject\"\n
Add OrbiterVariables to the Project or override an existing variable with new properties >>> OrbiterProject().add_variables(OrbiterVariable(key=\"foo\", value=\"bar\")).variables\n{'foo': OrbiterVariable(key='foo', value='bar')}\n\n>>> OrbiterProject().add_variables([OrbiterVariable(key=\"foo\", value=\"bar\")]).variables\n{'foo': OrbiterVariable(key='foo', value='bar')}\n
Tip Validation requires an OrbiterVariable to be passed >>> # noinspection PyTypeChecker\n... OrbiterProject().add_variables('foo')\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n... OrbiterProject().add_variables(['foo'])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description variables OrbiterVariable | Iterable[OrbiterVariable] List of OrbiterVariable Returns: Type Description OrbiterProject self Source code in orbiter/objects/project.py def add_variables(\n self, variables: OrbiterVariable | Iterable[OrbiterVariable]\n) -> \"OrbiterProject\":\n \"\"\"Add [OrbiterVariables][orbiter.objects.variable.OrbiterVariable] to the Project\n or override an existing variable with new properties\n\n ```pycon\n >>> OrbiterProject().add_variables(OrbiterVariable(key=\"foo\", value=\"bar\")).variables\n {'foo': OrbiterVariable(key='foo', value='bar')}\n\n >>> OrbiterProject().add_variables([OrbiterVariable(key=\"foo\", value=\"bar\")]).variables\n {'foo': OrbiterVariable(key='foo', value='bar')}\n\n ```\n\n !!! tip\n\n Validation requires an `OrbiterVariable` to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_variables('foo')\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n ... OrbiterProject().add_variables(['foo'])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param variables: List of [OrbiterVariable][orbiter.objects.variable.OrbiterVariable]\n :type variables: List[OrbiterVariable] | OrbiterVariable\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n for variable in (\n [variables] if isinstance(variables, OrbiterVariable) else variables\n ):\n self.variables[variable.key] = variable\n return self\n
"},{"location":"objects/project/#orbiter.objects.project.OrbiterProject.analyze","title":"analyze","text":"analyze(\n output_fmt: Literal[\"json\", \"csv\", \"md\"] = \"md\",\n output_file=None,\n)\n
Print an analysis of the project to the console. Tip Looks for a specific [task_type=XYZ] in the Task's doc_md property or uses type(task) to infer the type of task. >>> from orbiter.objects.operators.empty import OrbiterEmptyOperator\n>>> OrbiterProject().add_dags([\n... OrbiterDAG(file_path=\"\", dag_id=\"foo\", orbiter_kwargs={\"file_path\": \"foo.py\"},\n... tasks={\"bar\": OrbiterEmptyOperator(task_id=\"bar\")}\n... ),\n... OrbiterDAG(file_path=\"\", dag_id=\"baz\", orbiter_kwargs={\"file_path\": \"baz.py\"},\n... tasks={\"bop\": OrbiterEmptyOperator(task_id=\"bop\")}\n... )\n... ]).analyze()\n... # doctest: +ELLIPSIS\n\u250f\u2501...\n...Analysis...\n\u2517\u2501...\n<BLANKLINE>\n<BLANKLINE>\n DAGs OrbiterEmptyOperator\n \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n foo.py 1 1\n baz.py 1 1\n Totals 2 2\n<BLANKLINE>\n
Source code in orbiter/objects/project.py @validate_call\ndef analyze(\n self, output_fmt: Literal[\"json\", \"csv\", \"md\"] = \"md\", output_file=None\n):\n \"\"\"Print an analysis of the project to the console.\n\n !!! tip\n\n Looks for a specific `[task_type=XYZ]` in the Task's `doc_md` property\n or uses `type(task)` to infer the type of task.\n\n ```pycon\n >>> from orbiter.objects.operators.empty import OrbiterEmptyOperator\n >>> OrbiterProject().add_dags([\n ... OrbiterDAG(file_path=\"\", dag_id=\"foo\", orbiter_kwargs={\"file_path\": \"foo.py\"},\n ... tasks={\"bar\": OrbiterEmptyOperator(task_id=\"bar\")}\n ... ),\n ... OrbiterDAG(file_path=\"\", dag_id=\"baz\", orbiter_kwargs={\"file_path\": \"baz.py\"},\n ... tasks={\"bop\": OrbiterEmptyOperator(task_id=\"bop\")}\n ... )\n ... ]).analyze()\n ... # doctest: +ELLIPSIS\n \u250f\u2501...\n ...Analysis...\n \u2517\u2501...\n <BLANKLINE>\n <BLANKLINE>\n DAGs OrbiterEmptyOperator\n \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\n foo.py 1 1\n baz.py 1 1\n Totals 2 2\n <BLANKLINE>\n\n ```\n \"\"\"\n if output_file is None:\n output_file = sys.stdout\n\n _task_type = re.compile(r\"\\[task_type=(?P<task_type>[A-Za-z0-9-_]+)\")\n\n def get_task_type(task):\n match = _task_type.match(getattr(task, \"doc_md\", None) or \"\")\n match_or_task_type = (\n match.groupdict().get(\"task_type\") if match else None\n ) or type(task).__name__\n return match_or_task_type\n\n dag_analysis = [\n {\n \"file\": dag.orbiter_kwargs.get(\"file_path\", dag.file_path),\n \"dag_id\": dag.dag_id,\n \"task_types\": [get_task_type(task) for task in dag.tasks.values()],\n }\n for dag in self.dags.values()\n ]\n\n file_analysis = {}\n for analysis in dag_analysis:\n analysis_output = file_analysis.get(analysis[\"file\"], {})\n analysis_output[\"DAGs\"] = analysis_output.get(\"DAGs\", 0) + 1\n tasks_of_type = reduce(\n lambda acc, task_type: acc | {task_type: acc.get(task_type, 0) + 1},\n analysis[\"task_types\"],\n dict(),\n )\n analysis_output |= tasks_of_type\n file_analysis[analysis[\"file\"]] = analysis_output\n\n file_analysis = [{\"\": k} | v for k, v in file_analysis.items()]\n totals = {\"\": \"Totals\"}\n for file in file_analysis:\n for k, v in file.items():\n if k != \"\":\n totals[k] = totals.get(k, 0) + v\n file_analysis.append(totals)\n\n if output_fmt == \"json\":\n import json\n\n json.dump(file_analysis, output_file, default=str)\n elif output_fmt == \"csv\":\n import csv\n\n writer = csv.DictWriter(output_file, fieldnames={\"\"} | totals.keys())\n writer.writeheader()\n writer.writerows(file_analysis)\n elif output_fmt == \"md\":\n from rich.console import Console\n from rich.markdown import Markdown\n from tabulate import tabulate\n\n console = Console(file=output_file)\n\n # DAGs EmptyOp\n # file_a 1 1\n table = tabulate(\n tabular_data=file_analysis,\n headers=\"keys\",\n tablefmt=\"pipe\",\n # https://github.com/Textualize/rich/issues/3027\n missingval=\"\u2800\", # (special 'braille space' character)\n )\n console.print(\n Markdown(\n f\"# Analysis\\n{table}\",\n style=\"magenta\",\n )\n )\n
"},{"location":"objects/workflow/","title":"Workflow","text":"Airflow workflows are represented by a DAG which is a Directed Acyclic Graph (of Tasks). "},{"location":"objects/workflow/#diagram","title":"Diagram","text":"classDiagram\n direction LR\n class OrbiterRequirement[\"orbiter.objects.requirement.OrbiterRequirement\"] {\n package: str | None\n module: str | None\n names: List[str] | None\n sys_package: str | None\n }\n\n OrbiterDAG \"via schedule\" --> OrbiterTimetable\n OrbiterDAG --> \"many\" OrbiterOperator\n OrbiterDAG --> \"many\" OrbiterTaskGroup\n OrbiterDAG --> \"many\" OrbiterRequirement\n class OrbiterDAG[\"orbiter.objects.dag.OrbiterDAG\"] {\n imports: List[OrbiterRequirement]\n file_path: str\n dag_id: str\n schedule: str | OrbiterTimetable | None\n catchup: bool\n start_date: DateTime\n tags: List[str]\n default_args: Dict[str, Any]\n params: Dict[str, Any]\n doc_md: str | None\n tasks: Dict[str, OrbiterOperator]\n kwargs: dict\n orbiter_kwargs: dict\n orbiter_conns: Set[OrbiterConnection]\n orbiter_vars: Set[OrbiterVariable]\n orbiter_env_vars: Set[OrbiterEnvVar]\n orbiter_includes: Set[OrbiterInclude]\n }\n click OrbiterDAG href \"#orbiter.objects.dag.OrbiterDAG\" \"OrbiterDAG Documentation\"\n\n OrbiterTaskGroup --> \"many\" OrbiterRequirement\n class OrbiterTaskGroup[\"orbiter.objects.task.OrbiterTaskGroup\"] {\n task_group_id: str\n tasks: List[OrbiterOperator | OrbiterTaskGroup]\n add_downstream(str | List[str] | OrbiterTaskDependency)\n }\n\n OrbiterOperator --> \"many\" OrbiterRequirement\n OrbiterOperator --> \"one\" OrbiterPool\n OrbiterOperator --> \"many\" OrbiterConnection\n OrbiterOperator --> \"many\" OrbiterVariable\n OrbiterOperator --> \"many\" OrbiterEnvVar\n class OrbiterOperator[\"orbiter.objects.task.OrbiterOperator\"] {\n imports: List[OrbiterRequirement]\n operator: str\n task_id: str\n pool: str | None\n pool_slots: int | None\n trigger_rule: str | None\n downstream: Set[str]\n add_downstream(str | List[str] | OrbiterTaskDependency)\n }\n\n class OrbiterTimetable[\"orbiter.objects.timetables.OrbiterTimetable\"] {\n imports: List[OrbiterRequirements]\n orbiter_includes: Set[OrbiterIncludes]\n **kwargs: dict\n }\n click OrbiterTimetable href \"#orbiter.objects.timetables.OrbiterTimetable\" \"OrbiterTimetable Documentation\"\n\n class OrbiterConnection[\"orbiter.objects.connection.OrbiterConnection\"] {\n conn_id: str\n conn_type: str\n **kwargs\n }\n\n class OrbiterEnvVar[\"orbiter.objects.env_var.OrbiterEnvVar\"] {\n key: str\n value: str\n }\n\n class OrbiterPool[\"orbiter.objects.pool.OrbiterPool\"] {\n name: str\n description: str | None\n slots: int | None\n }\n\n class OrbiterRequirement[\"orbiter.objects.requirement.OrbiterRequirement\"] {\n package: str | None\n module: str | None\n names: List[str] | None\n sys_package: str | None\n }\n\n class OrbiterVariable[\"orbiter.objects.variable.OrbiterVariable\"] {\n key: str\n value: str\n }
"},{"location":"objects/workflow/#orbiter.objects.dag.OrbiterDAG","title":"orbiter.objects.dag.OrbiterDAG","text":"Represents an Airflow DAG, with its tasks and dependencies. Renders to a .py file in the /dags folder Parameters: Name Type Description file_path str File path of the DAG, relative to the /dags folder (filepath=my_dag.py would render to dags/my_dag.py) dag_id str The dag_id. Must be unique and snake_case. Good practice is to set dag_id == file_path schedule str | OrbiterTimetable, optional The schedule for the DAG. Defaults to None (only runs when manually triggered) catchup bool, optional Whether to catchup runs from the start_date to now, on first run. Defaults to False start_date DateTime, optional The start date for the DAG. Defaults to Unix Epoch tags List[str], optional Tags for the DAG, used for sorting and filtering in the Airflow UI default_args Dict[str, Any], optional Default arguments for any tasks in the DAG params Dict[str, Any], optional Params for the DAG doc_md str, optional Documentation for the DAG with markdown support kwargs dict, optional Additional keyword arguments to pass to the DAG **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/workflow/#orbiter.objects.dag.OrbiterDAG.add_tasks","title":"add_tasks","text":"add_tasks(\n tasks: (\n OrbiterOperator\n | OrbiterTaskGroup\n | Iterable[OrbiterOperator | OrbiterTaskGroup]\n ),\n) -> \"OrbiterDAG\"\n
Add one or more OrbiterOperators to the DAG >>> from orbiter.objects.operators.empty import OrbiterEmptyOperator\n>>> OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks(OrbiterEmptyOperator(task_id=\"bar\")).tasks\n{'bar': bar_task = EmptyOperator(task_id='bar')}\n\n>>> OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks([OrbiterEmptyOperator(task_id=\"bar\")]).tasks\n{'bar': bar_task = EmptyOperator(task_id='bar')}\n
Tip Validation requires a OrbiterTaskGroup, OrbiterOperator (or subclass), or list of either to be passed >>> # noinspection PyTypeChecker\n... OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks(\"bar\")\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n>>> # noinspection PyTypeChecker\n... OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks([\"bar\"])\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\npydantic_core._pydantic_core.ValidationError: ...\n
Parameters: Name Type Description tasks OrbiterOperator | OrbiterTaskGroup | Iterable[OrbiterOperator | OrbiterTaskGroup] List of OrbiterOperator, or OrbiterTaskGroup or subclass Returns: Type Description OrbiterProject self Source code in orbiter/objects/dag.py def add_tasks(\n self,\n tasks: (\n OrbiterOperator\n | OrbiterTaskGroup\n | Iterable[OrbiterOperator | OrbiterTaskGroup]\n ),\n) -> \"OrbiterDAG\":\n \"\"\"\n Add one or more [`OrbiterOperators`][orbiter.objects.task.OrbiterOperator] to the DAG\n\n ```pycon\n >>> from orbiter.objects.operators.empty import OrbiterEmptyOperator\n >>> OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks(OrbiterEmptyOperator(task_id=\"bar\")).tasks\n {'bar': bar_task = EmptyOperator(task_id='bar')}\n\n >>> OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks([OrbiterEmptyOperator(task_id=\"bar\")]).tasks\n {'bar': bar_task = EmptyOperator(task_id='bar')}\n\n ```\n\n !!! tip\n\n Validation requires a `OrbiterTaskGroup`, `OrbiterOperator` (or subclass), or list of either to be passed\n ```pycon\n >>> # noinspection PyTypeChecker\n ... OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks(\"bar\")\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n >>> # noinspection PyTypeChecker\n ... OrbiterDAG(file_path=\"\", dag_id=\"foo\").add_tasks([\"bar\"])\n ... # doctest: +IGNORE_EXCEPTION_DETAIL\n Traceback (most recent call last):\n pydantic_core._pydantic_core.ValidationError: ...\n\n ```\n :param tasks: List of [OrbiterOperator][orbiter.objects.task.OrbiterOperator], or OrbiterTaskGroup or subclass\n :type tasks: OrbiterOperator | OrbiterTaskGroup | Iterable[OrbiterOperator | OrbiterTaskGroup]\n :return: self\n :rtype: OrbiterProject\n \"\"\"\n if (\n isinstance(tasks, OrbiterOperator)\n or isinstance(tasks, OrbiterTaskGroup)\n or issubclass(type(tasks), OrbiterOperator)\n ):\n tasks = [tasks]\n\n for task in tasks:\n try:\n task_id = getattr(task, \"task_id\", None) or getattr(\n task, \"task_group_id\"\n )\n except AttributeError:\n raise AttributeError(\n f\"Task {task} does not have a task_id or task_group_id attribute\"\n )\n self.tasks[task_id] = task\n return self\n
"},{"location":"objects/workflow/#timetables","title":"Timetables","text":""},{"location":"objects/workflow/#orbiter.objects.timetables.OrbiterTimetable","title":"orbiter.objects.timetables.OrbiterTimetable","text":"An Airflow Timetable reference. Utilizes OrbiterInclude to add a file to a /plugins folder to register the timetable. Parameters: Name Type Description **kwargs any other kwargs to provide to Timetable "},{"location":"objects/workflow/#orbiter.objects.timetables.multi_cron_timetable","title":"orbiter.objects.timetables.multi_cron_timetable","text":""},{"location":"objects/workflow/#orbiter.objects.timetables.multi_cron_timetable.OrbiterMultiCronTimetable","title":"orbiter.objects.timetables.multi_cron_timetable.OrbiterMultiCronTimetable","text":"An Airflow Timetable that can be supplied with multiple cron strings. >>> OrbiterMultiCronTimetable(cron_defs=[\"*/5 * * * *\", \"*/7 * * * *\"])\nMultiCronTimetable(cron_defs=['*/5 * * * *', '*/7 * * * *'])\n
Parameters: Name Type Description cron_defs List[str] A list of cron strings timezone str The timezone to use for the timetable period_length int The length of the period period_unit str The unit of the period "},{"location":"objects/Tasks/","title":"Overview","text":"Airflow Tasks are units of work. An Operator is a pre-defined task with specific functionality. Operators can be looked up in the Astronomer Registry. The easiest way to create an operator in a translation to use an existing subclass of OrbiterOperator (e.g. OrbiterBashOperator). If an OrbiterOperator subclass doesn't exist for your use case, you can: 1) Utilize OrbiterTask from orbiter.objects.requirement import OrbiterRequirement\nfrom orbiter.objects.task import OrbiterTask\nfrom orbiter.rules import task_rule\n\n@task_rule\ndef my_rule(val: dict):\n return OrbiterTask(\n task_id=\"my_task\",\n imports=[OrbiterRequirement(\n package=\"apache-airflow\",\n module=\"airflow.operators.trigger_dagrun\",\n names=[\"TriggerDagRunOperator\"],\n )],\n ...\n )\n
2) Create a new subclass of OrbiterOperator, which can be beneficial if you are using it frequently in separate @task_rules from orbiter.objects.task import OrbiterOperator\nfrom orbiter.objects.requirement import OrbiterRequirement\nfrom orbiter.rules import task_rule\n\nclass OrbiterTriggerDagRunOperator(OrbiterOperator):\n # Define the imports required for the operator, and the operator name\n imports = [\n OrbiterRequirement(\n package=\"apache-airflow\",\n module=\"airflow.operators.trigger_dagrun\",\n names=[\"TriggerDagRunOperator\"],\n )\n ]\n operator: str = \"PythonOperator\"\n\n # Add fields should be rendered in the output\n render_attributes = OrbiterOperator.render_attributes + [\n ...\n ]\n\n # Add the fields that are required for the operator here, with their types\n # Not all Airflow Operator fields are required, just the ones you will use.\n trigger_dag_id: str\n ...\n\n@task_rule\ndef my_rule(val: dict):\n return OrbiterTriggerDagRunOperator(...)\n
"},{"location":"objects/Tasks/#diagram","title":"Diagram","text":"classDiagram\n direction LR\n OrbiterOperator \"implements\" <|-- OrbiterTask\n OrbiterOperator --> \"many\" OrbiterCallback\n class OrbiterOperator[\"orbiter.objects.task.OrbiterOperator\"] {\n imports: List[OrbiterRequirement]\n operator: str\n task_id: str\n pool: str | None\n pool_slots: int | None\n trigger_rule: str | None\n downstream: Set[str]\n add_downstream(str | List[str] | OrbiterTaskDependency)\n }\n click OrbiterOperator href \"#orbiter.objects.task.OrbiterOperator\" \"OrbiterOperator Documentation\"\n\n class OrbiterTask[\"orbiter.objects.task.OrbiterTask\"] {\n <<OrbiterOperator>>\n <<OrbiterOperator>>\n imports: List[OrbiterRequirement]\n task_id: str\n **kwargs\n }\n click OrbiterTask href \"#orbiter.objects.task.OrbiterTask\" \"OrbiterTask Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterBashOperator\n class OrbiterBashOperator[\"orbiter.objects.operators.bash.OrbiterBashOperator\"] {\n <<OrbiterOperator>>\n operator = \"BashOperator\"\n task_id: str\n bash_command: str\n }\n click OrbiterBashOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.bash.OrbiterBashOperator\" \"OrbiterBashOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterEmailOperator\n class OrbiterEmailOperator[\"orbiter.objects.operators.smtp.OrbiterEmailOperator\"] {\n <<OrbiterOperator>>\n operator = \"EmailOperator\"\n task_id: str\n to: str | list[str]\n subject: str\n html_content: str\n files: list | None\n conn_id: str\n }\n click OrbiterEmailOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.smtp.OrbiterEmailOperator\" \"OrbiterEmailOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterEmptyOperator\n class OrbiterEmptyOperator[\"orbiter.objects.operators.empty.OrbiterEmptyOperator\"] {\n <<OrbiterOperator>>\n operator = \"BashOperator\"\n task_id: str\n }\n click OrbiterEmptyOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.empty.OrbiterEmptyOperator\" \"OrbiterEmptyOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterPythonOperator\n class OrbiterPythonOperator[\"orbiter.objects.operators.python.OrbiterPythonOperator\"] {\n <<OrbiterOperator>>\n operator = \"PythonOperator\"\n task_id: str\n python_callable: Callable\n op_args: list | None\n op_kwargs: dict | None\n }\n click OrbiterPythonOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.python.OrbiterPythonOperator\" \"OrbiterPythonOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterSQLExecuteQueryOperator\n class OrbiterSQLExecuteQueryOperator[\"orbiter.objects.operators.sql.OrbiterSQLExecuteQueryOperator\"] {\n <<OrbiterOperator>>\n operator = \"SQLExecuteQueryOperator\"\n task_id: str\n conn_id: str\n sql: str\n }\n click OrbiterSQLExecuteQueryOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.sql.OrbiterSQLExecuteQueryOperator\" \"OrbiterSQLExecuteQueryOperator Documentation\"\n\n OrbiterOperator \"implements\" <|-- OrbiterSSHOperator\n class OrbiterSSHOperator[\"orbiter.objects.operators.ssh.OrbiterSSHOperator\"] {\n <<OrbiterOperator>>\n operator = \"SSHOperator\"\n task_id: str\n ssh_conn_id: str\n command: str\n environment: Dict[str, str] | None\n }\n click OrbiterSSHOperator href \"Operators_and_Callbacks/operators#orbiter.objects.operators.ssh.OrbiterSSHOperator\" \"OrbiterSSHOperator Documentation\"\n\n class OrbiterCallback[\"orbiter.objects.callbacks.OrbiterCallback\"] {\n function: str\n }\n click OrbiterCallback href \"Operators_and_Callbacks/callbacks#orbiter.objects.callbacks.OrbiterCallback\" \"OrbiterCallback Documentation\"\n\n OrbiterCallback \"implements\" <|-- OrbiterSmtpNotifierCallback\n class OrbiterSmtpNotifierCallback[\"orbiter.objects.callbacks.smtp.OrbiterSmtpNotifierCallback\"] {\n <<OrbiterCallback>>\n to: str\n from_email: str\n smtp_conn_id: str\n subject: str\n html_content: str\n cc: str | Iterable[str]\n }\n click OrbiterSmtpNotifierCallback href \"Operators_and_Callbacks/callbacks#orbiter.objects.callbacks.smtp.OrbiterSmtpNotifierCallback\" \"OrbiterSmtpNotifierCallback Documentation\"
"},{"location":"objects/Tasks/#orbiter.objects.task.OrbiterOperator","title":"orbiter.objects.task.OrbiterOperator","text":"Abstract class representing a Task in Airflow. Must be subclassed (such as OrbiterBashOperator, or OrbiterTask). Subclassing Example: >>> from orbiter.objects import OrbiterRequirement\n>>> class OrbiterMyOperator(OrbiterOperator):\n... imports: ImportList = [OrbiterRequirement(package=\"apache-airflow\")]\n... operator: str = \"MyOperator\"\n\n>>> foo = OrbiterMyOperator(task_id=\"task_id\"); foo\ntask_id_task = MyOperator(task_id='task_id')\n
Adding single downstream tasks: >>> from orbiter.ast_helper import render_ast\n>>> render_ast(foo.add_downstream(\"downstream\")._downstream_to_ast())\n'task_id_task >> downstream_task'\n
Adding multiple downstream tasks: >>> render_ast(foo.add_downstream([\"a\", \"b\"])._downstream_to_ast())\n'task_id_task >> [a_task, b_task, downstream_task]'\n
Note Validation - task_id in OrbiterTaskDependency must match this task_id >>> foo.add_downstream(OrbiterTaskDependency(task_id=\"other\", downstream=\"bar\")).downstream\n... # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\nValueError: Task dependency ... has a different task_id than task_id\n
Parameters: Name Type Description imports List[OrbiterRequirement] List of requirements for the operator task_id str The task_id for the operator, must be unique and snake_case trigger_rule str, optional Conditions under which to start the task (docs) pool str, optional Name of the pool to use pool_slots int, optional Slots for this task to occupy operator str, optional Operator name downstream Set[str], optional Downstream tasks, defaults to set() **kwargs Other properties that may be passed to operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/#orbiter.objects.task.OrbiterTaskDependency","title":"orbiter.objects.task.OrbiterTaskDependency","text":"Represents a task dependency, which is added to either an OrbiterOperator or an OrbiterTaskGroup. Parameters: Name Type Description task_id str The task_id for the operator downstream str | List[str] downstream task(s) "},{"location":"objects/Tasks/#orbiter.objects.task.OrbiterTask","title":"orbiter.objects.task.OrbiterTask","text":"A generic version of OrbiterOperator that can be instantiated directly. The operator that is instantiated is inferred from the imports field, via the first *Operator or *Sensor import. View info for specific operators at the Astronomer Registry. >>> from orbiter.objects.requirement import OrbiterRequirement\n>>> OrbiterTask(task_id=\"foo\", bash_command=\"echo 'hello world'\", other=1, imports=[\n... OrbiterRequirement(package=\"apache-airflow\", module=\"airflow.operators.bash\", names=[\"BashOperator\"])\n... ])\nfoo_task = BashOperator(task_id='foo', bash_command=\"echo 'hello world'\", other=1)\n\n>>> def foo():\n... pass\n>>> OrbiterTask(task_id=\"foo\", python_callable=foo, other=1, imports=[\n... OrbiterRequirement(package=\"apache-airflow\", module=\"airflow.sensors.python\", names=[\"PythonSensor\"])\n... ])\ndef foo():\n pass\nfoo_task = PythonSensor(task_id='foo', other=1, python_callable=foo)\n
Parameters: Name Type Description task_id str The task_id for the operator. Must be unique and snake_case imports List[OrbiterRequirement] List of requirements for the operator. The Operator is inferred from first *Operator or *Sensor imported. **kwargs Any other keyword arguments to be passed to the operator "},{"location":"objects/Tasks/#orbiter.objects.task_group.OrbiterTaskGroup","title":"orbiter.objects.task_group.OrbiterTaskGroup","text":"Represents a TaskGroup in Airflow, which contains multiple tasks >>> from orbiter.objects.operators.bash import OrbiterBashOperator\n>>> from orbiter.ast_helper import render_ast\n>>> OrbiterTaskGroup(task_group_id=\"foo\", tasks=[\n... OrbiterBashOperator(task_id=\"b\", bash_command=\"b\"),\n... OrbiterBashOperator(task_id=\"a\", bash_command=\"a\").add_downstream(\"b\"),\n... ], downstream={\"c\"})\nwith TaskGroup(group_id='foo') as foo:\n b_task = BashOperator(task_id='b', bash_command='b')\n a_task = BashOperator(task_id='a', bash_command='a')\n a_task >> b_task\n\n>>> render_ast(OrbiterTaskGroup(task_group_id=\"foo\", tasks=[], downstream={\"c\"})._downstream_to_ast())\n'foo >> c_task'\n
Parameters: Name Type Description task_group_id str The id of the TaskGroup tasks List[OrbiterOperator | OrbiterTaskGroup] The tasks in the TaskGroup **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/callbacks/","title":"Callbacks","text":"Airflow callback functions are often used to send emails, slack messages, or other notifications when a task fails, succeeds, or is retried. They can also run any general Python function. "},{"location":"objects/Tasks/Operators_and_Callbacks/callbacks/#orbiter.objects.callbacks.OrbiterCallback","title":"orbiter.objects.callbacks.OrbiterCallback","text":"Represents an Airflow callback function, which might be used in DAG.on_failure_callback, or Task.on_success_callback, or etc. Can be instantiated directly as a bare callback function (with no arguments): >>> from orbiter.objects.dag import OrbiterDAG\n>>> from orbiter.objects.include import OrbiterInclude\n>>> my_callback = OrbiterCallback(\n... function=\"my_callback\",\n... imports=[OrbiterRequirement(module=\"my_callback\", names=[\"my_callback\"])],\n... orbiter_includes={OrbiterInclude(filepath=\"my_callback.py\", contents=\"...\")}\n... )\n>>> OrbiterDAG(dag_id='', file_path='', on_failure_callback=my_callback)\n... # doctest: +ELLIPSIS\nfrom airflow import DAG\nfrom my_callback import my_callback\n...\nwith DAG(... on_failure_callback=my_callback):\n
or be subclassed: >>> class OrbiterMyCallback(OrbiterCallback):\n... function: str = \"my_callback\"\n... foo: str\n... bar: str\n... render_attributes: RenderAttributes = [\"foo\", \"bar\"]\n>>> OrbiterMyCallback(foo=\"fop\", bar=\"bop\")\nmy_callback(foo='fop', bar='bop')\n
Parameters: Name Type Description function str The name of the function to call **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/callbacks/#orbiter.objects.callbacks.smtp","title":"orbiter.objects.callbacks.smtp","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/callbacks/#orbiter.objects.callbacks.smtp.OrbiterSmtpNotifierCallback","title":"orbiter.objects.callbacks.smtp.OrbiterSmtpNotifierCallback","text":"An Airflow SMTP Callback Note Use smtp_conn_id and reference an SMTP Connection. You can use the **conn_id(\"SMTP\", conn_type=\"smtp\") utility function to set both properties at once. >>> [_import] = OrbiterSmtpNotifierCallback(to=\"foo@test.com\").imports; _import\nOrbiterRequirements(names=[send_smtp_notification], package=apache-airflow-providers-smtp, module=airflow.providers.smtp.notifications.smtp, sys_package=None)\n\n>>> OrbiterSmtpNotifierCallback(to=\"foo@test.com\", from_email=\"bar@test.com\", subject=\"Hello\", html_content=\"World\")\nsend_smtp_notification(to='foo@test.com', from_email='bar@test.com', smtp_conn_id='SMTP', subject='Hello', html_content='World')\n
Parameters: Name Type Description to str | Iterable[str] The email address to send to from_email str, optional The email address to send from smtp_conn_id str, optional The connection id to use (Note: use the **conn_id(...) utility function). Defaults to \"SMTP\" subject str, optional The subject of the email html_content str, optional The content of the email cc str | Iterable[str], optional The email address to cc **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/","title":"Operators","text":"Note These operators are included and are intended to represent some of the most common Airflow Operators, but not all Airflow Operators. Additional Operators can be created by subclassing OrbiterOperator or using OrbiterTask directly. Review the Astronomer Registry to find additional Airflow Operators. "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.bash","title":"orbiter.objects.operators.bash","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.bash.OrbiterBashOperator","title":"orbiter.objects.operators.bash.OrbiterBashOperator","text":" Bases: OrbiterOperator An Airflow BashOperator. Used to run shell commands. >>> OrbiterBashOperator(task_id=\"foo\", bash_command=\"echo 'hello world'\")\nfoo_task = BashOperator(task_id='foo', bash_command=\"echo 'hello world'\")\n
Parameters: Name Type Description task_id str The task_id for the operator bash_command str The shell command to execute **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.empty","title":"orbiter.objects.operators.empty","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.empty.OrbiterEmptyOperator","title":"orbiter.objects.operators.empty.OrbiterEmptyOperator","text":" Bases: OrbiterOperator An Airflow EmptyOperator. Does nothing. >>> OrbiterEmptyOperator(task_id=\"foo\")\nfoo_task = EmptyOperator(task_id='foo')\n
Parameters: Name Type Description task_id str The task_id for the operator **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.python","title":"orbiter.objects.operators.python","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.python.OrbiterPythonOperator","title":"orbiter.objects.operators.python.OrbiterPythonOperator","text":" Bases: OrbiterOperator An Airflow PythonOperator. Used to execute any Python Function. >>> def foo(a, b):\n... print(a + b)\n>>> OrbiterPythonOperator(task_id=\"foo\", python_callable=foo)\ndef foo(a, b):\n print(a + b)\nfoo_task = PythonOperator(task_id='foo', python_callable=foo)\n
Parameters: Name Type Description task_id str The task_id for the operator python_callable Callable The python function to execute op_args list | None, optional The arguments to pass to the python function, defaults to None op_kwargs dict | None, optional The keyword arguments to pass to the python function, defaults to None **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.smtp","title":"orbiter.objects.operators.smtp","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.smtp.OrbiterEmailOperator","title":"orbiter.objects.operators.smtp.OrbiterEmailOperator","text":" Bases: OrbiterOperator An Airflow EmailOperator. Used to send emails. >>> OrbiterEmailOperator(\n... task_id=\"foo\", to=\"humans@astronomer.io\", subject=\"Hello\", html_content=\"World!\"\n... )\nfoo_task = EmailOperator(task_id='foo', to='humans@astronomer.io', subject='Hello', html_content='World!', conn_id='SMTP')\n
Parameters: Name Type Description task_id str The task_id for the operator to str | list[str] The recipient of the email subject str The subject of the email html_content str The content of the email files list, optional The files to attach to the email, defaults to None conn_id str, optional The SMTP connection to use. Defaults to \"SMTP\" and sets orbiter_conns property **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.sql","title":"orbiter.objects.operators.sql","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.sql.OrbiterSQLExecuteQueryOperator","title":"orbiter.objects.operators.sql.OrbiterSQLExecuteQueryOperator","text":" Bases: OrbiterOperator An Airflow Generic SQL Operator. Used to run SQL against any Database. >>> OrbiterSQLExecuteQueryOperator(\n... task_id=\"foo\", conn_id='sql', sql=\"select 1;\"\n... )\nfoo_task = SQLExecuteQueryOperator(task_id='foo', conn_id='sql', sql='select 1;')\n
Parameters: Name Type Description task_id str The task_id for the operator conn_id str The SQL connection to utilize. (Note: use the **conn_id(...) utility function) sql str The SQL to execute **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.ssh","title":"orbiter.objects.operators.ssh","text":""},{"location":"objects/Tasks/Operators_and_Callbacks/operators/#orbiter.objects.operators.ssh.OrbiterSSHOperator","title":"orbiter.objects.operators.ssh.OrbiterSSHOperator","text":" Bases: OrbiterOperator An Airflow SSHOperator. Used to run shell commands over SSH. >>> OrbiterSSHOperator(task_id=\"foo\", ssh_conn_id=\"SSH\", command=\"echo 'hello world'\")\nfoo_task = SSHOperator(task_id='foo', ssh_conn_id='SSH', command=\"echo 'hello world'\")\n
Parameters: Name Type Description task_id str The task_id for the operator ssh_conn_id str The SSH connection to use. (Note: use the **conn_id(...) utility function) command str The command to execute environment dict, optional The environment variables to set, defaults to None **kwargs Extra arguments to pass to the operator **OrbiterBase OrbiterBase inherited properties "}]}
\ No newline at end of file
diff --git a/sitemap.xml b/sitemap.xml
index fbfb3fe..19265c2 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -2,67 +2,67 @@
https://astronomer.github.io/orbiter/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/cli/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/origins/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/Rules_and_Rulesets/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/Rules_and_Rulesets/rules/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/Rules_and_Rulesets/rulesets/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/Rules_and_Rulesets/template/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/objects/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/objects/project/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/objects/workflow/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/objects/Tasks/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/objects/Tasks/Operators_and_Callbacks/callbacks/
- 2024-08-28
+ 2024-09-05
daily
https://astronomer.github.io/orbiter/objects/Tasks/Operators_and_Callbacks/operators/
- 2024-08-28
+ 2024-09-05
daily
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 6437cc7..669d5eb 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ
|