Releases: goto/optimus
v0.18.2
v0.18.0
v0.17.3
v0.17.2
v0.17.1
v0.17.0
Main Changes
Replay New Features and Bugs
Replay Life-Cycle Events Notification Subscription
Optimus now emits notification events upon changes in replay state. Supported lifecycle events include:
- Success
- Failure
- Cancelation
- Timeout
Support Replay Cancelation of Runs
Replay Cancelation now also stop runs on the scheduler
- Changelog Reference:
Replay Restarts Across Server Restarts
-
Optimus now ensures that replays are handled properly across server restarts, allowing smoother continuation of processes after disruptions.
-
Changelog Reference:
- Commit:
b5d007ed- Handle replay across Optimus server restarts (#279)
- Commit:
New Job Run API Powered by Optimus DB
-
A new Job Run API has been introduced, leveraging the Optimus database to provide efficient access to job run details.
-
Changelog Reference:
- Commit:
541691f9- Get job run API (#281)
- Commit:
Add New Variables Field in Project & Namespace Specification
In previous versions, GLOBAL__ macros used in job specifications are fetched from project & namespace's config field. The usecase of the config field is also mixed with job deployment-related configurations (e.g. SCHEDULER_HOST, STORAGE_PATH).
To separate the usecase, GLOBAL__ macros are now fetched from a new field, variables, added in both project & namespace entity. The precedence behavior is kept the same: namespace variables will override project variables.
Variables can be defined in Optimus project configuration file: optimus.yaml.
version: 1
host: optimus:80
project:
name: my-optimus-project
config:
scheduler_host: <airflow_host>
scheduler_version: <airflow_version>
variables:
bq_data_project: bq_project
namespace:
- name: my-namespace
config: {}
variables:
bq_data_project: bq_namespace_project- Changelog Reference:
- Commit:
ffd4deec- Add support for variables in project & support resource templating (#277)
- Commit:
Enhanced Optimus Resource Management
Optimus now introduces a new version of resource spec: version 2.
Changes in Resource Name & Resource URNs
In concept, this new version of resource spec "detaches" resource name from the Resource URN creation, which is used to create the actual resource in datastore.
This allows more flexibility for resource names definition that are no longer coupled with the datastore-specific implementation. Detailed changes in resource specification for each supported datastore can be seen below.
BigQuery
To define BigQuery resource names, instead of deriving resource URN from the resource name (<project>.<dataset>.<name>), it will be derived from 3 new fields inside spec: project, dataset, and name. Resource names will be used for unique identifier only. For example, to define a BigQuery table, below spec is used
version: 2
name: business_unit.table_name
type: table
labels: {}
spec:
project: "my_data_project"
dataset: "my_business_unit"
name: "my_table_name"
schema:
...The spec above will result in an Optimus resource named business_unit.table_name, but the actual table created in BigQuery will be my_data_project.my_business_unit.my_table_name. Then the resulting resource URN will be bigquery://my_data_project:my_business_unit.my_table_name
Templating in Resource Spec
Like job config in specs, Resource Spec v2 also supports templating using GLOBAL__ macros. Combined with the new resource URN generation behavior in the above section, templating allows for resource parameterization based on the Optimus project target for the resource to be deployed.
Using spec templating with the above resource spec, the new spec can be written like this
version: 2
name: business_unit.table_name
type: table
labels: {}
spec:
project: "{{.GLOBAL__BQ_DATA_PROJECT}}"
dataset: "my_business_unit"
name: "my_table_name"
schema:
...The spec will be compiled when the resource is created/updated, by referring to project & namespace variables.
- Changelog Reference:
Bug Fixes and Refactoring
-
Several bug fixes and improvements were made, including refactoring metrics collection and enhancing logging and alert handling.
-
Changelog Reference:
- Commit:
d70c112b- Refactor metric collection (#254)
- Commit:
This update enhances Optimus with improved replay handling, new API functionalities, and better resource management, while also addressing key bugs and optimizations.
v0.16.2
Main Changes
Support Capability to Ignore upstream checks while running Replays
With this release, users will be able to ignore the upstream wait sensor checks, by adding the following in replay jobConfig IGNORE_UPSTREAM=True
Prepare to support Airflow 2.9.x
Support for Airflow 2.9.x DAGs are added in this release. Backward compatibility for in-between versions are also supported, with current version support for 2.1, 2.4, 2.6, 2.9. So if a project's Airflow version is 2.7 or 2.8, it will use the DAG from version 2.6, 2.10 will use 2.9, and also for other below versions.
Support More Optimus Events
Optimus shall now Emit events on
- Job : Updation and Deletion
- Job Run : Success, Failure and SLA miss
- Resource : Updation and Deletion
[Refactor] Metric collection
- Changed the Job Run metrics
- Removed the need to maintain the Prometheus metrics maps and their mutex thereby simplifying metric collection flow
- Stopped tracking a few feature adoption metrics
Changelog
v0.16.0
Main Changes
Bulk Delete API & Integration with Optimus Apply for Job Deletion
With this release, single & multiple job deletions from Optimus Apply command will go through bulk delete API instead of delete single job specification API. This change was made so order of deletion can be applied
Optimus Job Validate: Add support for Model resource
Optimus job validate command will also validate BQ model resource used in BQ queries
Bugfix
- Add logical date validation in replay