Skip to content

v0.17.0
Compare
Choose a tag to compare
@anshuman-gojek anshuman-gojek released this 21 Oct 14:22
· 8 commits to main since this release
v0.17.0
b52a9f1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Main Changes

Replay New Features and Bugs

Replay Life-Cycle Events Notification Subscription

Optimus now emits notification events upon changes in replay state. Supported lifecycle events include:

  1. Success
  2. Failure
  3. Cancelation
  4. Timeout
  • Changelog Reference:
    • Commit: a7bb2c56 - Feat: Add replay lifecycle events (#271) (#275)

Support Replay Cancelation of Runs

Replay Cancelation now also stop runs on the scheduler

  • Changelog Reference:
    • Commit: b88fa7f7 - Replay cancel runs (#273) (#276)
    • Commit: b52a9f19 - Replay cancel scheduled cleared reruns (#283)

Replay Restarts Across Server Restarts

  • Optimus now ensures that replays are handled properly across server restarts, allowing smoother continuation of processes after disruptions.

  • Changelog Reference:

    • Commit: b5d007ed - Handle replay across Optimus server restarts (#279)

New Job Run API Powered by Optimus DB

  • A new Job Run API has been introduced, leveraging the Optimus database to provide efficient access to job run details.

  • Changelog Reference:

    • Commit: 541691f9 - Get job run API (#281)

Add New Variables Field in Project & Namespace Specification

In previous versions, GLOBAL__ macros used in job specifications are fetched from project & namespace's config field. The usecase of the config field is also mixed with job deployment-related configurations (e.g. SCHEDULER_HOST, STORAGE_PATH).

To separate the usecase, GLOBAL__ macros are now fetched from a new field, variables, added in both project & namespace entity. The precedence behavior is kept the same: namespace variables will override project variables.

Variables can be defined in Optimus project configuration file: optimus.yaml.

version: 1
host: optimus:80
project:
  name: my-optimus-project
  config:
    scheduler_host: <airflow_host>
    scheduler_version: <airflow_version>
  variables:
    bq_data_project: bq_project
namespace:
- name: my-namespace
  config: {}
  variables:
    bq_data_project: bq_namespace_project
  • Changelog Reference:
    • Commit: ffd4deec - Add support for variables in project & support resource templating (#277)

Enhanced Optimus Resource Management

Optimus now introduces a new version of resource spec: version 2.

Changes in Resource Name & Resource URNs

In concept, this new version of resource spec "detaches" resource name from the Resource URN creation, which is used to create the actual resource in datastore.

This allows more flexibility for resource names definition that are no longer coupled with the datastore-specific implementation. Detailed changes in resource specification for each supported datastore can be seen below.

BigQuery

To define BigQuery resource names, instead of deriving resource URN from the resource name (<project>.<dataset>.<name>), it will be derived from 3 new fields inside spec: project, dataset, and name. Resource names will be used for unique identifier only. For example, to define a BigQuery table, below spec is used

version: 2
name: business_unit.table_name
type: table
labels: {}
spec:
  project: "my_data_project"
  dataset: "my_business_unit"
  name: "my_table_name"
  schema:
    ...

The spec above will result in an Optimus resource named business_unit.table_name, but the actual table created in BigQuery will be my_data_project.my_business_unit.my_table_name. Then the resulting resource URN will be bigquery://my_data_project:my_business_unit.my_table_name

Templating in Resource Spec

Like job config in specs, Resource Spec v2 also supports templating using GLOBAL__ macros. Combined with the new resource URN generation behavior in the above section, templating allows for resource parameterization based on the Optimus project target for the resource to be deployed.

Using spec templating with the above resource spec, the new spec can be written like this

version: 2
name: business_unit.table_name
type: table
labels: {}
spec:
  project: "{{.GLOBAL__BQ_DATA_PROJECT}}"
  dataset: "my_business_unit"
  name: "my_table_name"
  schema:
    ...

The spec will be compiled when the resource is created/updated, by referring to project & namespace variables.

  • Changelog Reference:
    • Commit: ffd4deec - Add support for variables in project & support resource templating (#277)
    • Commit: 51a43a27 - Support resource spec v2 (#274)

Bug Fixes and Refactoring

  • Several bug fixes and improvements were made, including refactoring metrics collection and enhancing logging and alert handling.

  • Changelog Reference:

    • Commit: d70c112b - Refactor metric collection (#254)

This update enhances Optimus with improved replay handling, new API functionalities, and better resource management, while also addressing key bugs and optimizations.

Assets 9
Loading