New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix compute_cost_per_minute #63

Merged

Kayrnt merged 1 commit into main from hourly-compute-fix

Nov 16, 2024

Collaborator

Kayrnt commented Nov 16, 2024 •

edited

Loading

compute_cost_per_minute model is partitioned by hour and leverage insert_overwrite so let's assume we have following setup:

First run at 1:30 AM, we get the data from 0 to 1:30
Then we run again at 3 AM
The model will read max partition = 1:30 AM
With former version, it would truncate at the minute then we read 1:30 to 2 AM but with the fixed one we will read from 1 AM to 2 AM.
With that data, we apply the group by minute and store the result on the 1 AM partition

So if we don't apply that fix, the second run would remove the data from 1 AM to 1:30 AM ending up with partial data.


          fix compute_cost_per_minute

ede890a

Kayrnt temporarily deployed to Approve Integration Tests

November 16, 2024 18:00

— with

GitHub Actions Inactive

Kayrnt temporarily deployed to Approve Integration Tests

November 16, 2024 18:00

— with

GitHub Actions Inactive

Kayrnt marked this pull request as ready for review

November 16, 2024 18:04

Kayrnt merged commit 76d1b06 into main

3 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet