feat: calculate usage incrementally #436
Conversation
apsantiso
temporarily deployed
to
development
GitHub Actions
Inactive
apsantiso
temporarily deployed
to
development
GitHub Actions
Inactive
migrations/20241120004033-create-calculate-last-day-usage-procedure.js
Outdated
Show resolved
Hide resolved
apsantiso
temporarily deployed
to
development
GitHub Actions
Inactive
| LEFT JOIN ( | ||
| SELECT user_id, MAX(period) AS last_period | ||
| FROM public.usages | ||
| GROUP BY user_id | ||
| ) mru |
There was a problem hiding this comment.
If a user does not have a record in usage, we would like not to calculate their usage so we can reduce edge cases such as an user having their last day usage calculated but files created before this implemented being ignored.
There was a problem hiding this comment.
I do not get that. If we create the first row on demand for already existing users, which could be the issue?
src/modules/usage/usage.usecase.ts
Outdated
| let mostRecentUsage = | ||
| await this.usageRepository.getMostRecentUsage(userUuid); | ||
|
|
||
| if (!mostRecentUsage) { | ||
| mostRecentUsage = await this.usageRepository.addFirstDailyUsage(userUuid); | ||
| } |
There was a problem hiding this comment.
If users do not have any usage row, we create one for them using all the files created until the last day. Related to: https://github.com/internxt/drive-server-wip/pull/436/files#r1858576469
apsantiso
force-pushed
the
feat/calculate-usage-incrementally
branch
6 times, most recently
from
73fd5ed to
0058b03
Compare
…ount towards usage calculation
apsantiso
force-pushed
the
feat/calculate-usage-incrementally
branch
from
0058b03 to
b3100c6
Compare
apsantiso
temporarily deployed
to
development
GitHub Actions
Inactive
apsantiso
force-pushed
the
feat/calculate-usage-incrementally
branch
from
f032e0f to
2f61538
Compare
apsantiso
temporarily deployed
to
development
GitHub Actions
Inactive
apsantiso
force-pushed
the
feat/calculate-usage-incrementally
branch
from
2f61538 to
f968a38
Compare
apsantiso
temporarily deployed
to
development
GitHub Actions
Inactive
|
The current method of calculating usage is overloading the database. Looping through all user-created files with a status other than "deleted" is unsustainable. This PR introduces the following improvements:
Scheduled Procedures: Introduce two procedures that should be triggered by a cron job:
00:00:00:000and23:59:59:9999Daily Cron Job Considerations: Ensure the daily procedure accounts for files that are created and deleted on the same day, where the net change (delta) would be zero.
Initial Monthly Row: Insert an initial monthly row the first time a user accesses the new /usage endpoint. This row reflects the total size of all the user's existing files until the last day (so it contains all files users created since they created their account until the last day)
Conditional Monthly Calculations: Begin generating monthly rows via the daily cron job only after the initial calculation is complete (i.e., when the user has at least one monthly row).
/usage endpoint uses all the rows inside usages table and also starts calculating files created - deleted since the most recent usage period (i.e, last usage period is 2024-11-24, we calculate on the fly the usage of files since 2024-11-25 until the actual date). Still pending to take into account if the last usage is a yearly row.