Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Archival #538

Open
wants to merge 12 commits into
base: develop
Choose a base branch
from
Open

Archival #538

wants to merge 12 commits into from

Conversation

lukasjuhrich
Copy link
Collaborator

@lukasjuhrich lukasjuhrich commented Feb 21, 2022 •
edited
Loading

Prepares #490

  • Discuss whether we want to do this yearly or regularly && adapt code accordingly
  • Test archival selection
    • test select_user_and_last_mem() select (used as CTE)
    • test cutoff filtering
    • test cutoff calculation from current date
    • test do-not-archive filtering
  • Implement archive_users (cleanup)
  • Add a web endpoint and button archiving all current users
  • Add a celerybeat task executing the archival

@lukasjuhrich lukasjuhrich force-pushed the archival branch 5 times, most recently from b5b4131 to cf0d6b1 Compare August 5, 2022 04:06
@lukasjuhrich lukasjuhrich force-pushed the archival branch 2 times, most recently from 364a437 to bbe334a Compare September 29, 2022 11:18
@lukasjuhrich
Copy link
Collaborator Author

This PR, as of now, does some base work that can directly be merged into develop.
All the rest can be coordinated in one or multiple different PRs (see the extended tasklist of #67).
Waiting for the tests to finish.

@lukasjuhrich lukasjuhrich marked this pull request as ready for review January 15, 2023 18:05
@lukasjuhrich lukasjuhrich changed the title Archival Archival preparations Jan 15, 2023
georgeto
georgeto reviewed Jan 16, 2023
View reviewed changes
pycroft/lib/user_deletion.py
def get_invalidated_archive_memberships() -> list[Membership]:
"""Get all memberships in `to_be_archived` of users who have an active `do-not-archive` property.
Selected are those users
- whose last membership in the member_group ended two weeks in the past,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks do me like we are now testing for >= 2 years and not weeks.

@lukasjuhrich lukasjuhrich changed the title Archival preparations Archival Apr 27, 2023
@lukasjuhrich
Add rudimentary archival test
ddd0c83
@lukasjuhrich
Extract timedelta from archivable membership filter
8da8f9f
@lukasjuhrich
reintroduce cast to silence warning
c0848ac
@lukasjuhrich
Add test for longer delta in archivable query
74c074f
@lukasjuhrich
Add archive_users stub and test
0144646
@lukasjuhrich
Extract last membership select from archive query
2e99189 
Introduces `select_users_and_last_mem` returning the `select`
@lukasjuhrich
Switch membership selection to from … select style
c0f1036 
This suits the relational algebra pipeline better, as the entities we
select from actually correspond to a projection done after all the
joins and filtering.

This change does not alter the compilation result of this select statement.
@lukasjuhrich
Decouple archivable users select statement from execution
e509d76
@lukasjuhrich
Switch archival query to from … select style
cee036d
@lukasjuhrich
Add comment explaining window args
e39606b
@lukasjuhrich
Switch archivable member selection from delta to year-based cutoff
9c9a2e8 
Fixes an oversight in the previous implementation of #470

Also, expand on the test suite substantially while retaining a decent
runtime.

Refs #67
@lukasjuhrich
Improve archivability test performance
f715e29 
1. We restrict the ranges to a smaller subset which is still
   meaningful.
2. We use the `select` statement itself rather than the wrapper method
   adding a ton of `joinedload`s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@georgeto georgeto georgeto left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lukasjuhrich @georgeto