[24.1] Fix quota usage with user object stores

[davelopez]

Attempt to fix #19320

How to test the changes?

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[davelopez]

I'm not sure if 9c29c90 is the proper fix, probably not. It does fix the issue if I try to recalculate my quota in the UI, but it breaks other quota usage unit tests in a way I don't understand...

I'll keep investigating, but any hint or confirmation would be much appreciated 😃

[davelopez]

The resulting SQL query is something like this:

UPDATE galaxy_user SET disk_usage = (
WITH per_user_histories AS
(
    SELECT id
    FROM history
    WHERE user_id = :id
        AND NOT purged
),
per_hist_hdas AS (
    SELECT DISTINCT dataset_id
    FROM history_dataset_association
    WHERE NOT purged
        AND history_id IN (SELECT id FROM per_user_histories)
)
SELECT COALESCE(SUM(COALESCE(dataset.total_size, dataset.file_size, 0)), 0)
FROM dataset
LEFT OUTER JOIN library_dataset_dataset_association ON dataset.id = library_dataset_dataset_association.dataset_id
WHERE dataset.id IN (SELECT dataset_id FROM per_hist_hdas)
    AND library_dataset_dataset_association.id IS NULL
    AND dataset.object_store_id NOT LIKE 'user_objects://%'
      
)
WHERE id = :id

It works both in Galaxy and directly against the database, but it always returns 0 in the unit tests 🤔

[jmchilton]

I would think the "correct solution" would be to gather all the object_store_ids for the user and build a virtual quota source with quota turned off and assign those all to that quota source and this would all just happen - but I think it might be less performant than this approach. This seems like a good shortcut though with better performance. There are probably some other queries that need to be updated - I think there is one that adjusts the quota for a new dataset and one that redoes all of the quota for a given user and they will need to be in sync I think.