QueryBuilder: Implementation of contains Filter Operator applied to JSON Fields for SQLite backend

[rabbull]

This PR introduces the contains filter operator for the SQLite backend, implemented using custom functions rather than SQLAlchemy's high-level abstractions. Comprehensive tests have been included to cover all potential use cases for contains on the SQLite backend, providing a detailed demonstration of its behavior across various scenarios.

The primary goal of this PR is to replicate the functionality of the contains operator from PostgreSQL on SQLite. However, due to the complex and sometimes unexpected behavior of PostgreSQL's containment operator (#6618), a pure SQL implementation would be highly complex and difficult to maintain. To address this, custom functions were chosen, leveraging the flexibility of high-level programming languages (Python, in this case) to handle complex control flows more effectively.

While invoking functions from other languages introduces potential performance limitations, this trade-off is acceptable compared to the alternative of lacking this functionality entirely. Benchmarks will be added to assess the performance impact and ensure it remains within acceptable bounds.

This implementation serves as a proof of concept to evaluate the feasibility of using custom functions for the contains operator. While functional, the current code structure is still preliminary and requires significant cleanup. If this approach proves successful, a subsequent refactor will address technical debt and improve the maintainability of the codebase. This PR is marked as WIP while the approach is evaluated, benchmarks are added, and the necessary refactor is performed.

Closes #6494

[codecov]

Codecov Report

Attention: Patch coverage is 88.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 77.86%. Comparing base (c532b34) to head (7e78b22).

Files with missing lines	Patch %	Lines
src/aiida/storage/sqlite_zip/utils.py	91.67%	2 Missing ⚠️
src/aiida/storage/sqlite_zip/orm.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6619      +/-   ##
==========================================
- Coverage   77.92%   77.86%   -0.05%     
==========================================
  Files         563      563              
  Lines       41671    41695      +24     
==========================================
- Hits        32467    32463       -4     
- Misses       9204     9232      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[GeigerJ2]

Let's maybe shelve this for now, until we have resolved contains for PSQL? Also, just out of curiosity, any reason for going through the custom implementation, rather than through SQLA, apart from the weird behavior for PSQL?

If we agree that the behavior for PSQL is fine, I'd vouch that we also go the SQLA route here. We just need to make sure the behavior is consistent for both backends, as well as document it somewhere.

[rabbull]

any reason for going through the custom implementation, rather than through SQLA, apart from the weird behavior for PSQL?

The main reason for opting for a custom implementation over SQLA is it's too hard to implement such an operator in SQL, as well as in SQLA, due to the insufficient support of recursive operations and effective branching in SQL. Even if a viable solution could be carried out, it is most likely to be unreadable and unmaintainable.

[rabbull]

Hi @GeigerJ2,

I’ve updated the tests for this feature by combining all tests for JSON filters dedicated on the SQLite backend into those on the PostgreSQL backend, and removed requires_psql marks for related test cases. This migration also helped uncover and fix a few bugs in the implementation of JSON filters for the PostgreSQL backend.

As for the placement of the custom function _json_contains, I believe its current position is acceptable for now. Since everything related to sqlite_dos is currently located in the sqlite_zip package, which doesn't seem a proper structure for me and is planned to be refactored in the near future (#6607), a partial refactor doesn't look worthwhile.

Please kindly review the new modification when you are available.

[GeigerJ2]

Hi @rabbull, will review this PR soon. Sorry for the delay!

[rabbull]

Hi @rabbull, will review this PR today. Sorry for the delay!

No worries. Take your time.

I was just trying to figure out why it is blocked by the codecov/patch. It seems including files that I didn't changed. I assumed it was because my branch is out-of-date so that comparing to the main branch a lot files looks modified, but it seems not the case because it is still blocking even if I merged main branch :(

[GeigerJ2]

As always, great work, @rabbull! Let's figure out why codecov fails, and then I'd say we get it merged!