Upgrade to SQLAlchemy 2.0

[chouinar]

Ticket

#82

Changes

Upgraded to SQLAlchemy 2.0, see context section for major changes in this version

Context for reviewers

SQLAlchemy 2.0 comes with an incredibly thorough migration guide: https://docs.sqlalchemy.org/en/20/changelog/migration_20.html

Thankfully 1.4 started nudging usage of SQLAlchemy to use the new approaches, so the changes needed weren't that large.

Noteworthy changes:

• DB model definitions have been adjusted. The python types now need to be wrapped in Mapped[...], and columns now get defined either with mapped_column OR nothing with SQLAlchemy figuring out the type from the python type (similar to Dataclasses or Pydantic). This largely results in more minimal class definitions especially ones with a bunch of basic columns.
• DB session/connection/engine logic adjusted slightly to have fewer edge cases. For the changes here that just required some adjustments to the underlying connection setup, something I'd done on another project already to fix some connection issues. Largely this comes down to connections no longer auto-committing and needing to be in with conn.begin() blocks to function properly.
• Raw SQL calls to execute should be wrapped in the text class
• DeclarativeBase is better supported and instantiated slightly differently. The Metadata object no longer works globally and has to be attached to the DeclarativeBase to work properly.
• Typing for MyPy is built into SQLAlchemy and no longer relies on libraries like SQLAlchemy-stubs

Testing

Since the DB hits virtually everything, I tested everything as thoroughly as possible.

Because of how the SQLAlchemy-stubs we removed work, they need to be completely deleted otherwise MyPy thinks they should run. If you are running outside of the docker container, run poetry install --no-root --all-extras --with dev --sync which will delete extra packages.

Basics

Tests, formatting, linting, all working, only required a few fixes to make SQLAlchemy happy.

SQLAlchemy warnings

When running unit tests, SQLAlchemy will output warnings for deprecated features. The only still usable deprecated feature we had was our getqueries which were adjusted. Prior to the fix, we would see this warning:

Migrations

They still work, adding a few new columns generates what we would expect and uses the mapping config added to the Base class.

Swagger

Was able to successfully use the local swagger endpoints and create/update/read from the DB which was populated like so:

[chouinar]

This isn't used and was giving me some type errors. There's a better way to do this anyways as Alembic has a utility now for checking the model parity for you which I'll create a ticket and add later.

[chouinar]

Batch mode no longer exists: https://docs.sqlalchemy.org/en/20/dialects/postgresql.html#psycopg2-fast-execution-helpers

There's another way to configure it to do a similar operation, but I wasn't sure if we wanted to keep it

[lorenyu]

One request to not access the private member _engine of DBClient, a question about establishing in a new connection in a finally clause, and a few other small suggestions, but otherwise LGTM!

Note for when we generate release notes, we might want to point out that this is a breaking change and provide resources for how to migrate (e.g. link to SQLAlchemy migration docs), similar to how we did it on template-infra (see
https://github.com/navapbc/template-infra/releases/tag/v0.3.0)

[lorenyu]

Looks great thanks!