docker compose failing at pip install due to missing dependencies

[dikwickley]

Description

When running docker compose -f docker-compose.yml -f docker-compose.dev.yml up dev-service

=> [dev-service internal] load .dockerignore                                                                                                      0.0s
 => => transferring context: 143B                                                                                                                  0.0s
 => [dev-service internal] load build definition from Dockerfile                                                                                   0.0s
 => => transferring dockerfile: 2.45kB                                                                                                             0.0s
 => [dev-service internal] load metadata for docker.io/library/python:3.9-buster                                                                   0.7s
 => [dev-service  1/13] FROM docker.io/library/python:3.9-buster@sha256:bae5b428ebf32d01a902718b0a58874cbf33d7a4b6a65b7cd7b21d48b0d2e2f1           0.0s
 => [dev-service internal] load build context                                                                                                      0.1s
 => => transferring context: 194.31kB                                                                                                              0.1s
 => CACHED [dev-service  2/13] RUN set -eux;                                                                                                       0.0s
 => CACHED [dev-service  3/13] RUN mkdir -p /etc/apt/keyrings;                                                                                     0.0s
 => CACHED [dev-service  4/13] RUN curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - ;     echo "deb http://apt.postgresql.  0.0s
 => CACHED [dev-service  5/13] RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/node  0.0s
 => CACHED [dev-service  6/13] RUN apt-get update &&     apt-get install -y --no-install-recommends         sudo         ca-certificates           0.0s
 => CACHED [dev-service  7/13] RUN localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8                                    0.0s
 => CACHED [dev-service  8/13] RUN apt-get update &&     apt-get install -y --no-install-recommends         postgresql-15 postgresql-client-15 po  0.0s
 => CACHED [dev-service  9/13] WORKDIR /code/                                                                                                      0.0s
 => CACHED [dev-service 10/13] COPY requirements* ./                                                                                               0.0s
 => ERROR [dev-service 11/13] RUN pip install --no-cache-dir -r requirements-dev.txt --force-reinstall sqlalchemy-filters                          1.6s
------
 > [dev-service 11/13] RUN pip install --no-cache-dir -r requirements-dev.txt --force-reinstall sqlalchemy-filters:
0.826 Collecting sqlalchemy_filters
0.826   Cloning https://github.com/mathesar-foundation/sqlalchemy-filters (to revision models_to_tables) to /tmp/pip-install-461qq5s5/sqlalchemy-filters_d80dc47043234f738a4aa95eaa753ae5
0.828   Running command git clone --filter=blob:none --quiet https://github.com/mathesar-foundation/sqlalchemy-filters /tmp/pip-install-461qq5s5/sqlalchemy-filters_d80dc47043234f738a4aa95eaa753ae5
1.224   fatal: could not read Username for 'https://github.com': No such device or address
1.243   error: subprocess-exited-with-error
1.243   
1.243   × git clone --filter=blob:none --quiet https://github.com/mathesar-foundation/sqlalchemy-filters /tmp/pip-install-461qq5s5/sqlalchemy-filters_d80dc47043234f738a4aa95eaa753ae5 did not run successfully.
1.243   │ exit code: 128
1.243   ╰─> See above for output.
1.243   
1.243   note: This error originates from a subprocess, and is likely not a problem with pip.
1.246 error: subprocess-exited-with-error
1.246 
1.246 × git clone --filter=blob:none --quiet https://github.com/mathesar-foundation/sqlalchemy-filters /tmp/pip-install-461qq5s5/sqlalchemy-filters_d80dc47043234f738a4aa95eaa753ae5 did not run successfully.
1.246 │ exit code: 128
1.246 ╰─> See above for output.
1.246 
1.246 note: This error originates from a subprocess, and is likely not a problem with pip.
1.541 
1.541 [notice] A new release of pip is available: 23.0.1 -> 23.3.1
1.541 [notice] To update, run: pip install --upgrade pip
------
failed to solve: process "/bin/sh -c pip install --no-cache-dir -r ${PYTHON_REQUIREMENTS} --force-reinstall sqlalchemy-filters" did not complete successfully: exit code: 1

It is failing because in the requirements .txt there is :

mathesar/requirements.txt

Line 26 in 08743e6

git+https://github.com/mathesar-foundation/sqlalchemy-filters@models_to_tables#egg=sqlalchemy_filters

and there does not exist any https://github.com/mathesar-foundation/sqlalchemy-filters
(there is though https://github.com/centerofci-archive/sqlalchemy-filters)

This is probably because of recent changes to the org name.

Expected behavior

Dependencies should be resolved with no issue.

To Reproduce

Clone a fresh repo or even try to rebuild the image by
docker compose -f docker-compose.yml -f docker-compose.dev.yml up dev-service --force-recreate --build dev-service

Environment

• OS: macOS 13.3.1;
• Browser: Chrome
• Other info:

Additional context

[dikwickley]

It could be solved in two ways:

1. Fork a version of sqlalchemy-filters in mathesar-foundation from centerofci-archive
2. Point the github repo to centerofci-archive in requirements.txt

[seancolsen]

Thanks for reporting this and performing some preliminary analysis, @dikwickley

@kgodey I've assigned this to you since I figure you would be the best person to make the decisions required to fix this.

I've not yet attempted to reproduce this, but assuming it is reproducible, this looks to be a high-priority issue since it has the potential to affect the workflow of all devs.

[kgodey]

We are no longer using sqlalchemy_filters (that's why it was moved to thecenterofci-archive org), so removing that dependency entirely should fix this. @dikwickley do you want to take a crack at this and confirm that it's fixed?

[dikwickley]

@kgodey sure!

[kgodey]

Great thanks, I've assigned it to you!

[dikwickley]

@kgodey i removed the package from requirement.txt and it installed fine. Next I ran tests for backend and here are few which failed

=========================== short test summary info ============================
FAILED mathesar/tests/api/test_record_api.py::test_record_search - NotImpleme...
FAILED mathesar/tests/api/test_table_api.py::test_create_table_and_normalize_json_data_file
FAILED mathesar/tests/api/test_table_api.py::test_table_patch_columns_two_name_changes
FAILED mathesar/tests/api/test_table_settings_api.py::test_update_table_settings_permission[db_editor_client_factory-200]
FAILED db/tests/records/operations/test_relevance.py::test_rank_and_filter_rows
==== 5 failed, 2328 passed, 10 skipped, 1124 warnings in 210.02s (0:03:30) =====

You did say that we are no longer using sqlalchemy_filters but it is still used here:

mathesar/db/records/operations/relevance.py

Line 2 in 08743e6

from sqlalchemy_filters import apply_sort

Which is ultimately used in

mathesar/db/records/operations/select.py

Lines 21 to 52 in 08743e6

	def get_records(
	table,
	engine,
	limit=None,
	offset=None,
	order_by=None,
	filter=None,
	group_by=None,
	search=None,
	duplicate_only=None,
	fallback_to_default_ordering=False,
	):
	"""
	Returns annotated records from a table.
	Args:
	table: SQLAlchemy table object
	engine: SQLAlchemy engine object
	limit: int, gives number of rows to return
	offset: int, gives number of rows to skip
	order_by: list of dictionaries, where each dictionary has a 'field' and
	'direction' field.
	search: list of dictionaries, where each dictionary has a 'column' and
	'literal' field.
	See: https://github.com/centerofci/sqlalchemy-filters#sort-format
	filter: a dictionary with one key-value pair, where the key is the filter id and
	the value is a list of parameters; supports composition/nesting.
	See: https://github.com/centerofci/sqlalchemy-filters#filters-format
	group_by: group.GroupBy object
	duplicate_only: list of column names; only rows that have duplicates across those rows
	will be returned
	"""

Till we are using the apply_transformations_deprecated i don't think we should get rid of sqlalchemy_filters

[kgodey]

Ugh, I did not know we were still using it in one place, thanks for finding this @dikwickley.

@mathemancer or @Anish9901, is this usage of the package easy to remove? Otherwise, I'll move the sqlalchemy_filters package back to the main organization and that should fix it.

[Anish9901]

The package should be easy to remove @kgodey,
@dikwickley try to rewrite this using sqlalchemy's order_by:

mathesar/db/records/operations/relevance.py

Lines 21 to 24 in cf92053

	filtered_ordered_cte = apply_sort(
	select(rank_cte).where(rank_cte.columns[SCORE_COL] > 0),
	{'field': SCORE_COL, 'direction': 'desc'}
	).cte()