Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

general fixes to adapt table model #417

Closed
wants to merge 126 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
5848756
[pre-commit.ci] pre-commit autoupdate (#394)
pre-commit-ci[bot] Nov 21, 2023
9d68766
initial tests multi_table design
melonora Nov 23, 2023
d5e3ee3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2023
a47c0cf
add mock new table
melonora Nov 25, 2023
e1c71aa
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Nov 25, 2023
f857d4e
create test class and cleanup
melonora Nov 25, 2023
3c32b27
additional cleanup
melonora Nov 25, 2023
3f47c3b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 25, 2023
bfb4c7c
additional cleanup
melonora Nov 25, 2023
8cc1acf
additional cleanup
melonora Nov 25, 2023
c4de59c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 25, 2023
c6d0976
add pseudo methods
melonora Nov 26, 2023
e8eeda8
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Nov 26, 2023
cd3def6
Change table type in init
melonora Nov 27, 2023
b815980
make tables plural and add to validation in __init__
melonora Nov 27, 2023
3031950
revert to old public accessor
melonora Nov 27, 2023
c149c71
Validate each table in dictionary
melonora Nov 27, 2023
cb826e0
iterate dict values
melonora Nov 27, 2023
da4cc86
add comment
melonora Nov 27, 2023
d6589a0
adjust table getter
melonora Nov 27, 2023
1ad33b0
Add tables getter
melonora Nov 27, 2023
1d1907b
Fix missing parenthesis
melonora Nov 27, 2023
bec136f
change to warnings.warn DeprecationWarning
melonora Nov 27, 2023
22e2bef
allow for backward compatibility in init
melonora Nov 27, 2023
ef0a2dc
[pre-commit.ci] pre-commit autoupdate (#408)
pre-commit-ci[bot] Nov 28, 2023
c75c684
fix dict subscriptable
melonora Nov 28, 2023
9819a09
fix string representation of sdata
melonora Nov 28, 2023
59b5d21
add deprecation decorator for future
melonora Nov 28, 2023
3cdc492
Allow for tables not annotating elements
melonora Nov 28, 2023
2c51b69
switch to using tables with deprecation
melonora Nov 28, 2023
54631ee
fix string representation
melonora Nov 28, 2023
29a37a7
write tables element group
melonora Nov 28, 2023
0c258e4
adjust io to multi_table
melonora Nov 29, 2023
60de077
Alter io to give None as default value for spatialdata attrs keys
melonora Nov 29, 2023
53f9ae5
add tables setter
melonora Dec 2, 2023
b1d451c
raise keyerror table getter
melonora Dec 2, 2023
0f8df0e
remove commented tables setter
melonora Dec 2, 2023
f12d8de
raise keyerror in table deleter
melonora Dec 2, 2023
d7a3092
add deprecation warning
melonora Dec 2, 2023
e94d994
fix tests
melonora Dec 2, 2023
f3d41c7
add DeprecationWarning
melonora Dec 2, 2023
396a970
comment test
melonora Dec 2, 2023
d646ca3
change setter into method
melonora Dec 2, 2023
433a9f3
circumvent mappingproxy set issue
melonora Dec 2, 2023
d3d2083
adjust set get test
melonora Dec 2, 2023
304aa38
add get table keys
melonora Dec 2, 2023
02ae3e5
add column getters
melonora Dec 3, 2023
b550292
add change set target table
melonora Dec 4, 2023
535d917
Give default table name
melonora Dec 5, 2023
5e231ac
fix spatialdata without table
melonora Dec 5, 2023
6a18365
add int32 because of windows and add docstring
melonora Dec 6, 2023
e844067
fix filtering by coordinate system
melonora Dec 6, 2023
9826d35
Change to Path to not be linux / mac specific
melonora Dec 6, 2023
b3eea20
Change to Path to not be linux / mac specific
melonora Dec 6, 2023
f18251f
table should annotate existing element
melonora Dec 6, 2023
88c1579
return table with AnnData having 0 rows
melonora Dec 6, 2023
2a6d675
Adjust for windows
melonora Dec 6, 2023
cfcc8ac
adjust for accessing table elements
melonora Dec 6, 2023
957c4c9
fix change annotation target
melonora Dec 7, 2023
36b1e0c
fix set annotation target
melonora Dec 7, 2023
9f5b8c2
fix/add tests
melonora Dec 7, 2023
0c07957
fix init from elements
melonora Dec 7, 2023
a1c0df2
fix init from elements tests
melonora Dec 7, 2023
230e38e
add validation check
melonora Dec 7, 2023
219d272
add table validation SpatialData.__init
melonora Dec 7, 2023
3051cf0
fix ruff
melonora Dec 7, 2023
1bc1f11
only concatenate if annotating
melonora Dec 7, 2023
3a51b4a
change into warning because of filtering
melonora Dec 7, 2023
8381f29
fix last tests
melonora Dec 7, 2023
dc0d186
adjust to tables
melonora Dec 7, 2023
9c3c483
use tables parameter
melonora Dec 8, 2023
b1d62ef
fix some mypy
melonora Dec 8, 2023
3817b78
some mypy fixes
melonora Dec 9, 2023
76ce40d
some more mypy
melonora Dec 9, 2023
caecb83
fix another mypy
melonora Dec 9, 2023
2dc7a74
circumvent typing error on py3.9
melonora Dec 11, 2023
cde90e2
mypy yet again
melonora Dec 11, 2023
a20cb95
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 11, 2023
854a5ce
fix pre_commit
melonora Dec 11, 2023
81a18bb
down to 12 mypy errors
melonora Dec 12, 2023
f85a093
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Dec 12, 2023
2c3138a
down to 1mypy error
melonora Dec 13, 2023
dd1db66
fixed mypy errors
melonora Dec 13, 2023
9ac3a15
fix set_table_annotation
melonora Dec 13, 2023
444a335
added docstring
melonora Dec 14, 2023
c9e7d99
refactor data loader (#299)
giovp Dec 14, 2023
fd7a8a0
add documentation
melonora Dec 15, 2023
d5d7743
add documentation
melonora Dec 15, 2023
15cd907
Merge branch 'main' into multi_table
melonora Dec 15, 2023
e61e007
minor adjustment docstring
melonora Dec 15, 2023
af1b93f
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Dec 15, 2023
05964c5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 15, 2023
5e16bbd
Added / adjusted docstrings
melonora Dec 15, 2023
432bb1d
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Dec 15, 2023
45b14a2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 15, 2023
67fd7d1
fix mypy after merge
melonora Dec 15, 2023
899d8c3
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Dec 15, 2023
351b7c9
refactor function name
melonora Dec 15, 2023
c620c45
[pre-commit.ci] pre-commit autoupdate (#411)
pre-commit-ci[bot] Dec 17, 2023
898b5ab
small fixes
LucaMarconato Dec 17, 2023
d592606
added gen_elements docstrings
melonora Dec 18, 2023
c479976
tiny comments
LucaMarconato Dec 18, 2023
94ebd70
fix ruff pre-commit
melonora Dec 18, 2023
6beb53b
removed types from docstring
LucaMarconato Dec 18, 2023
a708594
refactor of set_table_annotation_target
melonora Dec 18, 2023
c1d83f3
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Dec 18, 2023
b957906
add quotes
melonora Dec 18, 2023
31fca71
fix (?)
LucaMarconato Dec 18, 2023
e780d6f
refactor error messages
melonora Dec 19, 2023
4eed839
fix incremental update (#329)
giovp Dec 19, 2023
f7cebb5
add concatenate argument
melonora Dec 19, 2023
aca8965
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Dec 19, 2023
aff28c2
Merge branch 'main' into multi_table
melonora Dec 19, 2023
c2c062f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 19, 2023
c3690b7
add util functions to init
melonora Dec 19, 2023
415022c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 19, 2023
207b411
add util functions to init
melonora Dec 19, 2023
29f8ecf
Merge branch 'multi_table' of https://github.com/melonora/spatialdata…
melonora Dec 19, 2023
a5d4d5d
add tables class
melonora Dec 19, 2023
de74482
add table class
melonora Dec 21, 2023
31dc81d
add deprecation back
melonora Dec 21, 2023
d3eef8e
rename function in tests
melonora Dec 21, 2023
f16b357
rename function in tests
melonora Dec 21, 2023
3c80380
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 21, 2023
3a713a8
fix precommit
giovp Dec 21, 2023
798a978
update precommit and remove add_table, store_table and general fixes
giovp Dec 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions src/spatialdata/_core/query/spatial_query.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import dask.array as da
import numpy as np
import pandas as pd
from anndata import AnnData
from dask.dataframe.core import DataFrame as DaskDataFrame
from datatree import DataTree
from geopandas import GeoDataFrame
Expand All @@ -17,6 +18,7 @@
from tqdm import tqdm
from xarray import DataArray

from spatialdata._core._elements import Tables
from spatialdata._core.query._utils import get_bounding_box_corners
from spatialdata._core.spatialdata import SpatialData
from spatialdata._logging import logger
Expand Down Expand Up @@ -266,7 +268,9 @@ def _(
new_elements[element_type] = queried_elements

if filter_table:
tables = {name: _filter_table_by_elements(table, new_elements) for name, table in sdata.tables.items()}
tables: dict[str, AnnData] | Tables = {
name: _filter_table_by_elements(table, new_elements) for name, table in sdata.tables.items()
}
else:
tables = sdata.tables

Expand Down Expand Up @@ -645,7 +649,9 @@ def _polygon_query(

if filter_table:
elements = {"shapes": new_shapes, "points": new_points}
tables = {name: _filter_table_by_elements(table, elements) for name, table in sdata.tables.items()}
tables: dict[str, AnnData] | Tables = {
name: _filter_table_by_elements(table, elements) for name, table in sdata.tables.items()
}
else:
tables = sdata.tables
return SpatialData(shapes=new_shapes, points=new_points, images=new_images, tables=tables)
Expand Down Expand Up @@ -754,7 +760,9 @@ def polygon_query(
geodataframes[k] = vv
if filter_table:
elements = {"shapes": geodataframes}
tables = {name: _filter_table_by_elements(table, elements) for name, table in sdata.tables.items()}
tables: dict[str, AnnData] | Tables = {
name: _filter_table_by_elements(table, elements) for name, table in sdata.tables.items()
}
else:
tables = sdata.tables

Expand Down
128 changes: 17 additions & 111 deletions src/spatialdata/_core/spatialdata.py
Original file line number Diff line number Diff line change
Expand Up @@ -167,9 +167,8 @@ def __init__(
self.points[k] = v

if tables is not None:
self._tables: dict[str, AnnData] = {}

self._add_tables(table_mapping=tables)
for k, v in tables.items():
self.tables[k] = v

self._query = QueryManager(self)

Expand Down Expand Up @@ -672,7 +671,7 @@ def filter_by_coordinate_system(
# TODO: check whether full table dict should be returned or only those which annotate elements. Also check
# filtering with tables having potentially different keys.
if filter_tables:
tables = {}
tables: Tables = Tables(shared_keys=self._shared_keys)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comparing this to #410 there is a difference here. However, I am not certain that shared_keys needs to be passed here since we are returning a new SpatialData object. What do you think @giovp @LucaMarconato ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I agree, the shared keys should be instantiated in the return line 680. I think this can be closed!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After offline conversation with Giovanni, we agreed it is not required.

for table_name, table in self._tables.items():
tables[table_name] = _filter_table_by_coordinate_system(table, element_paths_in_coordinate_system)
else:
Expand Down Expand Up @@ -975,7 +974,7 @@ def write(
assert isinstance(self.path, Path)

@property
def tables(self) -> dict[str, AnnData]:
def tables(self) -> Tables:
"""
Return tables dictionary.

Expand All @@ -987,6 +986,14 @@ def tables(self) -> dict[str, AnnData]:
"""
return self._tables

@tables.setter
def tables(self, shapes: dict[str, GeoDataFrame]) -> None:
"""Set shapes."""
self._shared_keys = self._shared_keys - set(self._tables.keys())
self._tables = Tables(shared_keys=self._shared_keys)
for k, v in shapes.items():
self._tables[k] = v

@property
def table(self) -> None | AnnData:
"""
Expand All @@ -1003,8 +1010,8 @@ def table(self) -> None | AnnData:
stacklevel=2,
)
# Isinstance will still return table if anndata has 0 rows.
if isinstance(self._tables.get("table"), AnnData):
return self._tables["table"]
if isinstance(self.tables.get("table"), AnnData):
return self.tables["table"]
return None

@table.setter
Expand All @@ -1015,10 +1022,9 @@ def table(self, table: AnnData) -> None:
stacklevel=2,
)
TableModel().validate(table)
if self._tables.get("table") is not None:
if self.tables.get("table") is not None:
raise ValueError("The table already exists. Use del sdata.tables['table'] to remove it first.")
self._tables["table"] = table
self._store_tables(table_name="table")
self.tables["table"] = table

@table.deleter
def table(self) -> None:
Expand All @@ -1038,106 +1044,6 @@ def table(self) -> None:
# More informative than the error in the zarr library.
raise KeyError("table with name 'table' not present in the SpatialData object.")

def add_tables(
self,
table: None | AnnData = None,
table_name: None | str = None,
table_mapping: None | dict[str, AnnData] = None,
) -> None:
"""
Add AnnData tables to the SpatialData object.

Parameters
----------
table : None or AnnData, optional
The table to be added. If provided, it is added with the specified table_name.
If not provided, the table_mapping must be provided.

table_name : None or str, optional
The name of the table to be added. This parameter is required if table is provided.

table_mapping : None or dict[str, AnnData], optional
A dictionary mapping table names to tables. If provided, the tables are added
according to the mapping. This parameter is required if table is not provided.
"""
self._add_tables(table=table, table_name=table_name, table_mapping=table_mapping)

def _store_tables(self, table_name: None | str = None, table_mapping: None | dict[str, AnnData] = None) -> None:
"""
Write tables back to the SpatialData Zarr store.

This method is used to store tables in a backed `AnnData` object. If the `AnnData` object is backed, the tables
are stored in a Zarr store, under the "tables" group.
attribute.

Parameters
----------
table_name : None or str, optional
The name of the table to store. If not provided, all tables in `table_mapping` will be stored.
table_mapping : None or dict[str, AnnData], optional
A dictionary mapping table names to `AnnData` tables. If provided, all tables in `table_mapping` will be
stored.

Raises
------
TypeError
If both `table_name` and `table_mapping` are None.
"""
if self.is_backed():
from spatialdata._io.io_table import write_table

store = parse_url(self.path, mode="r+").store
root = zarr.group(store=store)
elem_group = root.require_group(name="tables")
if table_name:
write_table(table=self._tables[table_name], group=elem_group, name=table_name)
elif table_mapping:
for table_name in table_mapping:
write_table(table=self._tables[table_name], group=elem_group, name=table_name)
else:
raise TypeError("Missing arguments, either table_name or table_mapping should be provided.")

def _add_tables(
self,
table: None | AnnData = None,
table_name: None | str = None,
table_mapping: None | dict[str, AnnData] = None,
) -> None:
"""
Validate and add AnnData table(s) to the SpatialData object and save to the SpatialData Zarr store if backed.

Parameters
----------
table : None or AnnData, optional
An optional AnnData table to be added.
table_name : None or str, optional
An optional string representing the name of the table to be added.
table_mapping : None or dict[str, AnnData], optional
An optional dictionary mapping table names to AnnData tables. If specified, multiple tables are added at
once

Raises
------
ValueError
If the specified table already exists in the SpatialData.tables.
ValueError
If table is given but table_name is not specified.
"""
if table:
if table_name:
TableModel().validate(table)
if self._tables.get(table_name) is not None:
raise ValueError("The table already exists. Use del sdata.tables[<table_name>] to remove it first.")
self._tables[table_name] = table
self._store_tables(table_name=table_name)
else:
raise ValueError("Please provide a string value for the parameter table_name.")
elif table_mapping:
for table in table_mapping.values():
self.validate_table_in_spatialdata(table)
self._tables.update(table_mapping)
self._store_tables(table_mapping=table_mapping)

@staticmethod
def read(file_path: Path | str, selection: tuple[str] | None = None) -> SpatialData:
"""
Expand Down Expand Up @@ -1497,7 +1403,7 @@ def __setitem__(self, key: str, value: SpatialElement | AnnData) -> None:
elif schema == ShapesModel:
self.shapes[key] = value
elif schema == TableModel:
self.add_tables(table_name=key, table=value)
self.tables[key] = value
else:
raise TypeError(f"Unknown element type with schema: {schema!r}.")

Expand Down