iceberg table format support for filesystem destination
#2067
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- mypy upgrade needed to solve this issue: apache/iceberg-python#768 - uses <1.13.0 requirement on mypy because 1.13.0 gives error - new lint errors arising due to version upgrade are simply ignored
…-iceberg-filesystem
✅ Deploy Preview for dlt-hub-docs canceled.
|
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
jorritsandbrink
commented
Nov 16, 2024
jorritsandbrink
commented
Nov 16, 2024
jorritsandbrink
commented
Nov 16, 2024
jorritsandbrink
commented
Nov 16, 2024
|
@rudolfix / @sh-rp this PR isn't there yet but can you give an intermediate review? What still needs to happen:
Notes:
|
rudolfix
reviewed
Nov 21, 2024
dlt/common/libs/pyiceberg.py
Outdated
| """Returns single-table, ephemeral, in-memory Iceberg catalog.""" | ||
|
|
||
| # create in-memory catalog | ||
| catalog = SqlCatalog( |
There was a problem hiding this comment.
NOTE: how we get a catalog should be some kind of plugin. so we can easily plug glue or rest to filesystem
| table_id = f"{DLT_ICEBERG_NAMESPACE}.{table_name}" | ||
| table_path = f"{client.dataset_path}/{table_name}" | ||
| metadata_path = f"{table_path}/metadata" | ||
| if client.fs_client.exists(metadata_path): |
There was a problem hiding this comment.
is it possible to split the function that returns a catalog (even per table so a new catalog is simply empty) and function that does table maintenance (evolves schema or creates empty table)
IMO the code structure would be
with get_catalog() as catalog:
create_or_evolve_table()
write_data()There was a problem hiding this comment.
where catalog is abstract catalog class of pyiceberg
This reverts commit 54cd0bc.
jorritsandbrink
commented
Nov 27, 2024
rudolfix
requested changes
Nov 30, 2024
There was a problem hiding this comment.
code is good and ready to merge. we also have sufficient tests. we still need a little bit better docs.
pyiceberg.io:__init__.py:348 Defaulting to PyArrow FileIOmaybe we could hide this log? we set log levels during tests inpytest_configure- there are some tests that do not work in https://github.com/dlt-hub/dlt/actions/runs/12096164907/job/33730150792?pr=2067#step:8:4106 pls take a look
- we need help with improving the docs. my take would be to create a separate destination Delta / Iceberg where we could move most of the docs out of the filesystem WDYT?
- for both table formats you could write shortly how they are implemented
- in case of iceberg we also should describe how we write tables without catalog and mention the limitations (single writer!)
- we have a page where we enumerate table formats. I'll write some kind of introduction there next week
| @@ -395,6 +422,13 @@ def prepare_load_table(self, table_name: str) -> PreparedTableSchema: | |||
| if table["write_disposition"] == "merge": | |||
| table["write_disposition"] = "append" | |||
| table.pop("table_format", None) | |||
| merge_strategy = resolve_merge_strategy(self.schema.tables, table, self.capabilities) | |||
There was a problem hiding this comment.
OK probably we cannot avoid that. time to add supported write dispositions and a selector to destination capabilities
| elif schema_table.get("table_format") == "iceberg": | ||
| from dlt.common.libs.pyiceberg import _get_last_metadata_file | ||
|
|
||
| self._setup_iceberg(self._conn) |
There was a problem hiding this comment.
isn't the latest version of duckdb iceberg working with it?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #1996