[BUG][Spark] INSERT INTO struct evolution in map/arrays breaks when a column is renamed #3227

johanl-db · 2024-06-06T14:01:07Z

Bug

Describe the problem

Schema evolution in INSERT doesn't always work properly when the new column is added to a struct nested within an array or map. If another column is renamed, the operation fails when it should succeed.

Steps to reproduce

For example with a map, in python, renaming column key to renamed_key and added a field comment in the a struct inside the map:

(
  spark.createDataFrame([[1, {"event": (1, 1)}]], "key int, metrics map<string, struct<id: int, value: int>>")
    .write
    .option("overwriteSchema", "true")
    .format("delta")
    .saveAsTable("insert_map_schema_evolution")
)
(
  spark.createDataFrame([[1, {"event": (1, 1, "deprecated")}]], "renamed_key int, metrics map<string, struct<id: int, value: int, comment: string>>")
    .write
    .mode("append")
    .option("mergeSchema", "true")
    .format("delta")
    .insertInto("insert_map_schema_evolution")
)
[[DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION](https://docs.databricks.com/error-messages/error-classes.html#datatype_mismatch.cast_without_suggestion)] Cannot resolve "metrics" due to data type mismatch: cannot cast "MAP<STRING, STRUCT<id: INT, value: INT, comment: STRING>>" to "MAP<STRING, STRUCT<id: INT, value: INT>>".

Note that the struct inside the map isn't evolved to add the new field. Without the unrelated column being renamed, this works well:

(
  spark.createDataFrame([[1, {"event": (1, 1)}]], "key int, metrics map<string, struct<id: int, value: int>>")
    .write
    .option("overwriteSchema", "true")
    .format("delta")
    .saveAsTable("insert_map_schema_evolution")
)
(
  spark.createDataFrame([[1, {"event": (1, 1, "deprecated")}]], "key int, metrics map<string, struct<id: int, value: int, comment: string>>")
    .write
    .mode("append")
    .option("mergeSchema", "true")
    .format("delta")
    .insertInto("johan_lasperas.playground.insert_renamed_array_map")
)

Observed results

The operation fails.

Expected results

The operation succeeds, the table schema is changed to key int, metrics map<string, struct<id: int, value: int, comment: string>> and the data is inserted.

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

Yes. I can contribute a fix for this bug independently.
Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
No. I cannot contribute a bug fix at this time.

The text was updated successfully, but these errors were encountered:

Richard-code-gig · 2024-09-14T08:48:13Z

Hello @johanl-db,
Could you please assign this issue to me? I am confident I can help resolve it and would appreciate any guidance as needed.

johanl-db · 2024-09-16T07:35:02Z

@Richard-code-gig thanks for volunteering for this.
I would start by adding a test in e.g. deltaInsertIntoSuite to cover the problematic case and work from there.

I believe this only impacts INSERT by position:

SQL: INSERT INTO <table> VALUES (...) or `INSERT INTO SELECT ...
Scala/Python: df.write.insertInto()

INSERT by name (INSERT INTO <table> (columns) VALUES (...) / df.write.save() / df.write.saveAsTable()) is probably not affected - or better said, it doesn't really properly support schema enforcement / evolution already so supporting this specific case is irrelevant for now.

Once you have a test, head over to DeltaAnalysis.resolveQueryColumnsByOrdinal where the logic aligning schema for INSERT by position is. I believe the issue is that addCastToColumn doesn't properly support schema evolution in maps one way or another

Richard-code-gig · 2024-09-16T11:14:33Z

Okay thanks. I am picking it up now.

Pshak-20000 · 2024-10-21T08:26:42Z

I hope this message finds you well. I have reviewed the issue regarding schema evolution in Delta Lake, particularly the problem that arises when renaming a column while trying to insert new data into a table with nested structures.

I believe I can contribute a fix for this bug independently. Before I get started, I wanted to check if there are any specific guidelines or preferred practices for contributing to the Delta Lake codebase that I should be aware of.

I am excited about the opportunity to contribute to the project and help improve the functionality.

johanl-db · 2024-10-21T15:39:53Z

@Pshak-20000 You can take a look at https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md for general information about contributions to this project.

You may want to start by checking with @Richard-code-gig to see if he's still working on this issue or not.
The typical process is then to open a pull request against this repository with a proposed fix + tests, I'm happy to provide some guidance and support with code reviews

Richard-code-gig · 2024-10-21T17:05:06Z

Hello @johanl-db, @Pshak-20000 I am working on it and almost done. Johanl I need your help on something minor to complete the assignment, please let me know how I can reach you. Thanks

Pshak-20000 · 2024-10-23T05:08:13Z

You can reach me here or through my email ([email protected]) if that’s more convenient. Let me know what you need help with!!!. Looking forward to hearing from you.

johanl-db · 2024-10-23T10:09:21Z

Hello @johanl-db, @Pshak-20000 I am working on it and almost done. Johanl I need your help on something minor to complete the assignment, please let me know how I can reach you. Thanks

You can join the Delta slack workspace and reach me there: https://go.delta.io/slack
Or you can open a draft PR and we can have a discussion on the PR directly.

Richard-code-gig · 2024-11-13T08:55:27Z

Hi,
Just wanted to give an update!
I’ve successfully managed to get it working and am now in the process of writing the test cases and ensure backward compatibilities in line with the project guidelines. I expect to submit a PR sometime this week.
I initially thought it would be a simpler task, but it's taken a bit more effort than I anticipated.

johanl-db · 2024-11-14T10:40:32Z

@Richard-code-gig If you want some support/feedback from my side, you can open a PR even if it's not entirely finalized yet and we can discuss there

…mn renaming Resolved the issue described in [Bug delta-io#3227](delta-io#3227) where adding a field inside a struct (nested within a map) while renaming a top column caused the operation to fail. The fix focuses on handling schema changes without affecting the integrity of existing data structures, specifically avoiding issues with nested fields within a map and renamed columns. Signed-off-by: Sola Richard Olorunfemi <[email protected]>

…mn renaming Resolved the issue described in [Bug delta-io#3227](delta-io#3227) where adding a field inside a struct (nested within a map) while renaming a top column caused the operation to fail. The fix focuses on handling schema changes without affecting the integrity of existing data structures, specifically avoiding issues with nested fields within a map and renamed columns. Signed-off-by: Sola Richard Olorunfemi <[email protected]> fix!:renamed the added DeltaWriteExample to EvolutionWithMap

Richard-code-gig · 2024-11-18T05:39:50Z

@Richard-code-gig If you want some support/feedback from my side, you can open a PR even if it's not entirely finalized yet and we can discuss there

Hi Johanl-db,
I have submitted a PR

Richard-code-gig · 2024-11-18T07:59:14Z

Hi @johanl-db,
The PR is now successful. The 2 failing checks and the 1 dependent canceled check are caused by an unrelated issue, as observed in this commit and waiting to be addressed in this PR.

…mn renaming Resolved the issue described in [Bug delta-io#3227](delta-io#3227) where adding a field inside a struct (nested within a map) while renaming a top column caused the operation to fail. The fix focuses on handling schema changes without affecting the integrity of existing data structures, specifically avoiding issues with nested fields within a map and renamed columns. fix!:renamed the added DeltaWriteExample to EvolutionW ithMap fix!: Modified TypeWideningInsertSchemaEvolutionSuite to accommodate that schema evolution is now allowed for maps Signed-off-by: Sola Richard Olorunfemi <[email protected]>

…mn renaming Resolved the issue described in [Bug delta-io#3227](delta-io#3227) where adding a field inside a struct (nested within a map) while renaming a top column caused the operation to fail. The fix focuses on handling schema changes without affecting the integrity of existing data structures, specifically avoiding issues with nested fields within a map and renamed columns. fix!:renamed the added DeltaWriteExample to EvolutionW ithMap fix!: Modified TypeWideningInsertSchemaEvolutionSuite to accommodate that schema evolution is now allowed for maps Signed-off-by: Sola Richard Olorunfemi <[email protected]> fix!: addCastToMap to handle complex types. Added tests to cover new abilities

…mn renaming Resolved the issue described in [Bug delta-io#3227](delta-io#3227) where adding a field inside a struct (nested within a map) while renaming a top column caused the operation to fail. The fix focuses on handling schema changes without affecting the integrity of existing data structures, specifically avoiding issues with nested fields within a map and renamed columns. fix!:renamed the added DeltaWriteExample to EvolutionW ithMap fix!: Modified TypeWideningInsertSchemaEvolutionSuite to accommodate that schema evolution is now allowed for maps Signed-off-by: Sola Richard Olorunfemi <[email protected]> fix!: addCastToMap to handle complex types. Added tests to cover new abilities fix: resolved scalaStyle error

…mn renaming Resolved the issue described in [Bug delta-io#3227](delta-io#3227) where adding a field inside a struct (nested within a map) while renaming a top column caused the operation to fail. The fix focuses on handling schema changes without affecting the integrity of existing data structures, specifically avoiding issues with nested fields within a map and renamed columns. fix!:renamed the added DeltaWriteExample to EvolutionW ithMap fix!: Modified TypeWideningInsertSchemaEvolutionSuite to accommodate that schema evolution is now allowed for maps Signed-off-by: Sola Richard Olorunfemi <[email protected]> fix!: addCastToMap to handle complex types. Added tests to cover new abilities fix: resolved scalaStyle error fix: yet another scalaStyle issue fix!:made some schema evolution for maps tests in DeltaInsertIntoTableSuite more flexible fix: DeltaAnalysis

johanl-db added bug Something isn't working good first issue Good for newcomers good medium issue Good for those with Delta Lake experience and removed good first issue Good for newcomers labels Jun 6, 2024

johanl-db assigned Richard-code-gig Sep 16, 2024

Richard-code-gig linked a pull request Nov 16, 2024 that will close this issue

[Spark] Fix schema evolution issue with nested struct (within a map) and column renamed #3886

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG][Spark] INSERT INTO struct evolution in map/arrays breaks when a column is renamed #3227

[BUG][Spark] INSERT INTO struct evolution in map/arrays breaks when a column is renamed #3227

johanl-db commented Jun 6, 2024

Richard-code-gig commented Sep 14, 2024

johanl-db commented Sep 16, 2024

Richard-code-gig commented Sep 16, 2024

Pshak-20000 commented Oct 21, 2024

johanl-db commented Oct 21, 2024

Richard-code-gig commented Oct 21, 2024

Pshak-20000 commented Oct 23, 2024

johanl-db commented Oct 23, 2024

Richard-code-gig commented Nov 13, 2024

johanl-db commented Nov 14, 2024

Richard-code-gig commented Nov 18, 2024

Richard-code-gig commented Nov 18, 2024 •

edited

Loading

[BUG][Spark] INSERT INTO struct evolution in map/arrays breaks when a column is renamed #3227

[BUG][Spark] INSERT INTO struct evolution in map/arrays breaks when a column is renamed #3227

Comments

johanl-db commented Jun 6, 2024

Bug

Describe the problem

Steps to reproduce

Observed results

Expected results

Willingness to contribute

Richard-code-gig commented Sep 14, 2024

johanl-db commented Sep 16, 2024

Richard-code-gig commented Sep 16, 2024

Pshak-20000 commented Oct 21, 2024

johanl-db commented Oct 21, 2024

Richard-code-gig commented Oct 21, 2024

Pshak-20000 commented Oct 23, 2024

johanl-db commented Oct 23, 2024

Richard-code-gig commented Nov 13, 2024

johanl-db commented Nov 14, 2024

Richard-code-gig commented Nov 18, 2024

Richard-code-gig commented Nov 18, 2024 • edited Loading

Richard-code-gig commented Nov 18, 2024 •

edited

Loading