Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK] Add exclusion for integral types check in SchemaUtils.normalizeColumnNamesInDataType #2760

Merged
merged 3 commits into from
Mar 18, 2024
Merged

[SPARK] Add exclusion for integral types check in SchemaUtils.normalizeColumnNamesInDataType #2760

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ae68899
add exclusion for numeric types
ala Mar 18, 2024
793239c
Revert "add exclusion for numeric types"
ala Mar 18, 2024
7eb64e5
add exlusion for integral types
ala Mar 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions spark/src/main/scala/org/apache/spark/sql/delta/schema/SchemaUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,9 @@ def normalizeColumnNamesInDataType(
// When schema evolution adds a new column during MERGE, it can be represented with
// a NullType in the schema of the data written by the MERGE.
sourceDataType
case (_: IntegralType, _: IntegralType) =>
// The integral types can be cast to each other later on.
sourceDataType
case _ =>
if (Utils.isTesting) {
assert(sourceDataType == tableDataType,
Expand Down
15 changes: 14 additions & 1 deletion spark/src/test/scala/org/apache/spark/sql/delta/schema/SchemaUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1548,7 +1548,7 @@ class SchemaUtilsSuite extends QueryTest
.add("b", StringType)
val table = new StructType()
.add("B", StringType)
.add("A", LongType) // LongType != IntType
.add("A", StringType) // StringType != IntegerType
val exception = intercept[AssertionError] {
normalizeColumnNamesInDataType(
deltaLog = null,
Expand All @@ -1560,6 +1560,19 @@ class SchemaUtilsSuite extends QueryTest
assert(exception.getMessage.contains("Types without nesting should match"))
}

test("normalize column names in data type - different integral types") {
val source = new StructType()
.add("a", IntegerType)
.add("b", StringType)
val table = new StructType()
.add("B", StringType)
.add("A", LongType) // LongType != IntegerType
val expected = new StructType()
.add("A", IntegerType)
.add("B", StringType)
checkNormalizedColumnNamesInDataType(source, table, expected)
}

test("normalize column names in data type - nested structs") {
val source = new StructType()
.add("a1", IntegerType)
Expand Down
Loading