[Spark] Support external DSV2 catalog in RESTORE command #2033
Conversation
gengliangwang
changed the title
| properties: java.util.Map[String, String]): Table = { | ||
| val tablePath = getTablePath(ident.name()) | ||
| // Create an empty Delta table on the tablePath | ||
| spark.range(0).write.format("delta").save(tablePath.toString) |
There was a problem hiding this comment.
Shouldn't this use the passed-in schema? Otherwise it takes a DDL later to update it with info we already had...
There was a problem hiding this comment.
Alternatively, we can just create the DTV2 and call it good -- Delta knows how to handle the new/empty/missing directory case, tho it won't let you read such tables. Which comes back to the first comment -- if the table needs to be readable after this, it needs the correct schema, no?
There was a problem hiding this comment.
Shouldn't this use the passed-in schema?
Since it is a dummy catalog, I try to make the schema fixed as "id: long"
Alternatively, we can just create the DTV2
Could you show me some details of how to use it?
There was a problem hiding this comment.
DeltaTableV2(spark, tablePath.toString) should suffice? See DeltaTableV2.scala
|
FYI I am closing this and continuing my works on #2036 |
Which Delta project/connector is this regarding?
Description
How was this patch tested?
Does this PR introduce any user-facing changes?
Yes, users can use RESTORE command on the tables of their external DSV2 catalogs.