Standardize Write Mode to Merge for All Tables to Ensure Data Integrity and Avoid Duplication

[SouravSaxena3200]

I have noticed that in the consumer database, some tables are written using the append mode, while others use the merge mode. To reduce data duplication and improve data integrity, can we standardize the write mode to merge for all tables? Alternatively, introducing a configuration flag during the run to specify whether all tables should use the default mode or be forced into merge mode would provide greater flexibility and control over the data management process.

[gueniai]

Hi @SouravSaxena3200! Thank you for the suggestion! The tables have been deliberately designed the way they are. Utilizing the correct write mode allows us to create pipelines that are as performant as possible.
These should not lead to any duplication or data integrity issues, have you noticed anything of the sort?