Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How can we load GCS json files to bigquery with deduplication #3592

Open
PratapKumarChowdary opened this issue Nov 27, 2024 · 0 comments
Open

How can we load GCS json files to bigquery with deduplication #3592

PratapKumarChowdary opened this issue Nov 27, 2024 · 0 comments
Labels
api: bigquery Issues related to the googleapis/java-bigquery API.

Comments

@PratapKumarChowdary
Copy link

Hi Team,

I'm looking for possible ways to load our GCS files which were stored in a way month/day/hour to bigquery. And we would like to remove duplicates by scanning the record based on id and timestamp in json file and move the latest record to bigquery. Could you please help the possible ways to implement this.

Thank You
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/java-bigquery API. label Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
api: bigquery Issues related to the googleapis/java-bigquery API.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@PratapKumarChowdary