Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make LoadConfig customizable #5

Merged
merged 1 commit into from
Mar 4, 2021
Merged

make LoadConfig customizable #5

merged 1 commit into from
Mar 4, 2021

Conversation

daigotanaka
Copy link
Contributor

Implements #3

@daigotanaka
Copy link
Contributor Author

daigotanaka commented Jan 27, 2021

Currently, this implementation only supports the configs that can be given by booleans, numbers and strings but not special objects.
https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.LoadJobConfig.html

@daigotanaka
Copy link
Contributor Author

example target-config.json:

{
    "project_id": "{{ gcp_project_id }}",
    "dataset_id": "{{ dataset_id }}",
    "stream": false,
    "partition_by": "_sdc_batched_at",
    "numeric_type": "FLOAT64",
    "load_config": {
        "write_disposition": "WRITE_EMPTY"
    }
}

@daigotanaka
Copy link
Contributor Author

Note that if the LoadConfig prams given to the target might cause errors due to the incompatible table settings such as partitions and clustering-fields. target-bigquery-partition does not analyze the params to make adjustments when it creates tables and datasets.

@daigotanaka
Copy link
Contributor Author

Tested with write_disposition use cases and it works as it is intended with the cases of:

  • WRITE_APPEND
  • WRITE_TRUNCATE
  • WRITE_EMPTY

@daigotanaka daigotanaka merged commit b5031f3 into master Mar 4, 2021
@daigotanaka daigotanaka deleted the load-config branch March 4, 2021 21:32
@daigotanaka daigotanaka restored the load-config branch September 16, 2023 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant