BigQuery query configuration?

[judoole]

First off, this looks amazing. Great work.

When doing BigQuery, is it possible, per task, to set configuration such as priority, labels, maximum_bytes_billed etc?

[kaxil]

Hey @judoole ,

For loading file from a source to Bigquery, you can pass the configs that are available at https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationLoad

For example:

    aql.load_file(
        input_file=File("gs://astro-sdk/workspace/sample_pattern", conn_id="bigquery", filetype=FileType.CSV),
        output_table=Table(conn_id="bigquery", metadata=Metadata(schema="astro")),
        native_support_kwargs={
            "ignore_unknown_values": True,
            "allow_jagged_rows": True,
            "skip_leading_rows": "1",
        },
        use_native_support=True,
        enable_native_fallback=False,
    )

But I don't think we currently support adding configs on running a query.

[judoole]

I see, and thanks. Exiting project this, all the same. I couldn't quite make it work, and a feature such as impersonate_service_account was crucial. I was inspired and wrote a simple wrapper around the BigQueryInsertJobOperator instead. But it will probably come back and haunt me later.

[kaxil]

:) We could take it as a feature request and add it down the line if that is the only blocker for adopting it