Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

union should explicitly specify table name in casts #173

Open
wearpants opened this issue Nov 6, 2019 · 7 comments · May be fixed by wearpants/dbt-utils#1 or #864
Open

union should explicitly specify table name in casts #173

wearpants opened this issue Nov 6, 2019 · 7 comments · May be fixed by wearpants/dbt-utils#1 or #864
Labels
bug Something isn't working nested Related to BQ's nested and repeated columns (STRUCT / RECORD)

Comments

@wearpants
Copy link

wearpants commented Nov 6, 2019

Bigquery gets confused by tables that have a column with the same name in CASTs, and will try to cast the entire table instead of the column.

-- causes error, as foo is a struct
select cast(`foo` as STRING) as `foo` 
from `data-playground-1234`.`myschema`.`foo`

-- works when table name is specified
select cast(`foo`.`foo` as STRING) as `foo` 
from `data-playground-1234`.`myschema`.`foo`

I think this is a simple as adding table.name here, like:

{%- set col_name = adapter.quote(table.name) + '.' + adapter.quote(col_name) if col_name in table_columns[table] else 'null' %}

@clausherther
Copy link
Contributor

I would think that's a general problem with BigQuery and would have to be addressed everywhere you cast a column, not just in this macro? The other obvious "fix" is to not have columns with the same name as a table?

@wearpants
Copy link
Author

wearpants commented Nov 6, 2019 via email

@wearpants
Copy link
Author

wearpants commented Nov 6, 2019 via email

wearpants added a commit to wearpants/dbt-utils that referenced this issue Nov 7, 2019
better support for an edge case in Bigquery, which gets confused by casting a column with the same name as its parent table. Fixes dbt-labs#173
@drewbanin
Copy link
Contributor

I said as much in the linked PR, but to follow up on @clausherther's comment above: a good approach to address this issue on BQ might be to alias the table name instead of changing the source table/column names. I do wish that BigQuery was a little bit smarter here.... alas.

So, this works:

with data as (
  select 1 as data
)

select cast(data.data as string) from data;

and this also works:

with data as (
  select 1 as data
)

select cast(data as string) from data as data______dbt_generated______;

but this does not:

with data as (
  select 1 as data
)

select cast(data as string) from data;
--> Invalid cast from STRUCT<data INT64> to STRING at [7:13]

All things considered, I think i like the table alias more that the column qualification. It can be really challenging to get quoting/capitalization exactly right on different databases in different contexts. I think it's straightforward here (nice work on that fix @wearpants) but it other contexts, it may be preferable to alias tables instead.

@github-actions
Copy link

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

@github-actions github-actions bot added the Stale label Aug 27, 2023
@github-actions
Copy link

github-actions bot commented Sep 3, 2023

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 3, 2023
@adrian3ka adrian3ka linked a pull request Jan 23, 2024 that will close this issue
17 tasks
@adrian3ka
Copy link

Hi @drewbanin, I also face this issue in my development with BigQuery, please help me to review the changes if it's make sense to you.
Thanks!

@dbeatty10 dbeatty10 reopened this Apr 18, 2024
@dbeatty10 dbeatty10 added bug Something isn't working and removed Stale labels Apr 18, 2024
@dbeatty10 dbeatty10 added the nested Related to BQ's nested and repeated columns (STRUCT / RECORD) label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working nested Related to BQ's nested and repeated columns (STRUCT / RECORD)
Projects
None yet
5 participants