Need the new "unreachable" field to be supported by BigQuery::listDatasets API

[danielferraz-git]

We are facing the same problem as issue googleapis/google-api-ruby-client#16987.

We need java API support for the new "unreachable" property in the BigQuery::listDatasets API response.

This is urgent, as this change will take place starting on Feb 15th, as announced by Google.

[PhongChuong]

FYI, we're currently discussing how to approach this internally.
I'll post if there are any updates (ETA Feb [12, 13, 14] 2024).

[sjmisterm]

Thanks @PhongChuong .

Given we're getting an answer (and probably not a release) the earliest by Feb 12th, would it be possible for you to contact the BigQuery team and hold off this change while the community has time to adapt their software? The fact we're the only ones who submitted an issue shows a lot of lack of unawareness about the topic in general and certainly users won't upgrade timely and handle the new API results.

[PhongChuong]

Hi,
Sorry for the delay. We're still in discussion on how to forward the unreachable information to the users in a manageable manner.
I can pass the information to the BigQuery team to hold off this change. However, can you help me understand why that is needed?
In particular, what is concern regarding enabling the feature.

[sjmisterm]

@PhongChuong , sorry for the delay as I was on vacation.

As stated by Google itself in the announcement sent to users:

"Today, in the uncommon case of one or more GCP regions being unreachable for a prolonged amount of time, datasets.list and jobs.list API calls will fail if the response should contain data from the unreachable region.

Starting February 15, 2024, these calls will succeed, with the additional field unreachable added to mark the locations in which data might have been skipped from dataset.list and jobs.list response in case of a prolonged unreachability of the entire region or multi-region.

Under normal circumstances, the field will be empty. However, if you rely on getting a full list of all your datasets and jobs from dataset.list and jobs.list API, you should start interpreting values of the new field if present."

Our use case - and most users, I guess - depends on this call being 100% reliable, i.e., either failing with an exception or returning all the datasets. Now an empty return in a region might mean "temporarily off" or no "datasets at all", which completely break the API.

As suggested by Google itself:

"Check the content of this field to see if there are any unreachable locations. If your API integration with BigQuery depends on getting a full list of datasets/jobs from all regions/multi-regions for which relevant data exists, make sure to adjust it depending on specific circumstances.

You may choose to fail the workflow, issue a warning, or continue without warning in case an unreachable field is populated."

We must fail the workflow given our use case - and I guess that's what most users want to do -, but currently there is no way to find out the empty response is due to an unreachable location and to do so.

[PhongChuong]

@sjmisterm, thanks for the context.
We'll add the unreachable field to this library. I'll update with an ETA soon.

[sjmisterm]

Hi @PhongChuong ,

It's been a while since the last message. Is there an ETA already?

[PhongChuong]

Sorry for the delay. We're still trying to find staffing to work on this issue. I'll update as soon as we have a clear ETA.

[sjmisterm]

Hi @PhongChuong . It's been over 3 months since this change has been deployed and any partial results delivered by the API are wrongly assumed to be the whole data by all Java clients.

I understand you're not the one to blame, but this can potentially cause us a major problem at any time. Does Google really intend to fix this API issue or to revert this backend change?

[sjmisterm]

Hi @PhongChuong . Just pinging you again since it's been over three months from your last message and my previous one was ignored. Any news?

[PhongChuong]

Hi, @sjmisterm ,
Sorry. I should have been more on top of this issue and keep everyone posted.
Unfortunately, my original proposal for Unreachable field fell through. The issue is that listDatasets and listJobs returns Page of dataset, job respectively which requires a longer term solution to how we handle additional fields in a paginated resource. As a result, this change is not exclusive to BigQuery.

I'll bring this issue up again to see if we can raise the priority.

Meanwhile, if you have a support contract with Google, please create this issue in the support console will help us prioritize the effort.