Google Floods Part 3: Adding Inundation Maps; COUNTRY=mozambique

[lowriech]

This adds inundation polygons from the Google Floods API.

https://www.loom.com/share/5b430b093ee546139f5ccbe3299c60c9?sid=049273de-18ff-4b6a-8718-b1a164c82b52

[lowriech]

@gislawill this is updated to call gauges, rather than rely on a previous call passing its attributes

[github-actions]

Build succeeded and deployed at https://prism-1332.surge.sh
(hash 3222328 deployed at 2024-11-23T14:41:42)

[gislawill]

@gislawill - I've not been able to access the flood inundation layer through the surge deployment, but it works locally.

Oh weird, it looks like this https request to the prism-api is getting redirected to a http request by cloudflare the prism-api. I'm not sure why that would be but I'll look into it

[gislawill]

These two files are each about 4 MB — which is unfortunately how large the response from Google is at this endpoint.

[gislawill]

Note: on 11/7/24, I received this error message intermittently but very frequently. Hoping the google floods has resolved it. Or will need to reach out to them

Request failed at url post https://floodforecasting.googleapis.com/v1/floodStatus:searchLatestFloodStatusByArea?key={GOOGLE_FLOODS_API_KEY}: {'regionCode': 'ML'}
Response status code: 502
Response content: (abbreviated)

    <title>Error 502 (Server Error)!!1</title>
    <p>The server encountered a temporary error and could not complete your request.
    <p>Please try again in 30 seconds.  <ins>That's all we know.</ins>

[wadhwamatic]

looks good from my side Will. some surprising discrepancies in Cambodia between NASA SERVIR flood extent and the Google model, but we've also known there to be overestimation by the SERVIR tool.

[gislawill]

The 502 errors from Google Floods have continued and seem to be a result of us hitting the searchLatestFloodStatusByArea endpoint too frequently. I've just emailed the google floods team about this issue.

In the meantime, I've added google floods response caching which gets around the issue for now. I've set the cache timeout to only 10 minutes as we know this data can change quickly.

@ericboucher, I'm curious for your thoughts on this caching (a45de24). Specifically,

• I'm not seeing any cache clearing so far. These new cached files could bloat the EC2 instance
• Would it be helpful to include the cache timeout logic in the caching utilities for easier use elsewhere?

[ericboucher]

Please explicit the inputs here, esp. the cache_key and how it should be used.

[gislawill]

Added some add comments and updated the params to be more explicit

[ericboucher]

You could return get_json_file(cache_filepath) on the last line and remove the step above cache_age = ... (i.e. remove L59)

[gislawill]

Updated

[ericboucher]

@gislawill thanks for this. Yes I think it could be useful to add the cache timeout mechanism for other requests as well.

And for the cache clearing, I think it would be worth implementing but not sure if there is an easy way. We could do something simple like delete any file that's more than 30 days old or something like this. @wadhwamatic any thoughts on that point?

[wadhwamatic]

And for the cache clearing, I think it would be worth implementing but not sure if there is an easy way. We could do something simple like delete any file that's more than 30 days old or something like this. @wadhwamatic any thoughts on that point?

@ericboucher - yes, we should delete older files in this case and 30 days sounds like a good initial period to try