Forecast data is delayed - security blockage?

[blhoward2]

I have a Python script that is accessing the forecast API and right now the data it is retrieving is several day old, while the same link is returning current data in a browser. Is this the result of a rate limit blockage? I do have my email in the user agent header but haven’t been contacted. Is there someone that I can contact to get unblocked?

I’m happy to lower the frequency if that’s the issue.

[KTibow]

Which API? There's multiple ones.

[blhoward2]

The JSON forecast API.

[MatthewFlamm]

I'm seeing the same behavior also for https://api.weather.gov/gridpoints/{wfo}/{x},{y}/forecast/hourly

[blhoward2]

@matthew, it was indeed punishment for fetching too frequently. Changing my UserAgent made it work again temporarily and then they unblocked my original UserAgent after a couple of days. How frequently are you fetching the hourly? I changed mine so forecast is now ~3 hours, hourly is hourly, and observation is 15 minutes.

[MatthewFlamm]

If #85 was addressed, this would be easier. In this [comment] (https://github.com/weather-gov/api/issues/85#issuecomment-660640122), it says we will know if we were blocked since the info returned will be different, but in this case, we are seeing old data.

I will try lengthening the update interval, which is currently something like 15 minutes.

[blhoward2]

I read somewhere that they update the hourly hourly and by 45 minutes after the hour. So I added a scheduler and set mine to update at :50. I haven’t actually logged the update times to check, though.

[scadergit]

The bug returning old data is no way related to being blocked, nor is the UA related to rate limiting. The UA only applies to being allowed to make requests. (Aside, we're working on replacing the UA requirement with API keys.)

Unless you were contacted directly, or you contacted support and they confirmed, you were not blocked more than a few seconds. While I can't provide the limits, that gives you an idea of the interval, and you can even make multiple requests within that internal before getting temporarily blocked for the duration of the interval.

The bug returning old data is discussed in other issues. Please review those for more information.

[blhoward2]

@scadergit, I believe you but what is odd about that is that changing my UA instantly fixed it, and changing the UA back instantly went back to returning old data. Current data was always returned through my browser. I actually did contact Support about this and was told someone would look into it and email me back...but it's been a week. The "bug" lasted nearly a week returning old forecast data. Up until yesterday morning I was receiving the same forecast data generated on 7/12. I've had minor delays of a few hours before, and indeed my script accounts for and smoothes over those for display purposes, but nothing this bad.

[blhoward2]

@scadergit, sorry for multiple replies. I understand from the other comment that you aren't allowed to share the rate limits. Is there some standard of time at which forecasts get updated that you can share? E.g. is it accurate where I read that hourly forecasts are only updated each hour and around :45?

[MatthewFlamm]

I assume the other issue being referenced is #71

[scadergit]

The old date bug is completely random, so correlation is not causation. The CDN has a directive to ignore UA for cache keys...unless for some reason that got disabled...I'll verify this week.

[scadergit]

I assume the other issue being referenced is #71

Yes, that would be the primary thread for this bug.

[blhoward2]

@scadergit, thanks. I did set cache max-age=0 to test and expiring the cache didn't fix anything. If it helps I do log most of the data I receive back so I can probably pull the response from before the UA change (out of date), new UA (fixed), old UA (out of date) if that would help debug.

[scadergit]

@scadergit, sorry for multiple replies.

No worries...

Is there some standard of time at which forecasts get updated that you can share? E.g. is it accurate where I read that hourly forecasts are only updated each hour and around :45?

That is a good question...I should update the documentation with that information for the data provided. Offices vary on how often they submit new forecast data. Some on the hour, others every six hours...all more often in adverse weather or when a mistake is made.

[scadergit]

We're implementing a change to the CDN tomorrow that will unifiy their tracking id with ours. Once that is made, we can review the exact API response that is serving each of the old date responses. It will be called X-Edge-Request-Id. Please provide that once it is made available.

[ktrue]

Is there an additional header we need to add to the request to have the X-Edge-Request-Id returned with the response?

Currently, I have an additional header to the request added (per your previous instruction):

"Pragma: akamai-x-cache-on, akamai-x-get-request-id"

so results now return:

x-request-id: 4e1274bc-3a66-4b34-bff9-89a05635bc53
x-correlation-id: 4e1274bc-3a66-4b34-bff9-89a05635bc53
x-server-id: vm-bldr-nids-apiapp6.ncep.noaa.gov
x-akamai-request-id: 7648da5.44f8f7ef
x-cache: TCP_MISS from a184-26-53-223.deploy.akamaitechnologies.com (AkamaiGHost/10.1.0-29986438) (-)

[scadergit]

I believe we instructed the CDN to always return it, but it might require to be included in the Pragma.

[ktrue]

If it needs to be in the Pragma request, please let us know the exact syntax needed, and I'll add it to my scripts.