Another case of 403 Errors. #763
Replies: 13 comments 59 replies
-
|
Took a look at the error string and it appears that the IP address you were cycled into was previously used by a bad actor and is triggering security controls. What we have seen is that it typically takes a varying amount of time for the IP address to no longer be seen as abusive. Our only suggestion is to try to obtain a static IP from your ISP. |
Beta Was this translation helpful? Give feedback.
-
|
@hanksims -- I just got an update to my incident ticket showing your comments. Nice! |
Beta Was this translation helpful? Give feedback.
-
|
We have opened a case with Akamai. From what they have found so far is that there was a change that occurred on 8/31 with how the Linode IPs are advertised. Does that line up for everyone as when this all went sideways? |
Beta Was this translation helpful? Give feedback.
-
|
It does for me. My service started reporting errors Saturday (the 31st) in the AM. PST |
Beta Was this translation helpful? Give feedback.
-
|
@jah1007 My ticket with Linode is: Feel free to share that with them so they can cross reference the issue if it looks useful. |
Beta Was this translation helpful? Give feedback.
-
|
Other Linode user here: Since end of August we have the same problem on requests to api.weather.gov from our Linode servers. Thank you for clarification on this problem. For us, access does work from other computers. The suggestion below, using IPV4 works from our Linode servers. |
Beta Was this translation helpful? Give feedback.
-
|
I've been trying to figure out how to connect to/use the NWS API today for my Dakboard (the ones that are built in with the program/service are inaccurate for my location) and keep running into
Is it related? It looks like it but since the reference is different after the 18. I'm not sure. I hope though because otherwise I have no idea what's wrong |
Beta Was this translation helpful? Give feedback.
-
|
Just to be clear, you are providing a lat/lon in that call, correct? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I am. I just didn't want to share my location |
Beta Was this translation helpful? Give feedback.
-
|
I started seeing the same issue a few days ago, from my home Internet connection: |
Beta Was this translation helpful? Give feedback.
-
|
Interesting. So far in this discussion folks have been seeing the issue when using hosting from Akamai/Linode. Are you trying this from code hosted on your home system? |
Beta Was this translation helpful? Give feedback.
-
|
Linode is digging into this on their end. Update: 
|
Beta Was this translation helpful? Give feedback.
-
|
I can confirm that forcing IPv4 worked for me as well. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, wow, thank you! Forcing IPv4 works for me as well, both with curl and in my app. I tried forcing IPv6 earlier, but didn't think it would be using that by default. Wild! |
Beta Was this translation helpful? Give feedback.
-
|
Can confirm. So, Linode... are you advertising us as IP v 6? |
Beta Was this translation helpful? Give feedback.
-
|
@VioletPixel, what language are you using for your app? I'm using python. |
Beta Was this translation helpful? Give feedback.
-
|
I'm using PHP. |
Beta Was this translation helpful? Give feedback.
-
|
The below works for me on windows, even without the used-agent
curl -X GET "https://api.weather.gov/gridpoints/MTR/99,82/forecast" -H "accept: application/geo+json" -H "Feature-Flags: "
I sent the same request to a test server and the curl is adding the header User-Agent: curl/8.8.0 on its own
curl -X GET https://joeperkins.com/test/mgi/index.php -H "accept: application/geo+json" -H "Feature-Flags: "
Host: joeperkins.com
Connection: close
User-Agent: curl/8.8.0
Accept: application/geo+jso
From: John Price ***@***.***>
Sent: Wednesday, September 4, 2024 3:19 PM
To: weather-gov/api ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [weather-gov/api] Another case of 403 Errors. (Discussion #763)
A question from our friends at Linode:
image.png (view on web)<https://github.com/user-attachments/assets/5dee19d7-e90d-4837-8f69-1937eba8f329>
I've tried adding in the header in a call using curl. No luck. Has anyone else made calls using the recommended header with the User-Agent fields?
—
Reply to this email directly, view it on GitHub<#763 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADJHCGZ7N3OLN4OSYC5HUZTZU6BL5AVCNFSM6AAAAABNOHMBJKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJVGEYTKMQ>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
|
Ok. We might be getting closer. Thanks to everyone for the help on this! |
Beta Was this translation helpful? Give feedback.
-
|
I'm seeing the same issue from an IP at DigitalOcean that's been working for months with minimal traffic. Started around the same time & forcing IPv4 also works. Makes me think some configuration has changed on NWS's side that's blocking legitimate traffic, since we have people in the thread having the same issue on Akamai/Linode, home internet, and DigitalOcean. |
Beta Was this translation helpful? Give feedback.
-
|
From what little we have gotten in our ticket, the problem appears to be in how Akamai is classifying IPs. We haven’t made any changes either to our policy or to the origin. I am glad everyone has found a workaround. |
Beta Was this translation helpful? Give feedback.
-
|
@akleinsteiber, I agree multiple hosting services having this problem needs to be run to ground. Why is a work around suddenly needed? While this fix can be made to work for curl, we are using https requests in python that will need testing. What is to say this work around won't work next week? |
Beta Was this translation helpful? Give feedback.
-
|
NOAA ops update and my response. |
Beta Was this translation helpful? Give feedback.
-
|
Oops, is [email protected] associated with NOAA or another user like the rest of us? |
Beta Was this translation helpful? Give feedback.
-
|
In Python, I'm making https:// GET requests using the requests library. You can restrict it to just using IPV4 with this statement: My app is working now, but I'd have a much warmer feeling if I had an understanding of why things broke in the first place.. :/ Thanks to all for the excellent help! Very much obliged. John |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, mine still doesn't work. But then again, I'm not doing the coding directly but using the External data/JSON block on the Dakboard OS/site. I've got it set up to use a custom header that's my email (I replaced with generic in the screenshot below) but I'm still getting the error. Anyone have any advice? I've done coding but never used dakboard before nor have I used APIs before so I'm also learning on the fly, which makes debugging / problem solving difficult. |
Beta Was this translation helpful? Give feedback.
-
|
I set up a DAKBoard account and pointed that query to my own server to see what it does. It does seem to override the User-Agent in the custom query widget. But, it looks like DAKBoard is submitting the query from DigitalOcean (over IPv6 to my server). DigitalOcean is a cesspool of abusive activity, and Akamai may not like that. The default user-agent also claims to be IE 5 which is clearly fictitious. |
Beta Was this translation helpful? Give feedback.
-
|
I will admit that I have no idea what that means in terms of what I'm trying to do. My coding experience is mostly using python for research and data work. I have no experience with constructing custom monitors (like Dakboard), using APIs, or trying to build widgets / use external data -- I've just been googling my way and using trial and error, haha. So, is this just not going to work? I've been trying to make/code a weather widget to use on my Dakboard since the ones they link aren't accurate for my location and the NWS API seemed like the best (and most/only personally affordable) way to do so |
Beta Was this translation helpful? Give feedback.
-
|
Technically speaking, what you're doing should work and not generate the 403 error. But as noted in this thread, it seems the API's caching provider has ratcheted up their protective measures and it doesn't think the request from DAKBoard is legitimate and blocks it. At current there's likely little you can do about it until those measures are adjusted or whatever algorithm is in use settles down and removes the block. That said, if you want the current temperature, that info isn't available on the For example, for the lan/lon you were using, RXFO3 is the nearest station, and |
Beta Was this translation helpful? Give feedback.
-
|
Until this issue is fixed you may want/need to switch to a different weather API entirely, like http://pirateweather.net or similar. I don't think there's anything you can do since you're using a hosted service that doesn't support forcing IPv4, aside from maybe filing a support ticket with the DAKboard folks and asking them if it's something they can do on their end for you. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not a network security expert. If api.weather.gov is blocking certain domains, why would forcing our requests into IPV4 format "solve" the issue? I'm not complaining that there's a work around, but I want to understand it better. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Update from yesterday evening from the NOAA support team: Looks like @dwhitemv25's theory might be correct. |
Beta Was this translation helpful? Give feedback.
-
|
From Linode:
|
Beta Was this translation helpful? Give feedback.
-
|
The only suggestion provided by Akamai amounts to a game of whack-a-mole on our side unfortunately. This has to be fixed at the Akamai level. We are pushing each day to get an update but so far nothing fruitful as of yet. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
For those whose app might be hard to modify, the IPV6 fix by Linode mentioned above might be worth pursuing. |
Beta Was this translation helpful? Give feedback.
-
|
IPv6 is a network number and MAC address (simplification) , and modern operating systems today now randomly use a new MAC address every so often to increase privacy. The only option to block a IPv6 address is to bock the enter network that can be many clients depending on network topology.
IPv6 address - Wikipedia<https://en.wikipedia.org/wiki/IPv6_address>
< Network >< MAC >
2001:0db8:85a3:0000:0000:8a2e:0370:7334
IPv6 transition mechanism - Wikipedia<https://en.wikipedia.org/wiki/IPv6_transition_mechanism>
Also when going from IPv4 to IPv6 or IPv6 to IPv4 through a common NAT many networks can be involved.
From: John Price ***@***.***>
Sent: Friday, September 6, 2024 8:51 AM
To: weather-gov/api ***@***.***>
Cc: Joe Perkins ***@***.***>; Comment ***@***.***>
Subject: Re: [weather-gov/api] Another case of 403 Errors. (Discussion #763)
I'm not a network security expert. If api.weather.gov is blocking certain domains, why would forcing our requests into IPV4 format "solve" the issue? I'm not complaining that there's a work around, but I want to understand it better. Thanks!
—
Reply to this email directly, view it on GitHub<#763 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADJHCG6ZQZAMBG2ZRPG2XQDZVHFODAVCNFSM6AAAAABNOHMBJKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJXGA4TQOA>.
You are receiving this because you commented.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
|
For those still hanging in on this thread, I received a note from the Linode support team:
|
Beta Was this translation helpful? Give feedback.
-
|
Akamai pushed an update last Thursday which looked to resolve the blocking we were seeing. |
Beta Was this translation helpful? Give feedback.
-
|
Is anyone on this thread still being blocked with 403 errors since 8/31? If so, please provide the most recent Reference ID from the 403 error so we can work with our CDN to resolve these issues. |
Beta Was this translation helpful? Give feedback.
-
|
Reference #18.90f4d517.1729085594.4dcf729a |
Beta Was this translation helpful? Give feedback.
-
|
Reference Number: #18.8f643017.1729170606.2c47a804 |
Beta Was this translation helpful? Give feedback.
-
Your IP was blocked for exceeding rate limits. It will auto unblock when rate limits are no longer being exceeded. |
Beta Was this translation helpful? Give feedback.
-
I've read through some of the discussions around receiving 403 errors. But I'm having trouble solving our current problem with them.
Our app hits the API periodically to gather data for a number of locations. The API and app have been playing flawlessly together for months now but this weekend we've been shut down by 403 errors. Hitting the API from the server using curl yields:
<TITLE>Access Denied</TITLE>$ curl -X GET "https://api.weather.gov/gridpoints/MTR/99,82/forecast" -H "accept: application/geo+json" -H "Feature-Flags: "Access Denied
You don't have permission to access "http://api.weather.gov/gridpoints/MTR/99,82/forecast" on this server.
Reference #18.ca7cb17.1725143120.2fec2243
https://errors.edgesuite.net/18.ca7cb17.1725143120.2fec2243
The EdgeSuite link just returns an error. Have we been identified as a bot? Possibly making too many calls in a period of time? Any help would be greatly appreciated.
Beta Was this translation helpful? Give feedback.
All reactions