safe_requests feature request

[tjs-intel]

The current implementation of safe_requests parses the hostname from the request URL and checks it against a fairly limited set of IP addresses (namely 198.168.1.1 and the AWS instance metadata URL). This should be fairly easy to bypass by registering a domain name (or finding one that already exists) either with an A record that resolves to one of these restricted addresses, or by hosting a web server that returns a redirect to the restricted address.

I propose the following procedure for sandboxing requests.

1. Resolve the host to an IP address, ex. socket.gethostbyname()
2. Check that the IP address falls within a set of public IP address ranges
3. Swap the hostname of the URL with the resolved IP address
4. Hand the request off to requests

There are many edge cases to handle here, ex. tls hostname verification, proxies, redirects, and it may ultimately be easiest to add functionality to requests and/or urllib to support IP address allow/block lists.

[tjs-intel]

@drdavella

[drdavella]

Hi @tjs-intel thanks so much for opening this issue! I agree that ultimately upstreaming this to requests itself might be a good idea. However one of the main motivations for the security project is the fact that many commonly used libraries don't currently enable that kind of hardening.

You've identified an interesting gap here and I think it's worth working through a potential solution. I'm going to ask @matt- to chime in since I think he has a few ideas about how we could accomplish what you're suggesting. One such option might be to hook into a lower-level layer of the requests library and perform validation there.

[tjs-intel]

I tried a quick POC of this concept yesterday by adding an IP address check here in http.client:

https://github.com/python/cpython/blob/b254de71d0d1eb0232ff4abd403a4977ca3f2fe7/Lib/http/client.py#L1010

[matt-]

If we just swap out the host name with the resolved IP I think we may run into some issues with servers that check hostname header, or parse raw URL for some reason.

However I think we could hook into this by making the safe methods use a custom "Transport Adapter"

s = requests.Session()
s.mount('https://github.com/', MyAdapter())

I will have to dig more, but it looks like we could grab access to the "connection" before the request is made which would give us the resolved IP

[tjs-intel]

A relevant post on SO, showing how to resolve a hostname before making the connection: https://stackoverflow.com/a/22614367