[feature] add non-exhausive URI validation

[yozachar]

• Succeeds:
  • feature request: validators.uri() #40,
  • Support for mailto: and tel: protocols ? #50,
  • Magnet validator #68 &
  • ipfs uri validation #195
  • URL Validation: Add support for other schemes apart from https, http, ftp #327

---

Suggestions are welcome:

validators/src/validators/uri.py

Lines 24 to 41 in cdc987d

	@validator
	def uri(value: str, /):
	"""Return whether or not given value is a valid URI.
	Examples:
	>>> uri('mailto:[email protected]')
	# Output: True
	>>> uri('file:path.txt')
	# Output: ValidationError(func=uri, ...)
	Args:
	value:
	URI to validate.
	Returns:
	(Literal[True]): If `value` is a valid URI.
	(ValidationError): If `value` is an invalid URI.
	"""

[tofetpuzo]

are you taking any new suggestions?

[yozachar]

Constructive ones, please go ahead.

[tofetpuzo]

--- # Define a list of valid URI schemes => VALID_SCHEMES = ['http', 'https', 'ftp', 'mailto', 'file']

1. We could make the parameter-> value(str) flexible perhaps not restrict it to a type str, could be src/path/to/file; which could contain different urls
2. do a recursive search(regex expression perhaps) to match pattern

[tofetpuzo]

is it something we should do?

[yozachar]

Something similar yeah.

[tofetpuzo]

can I take up the challenge :)?

[yozachar]

Sure, go ahead.

[tofetpuzo]

Hey @yozachar ,can we use socket.inet_aton(ip) in python to validate ip addresses as a function then update the schema in uri.py?

   # URL-based schemes
    if any(
        value.startswith(item + "://") for item in {
            "ftp", "ftps", "git", "http", "https",
            "irc", "rtmp", "rtmps", "rtsp", "sftp",
            "ssh", "telnet", "gopher", "ldap", "sip",
            "nfs", "mqtt", "smb", "udp"
        }
    ):

[yozachar]

IP addresses are already validated in ip_address.py. Why use socket.inet_aton?

[tofetpuzo]

Hi, that is cool I looked at the message thread, something about this error below , was why I thought this feature was asked for I guess, see code below.

import validators

streamurl = "rtmp://192.168.1.123:1935/live/test"
print(validators.url(streamurl))

Output:
ValidationError(func=url, args={'value': 'rtmp://192.168.1.123:1935/live/test'})

Or does it mean we just want to validate other schemas? like the ones few ones below, that I could not find in uri.py

proposed schemas

"Uri":  {
"udp": udp://192.168.1.1:1234/path;param=value?,

"gopher": gopher://gopher.example.com/1/path;type=1?search#frag,\
"ldap": ldap://ldap.example.com/cn=John%20Doe,dc=example,dc=com;scope=one?sn#frag, \
"sip":sip://user:[email protected]/path;transport=tcp?subject=Hello#frag, \
"smb": smb://fileserver.example.com/share/path;param=value?query=1#fragment\
 }

Few Thoughts

1. Have a regex expression : that complies the schema above.
2. uri_regrex: re.compile()
   e.g

  # Validate using regex for URL-based schemes
    if uri_regex.match(value):
        return True

[yozachar]

Currently, url.py validates a string which is in the form of scheme:// + userinfo (optional) + host + port (optional) + path (optional) + query (optional) + fragment (optional). Any other protocol that adheres to this form can added here:

validators/src/validators/url.py

Lines 48 to 52 in cd01b4b

	in {
	"ftp", "ftps", "git", "http", "https",
	"irc", "rtmp", "rtmps", "rtsp", "sftp",
	"ssh", "telnet",
	}

and here:

validators/src/validators/uri.py

Lines 50 to 54 in cd01b4b

	value.startswith(item) for item in {
	"ftp", "ftps", "git", "http", "https",
	"irc", "rtmp", "rtmps", "rtsp", "sftp",
	"ssh", "telnet",
	}

Say, the incoming string is ldap://[2001:db8::7]/c=GB?objectClass?one. This is similar to http. But does ldap support userinfo, ports and fragments? If yes, you can simply put it along with the URL validation scheme list, as shown above. If no, there are two way to go about it.

1. Modify the url.py to conditionally validate components of the URL based upon the schema. So if the schema is ldap, skip userinfo, ports & fragment check.
2. The other way, would be more relevant to a URIs like magnet, which does not conform to the classic http URI format (e.g. magnet:?xt=urn:btih:0cd42065571809961e9661db2123a87d77c33964&dn=archlinux-2024.07.01-x86_64.iso). Here you can choose how to validated it. If it can validated it via a reasonably-long regex, you can put it within uri.py, if you need more elaborate approach, feel free to spin up a specific module level validator, if you need external dependency, add them as optional dependencies (see: feat: add validator for eth addresses #383).

[tofetpuzo]

Currently, url.py validates a string which is in the form of scheme:// + userinfo (optional) + host + port (optional) + path (optional) + query (optional) + fragment (optional). Any other protocol that adheres to this form can added here:

validators/src/validators/url.py

Lines 48 to 52 in cd01b4b

	in {
	"ftp", "ftps", "git", "http", "https",
	"irc", "rtmp", "rtmps", "rtsp", "sftp",
	"ssh", "telnet",
	}

and here:

validators/src/validators/uri.py

Lines 50 to 54 in cd01b4b

	value.startswith(item) for item in {
	"ftp", "ftps", "git", "http", "https",
	"irc", "rtmp", "rtmps", "rtsp", "sftp",
	"ssh", "telnet",
	}

Say, the incoming string is ldap://[2001:db8::7]/c=GB?objectClass?one. This is similar to http. But does ldap support userinfo, ports and fragments? If yes, you can simply put it along with the URL validation scheme list, as shown above. If no, there are two way to go about it.

1. Modify the url.py to conditionally validate components of the URL based upon the schema. So if the schema is ldap, skip userinfo, ports & fragment check.
2. The other way, would be more relevant to a URIs like magnet, which does not conform to the classic http URI format (e.g. magnet:?xt=urn:btih:0cd42065571809961e9661db2123a87d77c33964&dn=archlinux-2024.07.01-x86_64.iso). Here you can choose how to validated it. If it can validated it via a reasonably-long regex, you can put it within uri.py, if you need more elaborate approach, feel free to spin up a specific module level validator, if you need external dependency, add them as optional dependencies (see: feat: add validator for eth addresses #383).

This is very clear, I will work on it. Thank you 👍

[e3krisztian]

Hi, I would like to use the validators library, to filter possible url-s.

The urls include include ones with schemes ldap, ws, wss, and probably more currently rejected ones.
It looks like work on extending the list of url schemes has not progressed, and reading through the discussion it looks like a significant task to add validation of new schemes properly, so that is not the path I would go.

I think most users are happy with the default set of schemes, for the rest of us, a way to relax the scheme check/work around the supported scheme list would be a big help.

At the moment I can think of the below solutions:

1. Exposing _validate_netloc and _validate_optionals (as url_netloc and url_optionals) would allow a reimplementation of url for schemes in application code, that would never make into the library
2. Relaxing scheme check url(..., skip_scheme_check=True)
3. User supplied scheme list url(..., schemes: frozenset[str] = frozenset({"http", ...}) which is then passed to _validate_scheme

Would one or more of the above be an acceptable change to the library API?
Is there some other, preferred solution, that would work around the fixed list of supported schemes?

[yozachar]

Apologies for the late reply.

Thank you for the inputs!

1. Exposing _validate_netloc and _validate_optionals (as url_netloc and url_optionals) would allow a reimplementation of url for schemes in application code, that would never make into the library

Currently, the url(...) validator is tightly coupled with these supporting functions. I assume exposing those functions would require further modifications..

2. Relaxing scheme check url(..., skip_scheme_check=True)

This function already has many parameters..

3. User supplied scheme list url(..., schemes: frozenset[str] = frozenset({"http", ...}) which is then passed to _validate_scheme

Please read #396 (comment)

Would one or more of the above be an acceptable change to the library API?

Yes, if the changes do not stiffen the codebase.

Is there some other, preferred solution, that would work around the fixed list of supported schemes?

I am thinking about some class based validations to make this more flexible, but ideas are much welcome.

[e3krisztian]

No worries, I had another idea, and implemented it in

• feat: allow custom URL scheme validation #409

It can be optionally further enhanced with providing alternative validate_scheme implementations, like publicly exposing the default one, and maybe some of these:

def accept_all_schemes(scheme):
    return True

def accept_iana_schemes(scheme):
    # https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml
    # I do not intend to, but someone might want to implement this as well
    ...

def accept_schemes(*accepted_schemes):
    def validate(scheme):
        return scheme in accepted_schemes
    return validate

which could be used like

validators.url(value, validate_scheme=accept_all_schemes)
validators.url(value, validate_scheme=accept_schemes("http", "https"))