Use Anti-Captcha for Re-Captcha bypassing.

[pluja]

Recently the instance yotter.xyz has been 'blocked' by Google requiring it to solve a ReCaptcha in order to continue.

To do this we will use Anti-Captcha, just like Invidious and many other projects do.

[pluja]

It is already all done. Only thing that is left to do is to POST to Google to get the cookie and being able to use it in future requests, which is a thing that I don't really know how to do yet. This is what I'm doing right now, but Youtube don't gives me the cookie back.

requests.post("https://youtube.com/das_captcha", data=inputs,
                                                 headers={"Content-Type": "application/x-www-form-urlencoded",
                                                          "Accept-Language": "en-US,en;q=0.5",
                                                          "Referer": "https://www.youtube.com/das_captcha",
                                                          "Origin": "https://www.youtube.com"})

The content of data is all the input fields from the Captcha form and their values. Just like invidious does here.

I need to do a POST request with the captcha solution to Youtube. But I can't seem to figure out how to correctly do the request in order to get the cookie from Youtube. Invidious does this here

The response I am getting from google after this POST is the following:

{
	'Content-Encoding': 'gzip',
	'Cache-Control': 'no-cache',
	'X-Frame-Options': 'SAMEORIGIN',
	'X-Content-Type-Options': 'nosniff',
	'Content-Type': 'text/html; charset=utf-8',
	'Strict-Transport-Security': 'max-age=31536000',
	'P3P': 'CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."',
	'Expires': 'Tue, 27 Apr 1971 19:44:06 GMT',
	'Date': 'Sun, 11 Oct 2020 12:52:24 GMT',
	'Server': 'YouTube Frontend Proxy',
	'X-XSS-Protection': '0',
	'Set-Cookie': 'GPS=1; path=/; domain=.youtube.com; expires=Sun, 11-Oct-2020 13:22:24 GMT, YSC=besKXywhXVg; path=/; domain=.youtube.com; secure; httponly; samesite=None, VISITOR_INFO1_LIVE=ekfHv5jbTdI; path=/; domain=.youtube.com; secure; expires=Fri, 09-Apr-2021 12:52:24 GMT; httponly; samesite=None',
	'Alt-Svc': 'h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"',
	'Transfer-Encoding': 'chunked'
}

As you can see, the Set-Cookie field does NOT contain a cookie. Or, if it does it doesn't seem to me like it does. On the following lines there's a correct Captcha request that I have intercepted using OWASP ZAP tool:

POST https://www.youtube.com/das_captcha HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Referer: https://www.youtube.com/das_captcha
Content-Type: application/x-www-form-urlencoded
Content-Length: 822
Origin: https://www.youtube.com
Connection: keep-alive
Cookie: CONSENT=YES+DE.de+V14+BX; VISITOR_INFO1_LIVE=grulLEghNPs; goojf=05b708f2b789bd960a45bf2ae5f701dcc1oAAABBR1dmUWEyUnRZdWlMNXU0alhuQzJLRXJDX01FaXpTREpZS1g0RzVlNmRUck5IVWozN1BiSnVmOU1GNlMtM0hBemtrSmJPbzhsZDVwMDlnQVlHWU1MTEMyZUE=; GPS=1; YSC=N1eucO9hiC8
Upgrade-Insecure-Requests: 1
Host: www.youtube.com

Here you can see that there's a cookie goojf and a CONSENT field.

Finally, this is the response that google fives when you solve a captcha sending it by a post request to /das_captcha

Content-Length: 0
Location: https://www.youtube.com/
Expires: Tue, 27 Apr 1971 19:44:06 GMT
Content-Type: text/html; charset=utf-8
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
Date: Sun, 11 Oct 2020 11:30:59 GMT
Server: YouTube Frontend Proxy
X-XSS-Protection: 0
Set-Cookie: goojf=38991dcee3cfa8f25f28789b81f4cc51c1oAAABBR1dmUWEwT3JrYWltRDhCMFFXcmdkeXk4X1NvNkZIMnR5WXFIdTBRdVkwNldXaG9OWjc0UC1QTUp6aEZNQk5zQ3U1WUE5Z05OTUcxMHFfcjhKQi1zS2lDdFE=; path=/; domain=.youtube.com; expires=Sun, 11-Oct-2020 15:30:59 GMT
Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

[FireMasterK]

I need to do a POST request with the captcha solution to Youtube. But I can't seem to figure out how to correctly do the request in order to get the cookie from Youtube. Invidious does this here

All invidious does is take the form, put it into a variable by parsing the keys and values, solve the captcha with anti-captcha and then override the g-recaptcha-response variable with what you get.

NewPipe only stores cookies with the names s_gl and goojf as seen here:
https://github.com/TeamNewPipe/NewPipe/blob/ae33c6cf182d749ee640b02bde303f83b239b680/app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java#L227

[FireMasterK]

Another thing we might need to handle is https://github.com/iv-org/invidious/blob/812a21bce62e9e94340bd622734483c1cf9399fc/src/invidious/helpers/jobs.cr#L265-L318
which is another type of captcha (the one with the plain white screen), but that is currently not required.

[FireMasterK]

Also, any reason why we don't use a generic User-Agent such as Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0? It would be difficult to identify Yotter users that way. I got that User-Agent from privacy.resistFingerprinting

[pluja]

Yeah, that's a good idea. I will change the user agent. I will take a look at how NewPipe does that. But I'm not being able to get the goojf cookie.

[pluja]

@FireMasterK This is the response I'm getting from anti-captcha:

{'gRecaptchaResponse': '03AGdBq27OUBQ7M53tXAFTKmcKfM7GLs6QT9AXAyDksIobG_VB5g3LjI7SjFMz15CMaJKMg0_XBNyT8p5WqG2qc3UaWp-FcHuoPqOW2gXPVvzzkhhxYptoYOYYYvJ46a5CcJTqEmNh0br-m7PHSvNwhn_7dusOEnKDnpDY4gXNS-HmTFW13ETyR3M9Wos4oCYO5P7ZRj9dehK8CeeqKzKlDSvIzptSRTUIgL4wSP8An-TWKdbAeH4B5i9qbODuIRbScr9c8xaXzw92o_CSrRW9ZWhHX9SAe7sFSLSP1SR_CNlFh8CIE0hSbFYSUucfnxQIYfyLDG321xFlHmViF2XPF4gS5jCxlOzXy34cPcLMEmLV_HNZpaYzLsMRfxqB1mnZ5C7gcimBw0jctWzKXBYWKMonMmkWvLuHI76u7azqsYEXCm7UgkSZKz48oaASnorBgNNz-E0jvtkolbSjhBmIZ8yGht_2T-F6zPeRl6x40W0DMTO-5agAXui19P0j5f6RgRAOh3HgevbV'}
{'Cache-Control': 'no-cache', 'Content-Type': 'text/html; charset=utf-8', 'X-Frame-Options': 'SAMEORIGIN', 'Expires': 'Tue, 27 Apr 1971 19:44:06 GMT', 'Content-Encoding': 'gzip', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000', 'Date': 'Mon, 12 Oct 2020 10:21:12 GMT', 'Server': 'YouTube Frontend Proxy', 'X-XSS-Protection': '0', 'Alt-Svc': 'h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'}

I'm not getting any cookie. Here's the code I am using: https://github.com/ytorg/Yotter/blob/yotter-dev/youtube/util.py#L102-L157

[pluja]

Here is the Anti-captcha documentation: https://anticaptcha.atlassian.net/wiki/spaces/API/pages/9666606/NoCaptchaTaskProxyless+Google+Recaptcha+puzzle+solving+without+proxies

and an example using PHP:
https://github.com/AdminAnticaptcha/solving-captcha-concepts/blob/master/google-search-engine.php

[FireMasterK]

@FireMasterK This is the response I'm getting from anti-captcha:

{'gRecaptchaResponse': '03AGdBq27OUBQ7M53tXAFTKmcKfM7GLs6QT9AXAyDksIobG_VB5g3LjI7SjFMz15CMaJKMg0_XBNyT8p5WqG2qc3UaWp-FcHuoPqOW2gXPVvzzkhhxYptoYOYYYvJ46a5CcJTqEmNh0br-m7PHSvNwhn_7dusOEnKDnpDY4gXNS-HmTFW13ETyR3M9Wos4oCYO5P7ZRj9dehK8CeeqKzKlDSvIzptSRTUIgL4wSP8An-TWKdbAeH4B5i9qbODuIRbScr9c8xaXzw92o_CSrRW9ZWhHX9SAe7sFSLSP1SR_CNlFh8CIE0hSbFYSUucfnxQIYfyLDG321xFlHmViF2XPF4gS5jCxlOzXy34cPcLMEmLV_HNZpaYzLsMRfxqB1mnZ5C7gcimBw0jctWzKXBYWKMonMmkWvLuHI76u7azqsYEXCm7UgkSZKz48oaASnorBgNNz-E0jvtkolbSjhBmIZ8yGht_2T-F6zPeRl6x40W0DMTO-5agAXui19P0j5f6RgRAOh3HgevbV'}
{'Cache-Control': 'no-cache', 'Content-Type': 'text/html; charset=utf-8', 'X-Frame-Options': 'SAMEORIGIN', 'Expires': 'Tue, 27 Apr 1971 19:44:06 GMT', 'Content-Encoding': 'gzip', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000', 'Date': 'Mon, 12 Oct 2020 10:21:12 GMT', 'Server': 'YouTube Frontend Proxy', 'X-XSS-Protection': '0', 'Alt-Svc': 'h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'}

I'm not getting any cookie. Here's the code I am using: https://github.com/ytorg/Yotter/blob/yotter-dev/youtube/util.py#L102-L157

You won't get any cookie from anti-captcha, you'll get it from /das_captcha, you can notice this in your earlier comment too.

Finally, this is the response that google fives when you solve a captcha sending it by a post request to /das_captcha

Content-Length: 0
Location: https://www.youtube.com/
Expires: Tue, 27 Apr 1971 19:44:06 GMT
Content-Type: text/html; charset=utf-8
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
Date: Sun, 11 Oct 2020 11:30:59 GMT
Server: YouTube Frontend Proxy
X-XSS-Protection: 0
Set-Cookie: goojf=38991dcee3cfa8f25f28789b81f4cc51c1oAAABBR1dmUWEwT3JrYWltRDhCMFFXcmdkeXk4X1NvNkZIMnR5WXFIdTBRdVkwNldXaG9OWjc0UC1QTUp6aEZNQk5zQ3U1WUE5Z05OTUcxMHFfcjhKQi1zS2lDdFE=; path=/; domain=.youtube.com; expires=Sun, 11-Oct-2020 15:30:59 GMT
Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

Look at the Set-Cookie header here. Set-Cookie: goojf=389...

[pluja]

Yes... But this is from a request I made solving the captcha myself and using ZAP to intercept it...

I should be getting the cookie from anti-captcha, as they do send the cookie from google sites.

[FireMasterK]

That's not true from what I can see, you can't get the goojf cookie without posting the form to youtube.

[pluja]

https://github.com/iv-org/invidious/blob/812a21bce62e9e94340bd622734483c1cf9399fc/src/invidious/helpers/jobs.cr#L261

Take a look here at what does Invidious do.

[FireMasterK]

Ah, I see what you mean, isn't it in the solution JSONObject? Are you saying that it's omitted? 🤔

[pluja]

Yes, that's not there. I contacted the support and they said that the cookie is only returned for *.google.com domains.... So I don't see how Invidious is getting that cookie.

[FireMasterK]

Interesting, does it work without those cookies or does youtube error out?

[pluja]

I get a response, but I don't think it will let me bypass the captcha using that response. I will try....