Use the Declarative Net Request API

[LukeChannings]

Hey!

This is a change request to use the Declarative Net Request API using dynamic rules defined in the background script.

Cons:

• All rules require a Kagi token, this means that the user needs to set the token in the app whether or not they use Incognito

Pros:

• The dynamic rules are persisted and run even when the background script is killed
• The dynamic rules do not leak searches
• The Kagi app could potentially set all the redirect rules and allow the user to allow the extension for search engines they want to use it on, which might remove complexity introduced by the search engine selector in app. The only input the user needs is the token.

[vprelovac]

That is amazing, but I am very concerned about requirement for token. Setting up the extension is already hard as it is and this may be over the top.

If it is necessary for this method to work could the extension autoamtically fill it (provided the user is logged into their Kagi account?)

Also what kind of permissions this approach requires?

[LukeChannings]

I've thought about this and experimented a bit.

I've found that:

1. Injecting the token (based on the POC from https://kagifeedback.org/u/metaeaux) is necessary, if the user is already logged into kagi.com the existing cookie appears to be ignored by the redirect. I don't know why this is the case, but it's what I've observed.
2. It is possible to obviate the need for requesting the token from the user by using the Cookie API, essentially the background script could look up the session cookie directly and pass it on. I didn't suggest this because there are already concerns around permissions, and this would mean cookies for any allowed hosts would also be accessible from the background script as collateral damage

[LukeChannings]

It is possible to obviate the need for requesting the token from the user by using the Cookie API, essentially the background script could look up the session cookie directly and pass it on. I didn't suggest this because there are already concerns around permissions, and this would mean cookies for any allowed hosts would also be accessible from the background script as collateral damage

I explored an approach where the popover could temporarily request the cookies permission for kagi.com and pass the kagi_session value to the background script to enable the redirect rules.

This is not possible because Safari does not return HTTPOnly cookies. So even with the cookies permission this won't work.

[LukeChannings]

I am very concerned about requirement for token

What I have discovered (but isn't spelt out in the documentation) is that when you use a Redirect Rule with the Declarative Net Request API it won't use any cookies that would otherwise be in the request if the user had initiated it manually.

For example, consider the two scenarios:

a. A user navigates directly to https://kagi.com/search?q=wat3wat
b. A user navigates to https://duckduckgo.com/search?q=wat3wat, which is then redirected using the DNR API to https://kagi.com/search?q=wat3wat

In the first case the browser looks up the cookies stored for .kagi.com and sends them in the request. But in the second case the browser chooses not to send any cookies. Furthermore, it does not send cookies across redirects. I suspect this is for security reasons, there is maybe a sneaky way to exfiltrate secrets if the cookies are sent.

This ties our hands with regards to needing to know the the token beforehand, and the dance is not so straightforward:

1. We redirect https://duckduckgo.com/search?q=wat3wat to https://kagi.com/search?q=wat3wat&token=<token>
2. Kagi redirects https://kagi.com/search?q=wat3wat&token=<token> to https://kagi.com/search?q=wat3wat
3. We inject the token a second time as a cookie set-cookie kagi_session=<token> for kagi.com

We need to do it this way because if we didn't inject it in (1.) Kagi would redirect to https://kagi.com/signup, and if we didn't inject it in (3.) we would again be redirected to signup. We can only run one action per rule (we can't set the cookie and also redirect), so Kagi's redirect behaviour in (1.) is what enables this dance to happen.

No token injection

Token injected as query parameter only

Token injected as query parameter and cookie

[workwithnano]

@LukeChannings thanks for this PR and the research! I've got a POC in development around using this DNR api that I haven't finished yet, but I've got some learnings I can share:

• First of all, kudos for the thorough research on the cookie-stripping situation with DNR redirects. This took me a while to track down switching back and forth between Xcode, Safari, and Proxyman. Did you ever find official documentation for why DNR redirects do not attach the cookies already associated with the site? I don't see the privacy benefit there, so my hunch is it's a bug in WebKit or Safari. I'll file a bug with them about it to see if I can get an answer.
• Luckily, for non-private browsing, there's a workaround to the issue of DNR redirects not attaching cookies to the new request: redirect to "www.kagi.com" instead of "kagi.com". Kagi will then redirect to "kagi.com" and Safari will attach the cookies at that point. It's weird, but it works.
• To make sure we support all possible search engine domains for a given engine, if using the host_permissions manifest key we'd have to include 261 patterns (e.g. google.co.ve, duckduckgo.pl), which would result in Safari's Settings->Websites->Kagi Search for Safari list showing 261 domains. That is confusing for the end user to deal with, so ideally we avoid that scenario. The desired solution would only show the domains the user has explicitly granted/denied permissions to the extension for showing up in that list. This is possible by using optional_host_permissions, but adds a layer of complexity due to now needing activeTab to figure out the current tab's hostname so we request the correct host permissions for a single domain.
• For private browsing, we still have to get the session key somehow. While one option is to have the user manually paste it in the way we do now, I'm working with the Kagi team on a solution that would automatically fetch the token.
• I've also found that for this relatively simple use case, it's a bit cleaner to just use the regexSubstitution method for all of the search engines. All we have to do is swap the parameter key to the one used by each engine. The regex itself is less self-explanatory than the declarative queryTransform syntax, but it means we don't need to maintain different templates for every search engine.

I'll be sharing my POC within the next 2 days, but I'd like to keep this PR open so we can learn from each other and figure out the best solution for using the DNR api in the Kagi extension.

[LukeChannings]

@workwithnano

I've got a POC in development around using this DNR api that I haven't finished yet

I really didn't expect this PR to get merged since there are outstanding UX and testing problems. I hope it can be helpful for your implementation.

Luckily, for non-private browsing, there's a workaround to the issue of DNR redirects not attaching cookies to the new request: redirect to "www.kagi.com" instead of "kagi.com". Kagi will then redirect to "kagi.com" and Safari will attach the cookies at that point. It's weird, but it works.

This is strange. Does it attach cookies after 3 redirects? I tried variations and wasn't able to get Safari to send cookies with any redirect, I'll try with www. later to verify.

Did you ever find official documentation for why DNR redirects do not attach the cookies already associated with the site? I don't see the privacy benefit there, so my hunch is it's a bug in WebKit or Safari. I'll file a bug with them about it to see if I can get an answer.

Nope. There isn't a support note on MDN like there is for HTTPOnly, but my next step would be to verify the behaviour is unique to Safari and I was too lazy to do that. Could be a bug.

I've also found that for this relatively simple use case, it's a bit cleaner to just use the regexSubstitution method for all of the search engines.

This makes sense if you need to handle all TLDs as well. I needed to use regexSubstitution for Yahoo, since it uses a different search parameter name — I would worry about maintaining a single regex for all queries though, if only because it might make debugging harder. (testMatchOutcome would be useful here, but sadly Safari does not support it).

I'll be sharing my POC within the next 2 days, but I'd like to keep this PR open so we can learn from each other and figure out the best solution for using the DNR api in the Kagi extension

I'll be happy to test your POC when it's ready, no rush. I continue running my PR both on macOS and iOS, I'll update if any issues surface. So far I have had a smooth experience.

[applefreak]

The behavior where the cookies aren't being attached when redirected, I'm guessing has something to do with the SameSite cookie setting? I could be wrong but it was something I've encountered before.

MDN page for the Lax value

kagi_session cookie has SameSite set to Lax (default, according to MDN). So it comes down to whether or not Safari counts DNR redirects as cross-site request.

This would also explain why redirects from kagi.com to www.kagi.com attaches the cookie because it's consider same site.

[LukeChannings]

@applefreak

Why would cookies not be attached when redirecting from kagi.com/search&q=<search>&token=<token> to kagi.com/search&q=<search> if that is the case?

[metaeaux]

Great to see all the discussion and research here!

To make sure we support all possible search engine domains for a given engine, if using the host_permissions manifest key we'd have to include 261 patterns (e.g. google.co.ve, duckduckgo.pl), which would result in Safari's Settings->Websites->Kagi Search for Safari list showing 261 domains. That is confusing for the end user to deal with, so ideally we avoid that scenario. The desired solution would only show the domains the user has explicitly granted/denied permissions to the extension for showing up in that list. This is possible by using optional_host_permissions, but adds a layer of complexity due to now needing activeTab to figure out the current tab's hostname so we request the correct host permissions for a single domain.

@workwithnano you might be able to reduce permissions further by making activeTab an optional permission, and having the popup UI for Kagi provide configuration options, such as automatically detecting the search engine, or manually entering the domain in a text field or a dropdown list e.g. "google.co.ve", and the requesting permissions after the user takes an action. That way you might be able to get permissions down to only 2 host permissions, which is better than the 1.0 permission requirements.

[workwithnano]

I've pushed a PR to #60 with the fewest permissions and fewest chances for bugs I could manage.

@metaeaux making activeTab optional helps, as does making all the hosts optional_host_permissions. The dance of "enable activeTab, set up the redirects for the engine(s) you want, then you can disable activeTab" is annoying but I think it's not so bad. I added scripting as a "setup" permission along with activeTab which lets the user easily get the private session key from Kagi if they want private browsing to use Kagi.

@LukeChannings @applefreak the issue of the token as a query parameter not turning into a Set-Cookie before redirecting to the search query is a head scratcher. Luckily, having the cookie addition rule does the trick.

The remaining issue to resolve is allowing a !bang to redirect to the search engine even if we have a rule for that engine to redirect to Kagi. I'm not sure how to do that yet. Kagi uses the "default" search engine url for bangs, so we can make the assumption that !g means "google.com" and never "google.pl" or some other TLD. Perhaps we'll have to bake in the bangs for the Safari default search engines and create higher-priority rules that redirect directly to those main engine urls while stripping out the bang from the query.

[vprelovac]

The remaining issue to resolve is allowing a !bang to redirect to the search engine even if we have a rule for that engine to redirect to Kagi.

Not a lot of users would have a country specific bang so this should not be too much of an issue. Anyway can we check referrer to see if it is Kagi, while deciding to redirect?