-
Notifications
You must be signed in to change notification settings - Fork 100
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: basic support for Ipfs-Path-Affinity from IPIP-462 #592
base: main
Are you sure you want to change the base?
Conversation
This is first stab at leveraging these hints withing existing boxo/gateway codebase. It is pretty blunt, but will enable smart clients fetching sub-DAGs to work around any content routing gaps For more info and header semantics see ipfs/specs#462
// Confirm it is a valid content path | ||
affinityPath, err := path.NewPath(headerValue) | ||
if err != nil { | ||
logger.Debugw("skipping invalid Ipfs-Path-Affinity hint", "error", err) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
// Skip duplicated work if immutable affinity hint is a subset of requested immutable contentPath | ||
// (protect against broken clients that use affinity incorrectly) | ||
if !contentPath.Mutable() && !affinityPath.Mutable() && strings.HasPrefix(contentPath.String(), affinityPath.String()) { | ||
logger.Debugw("skipping redundant Ipfs-Path-Affinity hint", "affinity", affinityPath) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
// Process hint in background without blocking response logic for contentPath | ||
go func(contentPath path.Path, affinityPath path.Path, logger *zap.SugaredLogger) { | ||
var immutableAffinityPath path.ImmutablePath | ||
logger.Debugw("async processing of Ipfs-Path-Affinity hint", "affinity", affinityPath) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
if affinityPath.Mutable() { | ||
// Skip work if mutable affinity hint is a subset of mutable contentPath | ||
if contentPath.Mutable() && strings.HasPrefix(contentPath.String(), affinityPath.String()) { | ||
logger.Debugw("skipping redundant Ipfs-Path-Affinity hint", "affinity", affinityPath) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
} | ||
immutableAffinityPath, _, _, err = i.backend.ResolveMutable(r.Context(), affinityPath) | ||
if err != nil { | ||
logger.Debugw("error while resolving mutable Ipfs-Path-Affinity hint", "affinity", affinityPath, "error", err) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
} | ||
immutableAffinityPath, _, _, err = i.backend.ResolveMutable(r.Context(), affinityPath) | ||
if err != nil { | ||
logger.Debugw("error while resolving mutable Ipfs-Path-Affinity hint", "affinity", affinityPath, "error", err) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
// original contentPath was received and returned to HTTP | ||
// client before below get is done, the work is cancelled. | ||
|
||
logger.Debugw("started async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
|
||
logger.Debugw("started async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath) | ||
_, _, err = i.backend.GetBlock(r.Context(), immutableAffinityPath) | ||
logger.Debugw("ended async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath, "error", err) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
|
||
logger.Debugw("started async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath) | ||
_, _, err = i.backend.GetBlock(r.Context(), immutableAffinityPath) | ||
logger.Debugw("ended async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath, "error", err) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
_, _, err = i.backend.GetBlock(r.Context(), immutableAffinityPath) | ||
logger.Debugw("ended async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath, "error", err) | ||
} else { | ||
logger.Debugw("skipping Ipfs-Path-Affinity hint due to data being locally cached", "affinity", affinityPath) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Sensitive data returned by HTTP request headers
Sensitive data returned by HTTP request headers
Codecov ReportAttention: Patch coverage is
@@ Coverage Diff @@
## main #592 +/- ##
==========================================
- Coverage 59.78% 59.69% -0.10%
==========================================
Files 232 232
Lines 28124 28197 +73
==========================================
+ Hits 16815 16833 +18
- Misses 9837 9892 +55
Partials 1472 1472
|
@@ -233,6 +233,8 @@ func (i *handler) getOrHeadHandler(w http.ResponseWriter, r *http.Request) { | |||
return | |||
} | |||
|
|||
i.handlePathAffinityHints(w, r, contentPath, logger) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about only calling this if it's a request for raw
blocks or CAR files as indicated here? Or do you want to extend the scope?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like a good initial version, as it will trigger the search for providers for the affinity path, which are likely the same providers that contain the direct block.
I pushed a quick fix for the gateway sharness tests.
@@ -233,6 +233,8 @@ func (i *handler) getOrHeadHandler(w http.ResponseWriter, r *http.Request) { | |||
return | |||
} | |||
|
|||
i.handlePathAffinityHints(w, r, contentPath, logger) | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As someone not familiar with go nor this codebase, these variable names need work 😳
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
logger.Debugw("async processing of Ipfs-Path-Affinity hint", "affinity", affinityPath) | ||
if affinityPath.Mutable() { | ||
// Skip work if mutable affinity hint is a subset of mutable contentPath | ||
if contentPath.Mutable() && strings.HasPrefix(contentPath.String(), affinityPath.String()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like hasPrefix check could be pulled out into a more semantic function since it's used at least twice.
// These optional hints are mostly useful for trustless block requests. | ||
// See https://github.com/ipfs/specs/pull/462 | ||
func (i *handler) handlePathAffinityHints(w http.ResponseWriter, r *http.Request, contentPath path.Path, logger *zap.SugaredLogger) { | ||
headerName := "Ipfs-Path-Affinity" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
headerName := "Ipfs-Path-Affinity" | |
const headerName = "Ipfs-Path-Affinity" |
if !i.backend.IsCached(r.Context(), immutableAffinityPath) { | ||
// The intention of below code is to asynchronously preconnect | ||
// gateway with providers of the affinityPath in | ||
// Ipfs-Path-Affinity hint. Once connected, these peers can be | ||
// asked directly (via mechanism like bitswap) for blocks | ||
// related to main request for contentPath, and retrieve them, | ||
// even when no other routing system had them announced. If | ||
// original contentPath was received and returned to HTTP | ||
// client before below get is done, the work is cancelled. | ||
|
||
logger.Debugw("started async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath) | ||
_, _, err = i.backend.GetBlock(r.Context(), immutableAffinityPath) | ||
logger.Debugw("ended async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath, "error", err) | ||
} else { | ||
logger.Debugw("skipping Ipfs-Path-Affinity hint due to data being locally cached", "affinity", affinityPath) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor: Instead of if .. else
, check if cached and return if it is.
if !i.backend.IsCached(r.Context(), immutableAffinityPath) { | |
// The intention of below code is to asynchronously preconnect | |
// gateway with providers of the affinityPath in | |
// Ipfs-Path-Affinity hint. Once connected, these peers can be | |
// asked directly (via mechanism like bitswap) for blocks | |
// related to main request for contentPath, and retrieve them, | |
// even when no other routing system had them announced. If | |
// original contentPath was received and returned to HTTP | |
// client before below get is done, the work is cancelled. | |
logger.Debugw("started async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath) | |
_, _, err = i.backend.GetBlock(r.Context(), immutableAffinityPath) | |
logger.Debugw("ended async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath, "error", err) | |
} else { | |
logger.Debugw("skipping Ipfs-Path-Affinity hint due to data being locally cached", "affinity", affinityPath) | |
} | |
if i.backend.IsCached(r.Context(), immutableAffinityPath) { | |
logger.Debugw("skipping Ipfs-Path-Affinity hint due to data being locally cached", "affinity", affinityPath) | |
return | |
} | |
// The intention of below code is to asynchronously preconnect | |
// gateway with providers of the affinityPath in | |
// Ipfs-Path-Affinity hint. Once connected, these peers can be | |
// asked directly (via mechanism like bitswap) for blocks | |
// related to main request for contentPath, and retrieve them, | |
// even when no other routing system had them announced. If | |
// original contentPath was received and returned to HTTP | |
// client before below get is done, the work is cancelled. | |
logger.Debugw("started async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath) | |
_, _, err = i.backend.GetBlock(r.Context(), immutableAffinityPath) | |
logger.Debugw("ended async search for providers of Ipfs-Path-Affinity hint", "affinity", affinityPath, "error", err) |
// Skip duplicated work if immutable affinity hint is a subset of requested immutable contentPath | ||
// (protect against broken clients that use affinity incorrectly) | ||
if !contentPath.Mutable() && !affinityPath.Mutable() && strings.HasPrefix(contentPath.String(), affinityPath.String()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment is a little confusing about which is a prefix of the other.
if immutable affinity hint is a subset of requested immutable contentPath
It could mean that the contentPath should be the prefix, since the subset is a more specific (longer) path:
contentPath = "/top-level"
affinityPath = "/top-level/sub-path"
or it could me that the affinity path is a string subset of the longer content path:
contentPath = "/top-level/stuff/sub-stuff"
affinityPath = "/top-level/stuff"
This PR aims to kick-off discussion how we would support IPIP-462 in
boxo/gateway
.For more info and header semantics, see ipfs/specs#462
proposed scope
What I want to do for now is to add minimal code to start leveraging
Ipfs-Path-Affinity
hints within existingboxo/gateway
codebase, so we cna deploy it to our gateways and allow clients like service-worker-gateway or ipfs-chromium to pass hint and retrieve content even when internal CID was not announced directly.future scope
We may formalize this by extending
IPFSBackend
with explicit place to pass this hint, and then wire it up into routing system of Kubo, but it feels more involved and would like to do that in follow-up, rather than block on it here.