Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: refactor gateway api to operate on higher level semantics #176

Merged
merged 70 commits into from
Mar 29, 2023

Conversation

aschmahmann
Copy link
Contributor

@aschmahmann aschmahmann commented Feb 16, 2023

Kubo PR that runs sharness tests: ipfs/kubo#9681 (sharness in this PR will fail, but MUST be green in the linked PR)

This is a start at a gateway refactor so that we operate on higher level semantics rather than lower level ones ipfs/kubo#173.

This enables us to have all the gateway HTTP code separate from whether the data was retrieved over Bitswap, via requesting the entire result as a CAR file, etc.

This probably could've been done while still keeping a bunch of the traversal code in the gateway (and collecting metrics like first block times, etc.). However, IMO this seemed unnecessary given that it should be the job of the API implementations to verify the data is correct and the gateway code's job to "HTTP-ify" it and record some higher level metrics.

The result is the a lot of the code has been inverted, since instead of the order being "get a bit -> verify -> get more -> verify..." we have "resolve mutable -> do top level request and get async response -> process async response and emit the correct headers".

Given that the implementations (e.g. of the BlockGateway or the interface-go-ipfs-core one) aren't built yet this is all obviously still in flux, but wanted to put something up in case people think this approach is nuts 😅. For this to all be usable I suspect we'll probably need to have some shared utilities so we don't replicate some of the annoying gateway traversal code multiple times (e.g. handling _redirects).

Also addresses:

Requirements:

- [ ] ipfs/go-unixfs#142 No longer needed.

@@ -17,34 +16,110 @@ type Config struct {
Headers map[string][]string
}

// API defines the minimal set of API services required for a gateway handler.
// ImmutablePath TODO: This isn't great, as its just a signal and mutable paths fulfill the same interface
type ImmutablePath interface {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why interface ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was matching the other path types, doesn't have to be though. This was mostly a placeholder

gateway/gateway.go Outdated Show resolved Hide resolved
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I've run out of time today, will try to do proper read tomorrow, for now dropping partial review with feedback on some TODOs and _redirects (i'd de-emphasize it, it is niche feature, can we move it back to the same place as ipfs-404 thingy? i got gut feeling entire thing will get way less noisy)

gateway/gateway.go Outdated Show resolved Hide resolved
gateway/gateway.go Outdated Show resolved Hide resolved
logger.Debugw("serving ipns record", "path", contentPath)

i.addUserHeaders(w) // ok, _now_ write user's headers.
w.Header().Set("X-Ipfs-Path", contentPath.String()) // TODO: Should we even set this?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for completeness, yes
in practice we could skip these on verifiable responses and nobody would be impacted
(these matter only for implicit "deserialize for me pls" GET done for website hosting)

gateway/handler.go Outdated Show resolved Hide resolved
gateway/handler.go Outdated Show resolved Hide resolved
gateway/handler.go Outdated Show resolved Hide resolved
gateway/handler_defaults.go Outdated Show resolved Hide resolved
gateway/handler_defaults.go Outdated Show resolved Hide resolved
@BigLep BigLep linked an issue Feb 17, 2023 that may be closed by this pull request
…eway's problem. Implemented blocks gateway example
@aschmahmann
Copy link
Contributor Author

Updated: pushed some updates here, still need to rebase on the gateway PRs that have happened in the meanwhile.

There are still some TODOs to be figured out (and I think I found a bug in files package on the way). Two bigger picture questions I had here are:

  1. If/how we want to deal with sentinel errors so we know which HTTP Status Codes to return. Basically as we shuffle work out to the Gateway API implementations they now need to report back more than an opaque error so we know if it's a timeout/couldn't find the data, the data cannot (immutable)/does not (mutable) exist, internal processing errors (blockstore corruption returning the wrong data, etc.), or even "I'm busy go away" 429 errors.
  2. As we shuffle more logic into Gateway API implementations I get more nervous about our testing situation and how much it currently relies on kubo sharness tests.

I'd probably feel better about this if we could get kubo leveraging a blocks gateway implementation that lives here. How unreasonable of an approach is this?

Comment on lines 20 to +32
func NewBytesFile(b []byte) File {
return &ReaderFile{"", NewReaderFile(bytes.NewReader(b)), nil, int64(len(b))}
return &ReaderFile{"", bytesReaderCloser{bytes.NewReader(b)}, nil, int64(len(b))}
}

// TODO: Is this the best way to fix this bug?
// The bug is we want to be an io.ReadSeekCloser, but bytes.NewReader only gives a io.ReadSeeker and io.NopCloser
// effectively removes the io.Seeker ability from the passed in io.Reader
type bytesReaderCloser struct {
*bytes.Reader
}

func (b bytesReaderCloser) Close() error {
return nil
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a bug and a small fix for it. I might be missing something though about why it was done this way.

gateway/gateway.go Outdated Show resolved Hide resolved
}
defer data.Close()
case http.MethodGet:
gwMetadata, data, err = i.api.Get(ctx, imPath)
Copy link
Contributor Author

@aschmahmann aschmahmann Feb 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: handle range requests, probably by copying the golang HTTP library's parser

# Conflicts:
#	gateway/gateway_test.go
#	gateway/handler.go
#	gateway/handler_block.go
#	gateway/handler_codec.go
#	gateway/handler_tar.go
#	gateway/handler_unixfs.go
#	gateway/handler_unixfs__redirects.go
#	gateway/handler_unixfs_dir.go
#	go.mod
#	go.sum
@BigLep
Copy link
Contributor

BigLep commented Feb 28, 2023

2023-02-28 conversation:
This is a precursor for the graph work
Thins for 2023-02-28:
Catchup with other changes happening here.
Get it running
Get it to a place where Henrique can take W/Th or Adin gets back to it Friday

@codecov
Copy link

codecov bot commented Mar 1, 2023

Codecov Report

Merging #176 (360b031) into main (5bbb21b) will decrease coverage by 0.11%.
The diff coverage is 36.28%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     ipfs/kubo#176      +/-   ##
==========================================
- Coverage   48.13%   48.02%   -0.11%     
==========================================
  Files         267      269       +2     
  Lines       32571    32982     +411     
==========================================
+ Hits        15678    15841     +163     
- Misses      15236    15483     +247     
- Partials     1657     1658       +1     
Impacted Files Coverage Δ
examples/gateway/car/main.go 8.00% <0.00%> (ø)
examples/gateway/proxy/main.go 0.00% <0.00%> (ø)
gateway/dns.go 0.00% <0.00%> (ø)
gateway/handler_block.go 0.00% <0.00%> (ø)
gateway/handler_car.go 0.00% <0.00%> (ø)
gateway/handler_codec.go 0.00% <0.00%> (ø)
gateway/handler_ipns_record.go 0.00% <0.00%> (ø)
gateway/handler_tar.go 0.00% <0.00%> (ø)
gateway/handler_defaults.go 31.51% <31.51%> (ø)
gateway/handler_unixfs__redirects.go 37.09% <32.53%> (-2.25%) ⬇️
... and 9 more

... and 14 files with indirect coverage changes

Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gave it another pass, commented inline. (Only this PR, I did not look at sharness yet, I assumed we want the dust to settle first)

Highlights:

  • found some ways we could simplify+unify handling of path traversal errors
  • we may want to introduce ContentType at top level APIs to avoid forcing unnecessary sniffing and block fetches when we have explicit one in root node or CID codec
  • browser on often updated websites may hit an additional RTT around If-None-Match on /ipns/, but not sure if we should worry yet
  • general anxiety about huge refactor ;) 🫂

gateway/gateway.go Outdated Show resolved Hide resolved
gateway/gateway.go Outdated Show resolved Hide resolved
gateway/gateway.go Show resolved Hide resolved
gateway/handler.go Outdated Show resolved Hide resolved
gateway/handler.go Outdated Show resolved Hide resolved
gateway/handler_defaults.go Outdated Show resolved Hide resolved
gateway/handler_unixfs__redirects.go Outdated Show resolved Hide resolved
gateway/handler_unixfs_dir.go Outdated Show resolved Hide resolved
gateway/handler_unixfs_dir.go Outdated Show resolved Hide resolved
gateway/handler_unixfs_dir.go Outdated Show resolved Hide resolved
@hacdias hacdias requested a review from lidel March 21, 2023 10:58
@BigLep
Copy link
Contributor

BigLep commented Mar 21, 2023

2023-03-21 maintainer conversation: this is blocked from being merged until #206 lands.

# Conflicts:
#	examples/gateway/common/blocks.go
#	examples/go.mod
#	examples/go.sum
#	gateway/errors.go
#	gateway/gateway.go
#	gateway/gateway_test.go
#	gateway/handler.go
#	gateway/handler_car.go
#	gateway/handler_codec.go
#	gateway/handler_test.go
#	gateway/handler_unixfs.go
#	gateway/handler_unixfs__redirects.go
#	gateway/handler_unixfs_dir.go
#	go.mod
#	go.sum
@BigLep
Copy link
Contributor

BigLep commented Mar 28, 2023

Blocked on Kubo sharness test passing in ipfs/kubo#9681 (needs rebase), which is blocked on ipfs/kubo#9746 landing.

@hacdias
Copy link
Member

hacdias commented Mar 29, 2023

I rebased ipfs/kubo#9681 and everything is green 💚.

lidel added a commit to filecoin-saturn/caboose that referenced this pull request Mar 29, 2023
lidel added a commit to ipfs-inactive/bifrost-gateway that referenced this pull request Mar 29, 2023
lidel added a commit to ipfs-inactive/bifrost-gateway that referenced this pull request Mar 29, 2023
@aschmahmann aschmahmann merged commit b7d8da1 into main Mar 29, 2023
@hacdias hacdias deleted the feat/gateway-refactor branch March 29, 2023 18:47
Copy link
Member

@lidel lidel Mar 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hacdias @aschmahmann afaik this file is not used anywhere, but increases repo size by +14MB. 🙈
Too late to force-push main to fix it, but can we at least remove it in separate PR so there is no confusing examples/gateway/car-file-gateway/ ?

If we need fixtures for tests, we should always put them in testdata/ directory, which has special meaning in go tooling

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidel urgh, will do! Sorry missed that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hacdias added a commit that referenced this pull request Apr 3, 2023
* refactor gateway api to operate on semantics to closer match user requests
* rename gateway api interface to IPFSBackend
* add blocks gateway implementation to the gateway package
* refactored gateway examples
* add default DNS resolvers
* add default IPLD codecs

---------

Co-authored-by: Henrique Dias <[email protected]>
Co-authored-by: Marcin Rataj <[email protected]>
lidel added a commit that referenced this pull request Apr 5, 2023
removed depecated and unused TTFB histograms.
after #176
and #245
we now have api_call_duration_seconds histograms for higher resolution info
per backend operation
lidel added a commit that referenced this pull request Apr 5, 2023
removed depecated and unused TTFB histograms.
after #176
and #245
we now have api_call_duration_seconds histograms for higher resolution info
per backend operation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Refactor Gateway API so can extract out the request layer
5 participants