diff --git a/.gitignore b/.gitignore index 655f91c9f..cceea01b4 100644 --- a/.gitignore +++ b/.gitignore @@ -142,3 +142,4 @@ dist # reports reports +report.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 874dce5de..6a67d7d93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.6.0] - 2024-05-27 +### Changed +- The `--gateway-url` now defaults to `http://127.0.0.1:8080` to ensure no confusion with subdomain gateway feature. +- The `--subdomain-url` now defaults to `http://localhost:8080`, making it more friendly for local development. + - We also simplified the way `--subdomain-url` works. We no longer run implicit tests + against `http://localhost` in addition to the URL passed via + `--subdomain-url`. To test more than one domain, run test multiple times. +- DNSLink fixtures no longer depend on `--subdomain-url` and use unrelated `*.example.org` domains instead. + ## [0.5.2] - 2024-05-20 ### Changed - Fixed: relaxed dag-cbor error check ([#205](https://github.com/ipfs/gateway-conformance/pull/205)) diff --git a/Makefile b/Makefile index 4654f491c..af7293b45 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ test-cargateway: provision-cargateway fixtures.car gateway-conformance test-kubo-subdomains: provision-kubo gateway-conformance ./kubo-config.example.sh - ./gateway-conformance test --json reports/output.json --gateway-url http://127.0.0.1:8080 --subdomain-url http://example.com:8080 + ./gateway-conformance test --json reports/output.json --gateway-url http://127.0.0.1:8080 --subdomain-url http://localhost:8080 test-kubo: provision-kubo gateway-conformance ./gateway-conformance test --json reports/output.json --gateway-url http://127.0.0.1:8080 --specs -subdomain-gateway diff --git a/cmd/gateway-conformance/main.go b/cmd/gateway-conformance/main.go index 994ea8455..cdc91988b 100644 --- a/cmd/gateway-conformance/main.go +++ b/cmd/gateway-conformance/main.go @@ -90,16 +90,18 @@ func main() { Flags: []cli.Flag{ &cli.StringFlag{ Name: "gateway-url", + EnvVars: []string{"GATEWAY_URL"}, Aliases: []string{"url", "g"}, Usage: "The URL of the IPFS Gateway implementation to be tested.", - Value: "http://localhost:8080", + Value: "http://127.0.0.1:8080", Destination: &gatewayURL, }, &cli.StringFlag{ - Name: "subdomain-url", - Usage: "The Subdomain URL of the IPFS Gateway implementation to be tested.", - Value: "http://example.com", - Destination: &subdomainGatewayURL, + Name: "subdomain-url", + Aliases: []string{"sg"}, + EnvVars: []string{"SUBDOMAIN_GATEWAY_URL"}, + Usage: "URL of the HTTP Host that should be used when testing https://specs.ipfs.tech/http-gateways/subdomain-gateway/ functionality", + Value: "http://localhost:8080", // TODO: ideally, make these empty by default, and opt-in }, &cli.StringFlag{ Name: "json-output", diff --git a/docs/commands.md b/docs/commands.md index 58f60c555..3ba794570 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -30,8 +30,8 @@ The `test` command is the main command of the tool. It is used to test a given I | Input | Availability | Description | Default | |---|---|---|---| -| gateway-url | Both | The URL of the IPFS Gateway implementation to be tested. | http://localhost:8080 | -| subdomain-url | Both | The URL to be used in Subdomain feature tests based on Host HTTP header. | http://example.com | +| gateway-url | Both | The URL of the IPFS Gateway implementation to be tested. | http://127.0.0.1:8080 | +| subdomain-url | Both | The URL to be used in Subdomain feature tests based on Host HTTP header. | http://localhost:8080 | | json | Both | The path where the JSON test report should be generated. | `./report.json` | | xml | GitHub Action | The path where the JUnit XML test report should be generated. | `./report.xml` | | html | GitHub Action | The path where the one-page HTML test report should be generated. | `./report.html` | @@ -61,8 +61,8 @@ A few examples: | Use Case | gateway-url | subdomain-url | |----------|-------------|---------------| -| CI & Dev | http://127.0.0.1:8080 | http://example.com | -| Production | https://dweb.link | https://dweb.link | +| CI & Dev | `http://127.0.0.1:8080` | `http://localhost:8080` | +| Production | `https://ipfs.io` | `https://dweb.link` | #### Usage @@ -70,9 +70,10 @@ A few examples: ```yaml - name: Run gateway-conformance tests - uses: ipfs/gateway-conformance/.github/actions/test@v1 + uses: ipfs/gateway-conformance/.github/actions/test@v6 # TODO make sure to use latest with: - gateway-url: http://localhost:8080 + gateway-url: http://127.0.0.1:8080 + subdomain-url: http://localhost:8080 specs: +subdomain-gateway,-path-gateway json: report.json xml: report.xml @@ -84,7 +85,7 @@ A few examples: ##### Docker ```bash -docker run --network host -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance test --gateway-url http://localhost:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 30m +docker run --network host -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance test --gateway-url http://127.0.0.1:8080 --subdomain-url http://localhost:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 30m ``` ### extract-fixtures diff --git a/fixtures/dir_listing/dnslink.yml b/fixtures/dir_listing/dnslink.yml index 7e8be2607..1d9f51829 100644 --- a/fixtures/dir_listing/dnslink.yml +++ b/fixtures/dir_listing/dnslink.yml @@ -1,5 +1,5 @@ # yaml-language-server: $schema=../fixture.schema.json dnslinks: - website: - subdomain: website + dir-listing-website: + domain: dnslink-dir-listing.example.org path: /ipfs/bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i # ./rootDir/ diff --git a/fixtures/redirects_file/dnslink.yml b/fixtures/redirects_file/dnslink.yml index f5fb2cde9..69c5f2cc7 100644 --- a/fixtures/redirects_file/dnslink.yml +++ b/fixtures/redirects_file/dnslink.yml @@ -1,10 +1,10 @@ # yaml-language-server: $schema=fixture.schema.json dnslinks: custom-dnslink: - subdomain: dnslink-enabled-on-fqdn + domain: dnslink-enabled-on-fqdn.example.org # cid of ./redirects.car:/examples/ path: /ipfs/QmYBhLYDwVFvxos9h8CGU2ibaY66QNgv8hpfewxaQrPiZj dnslink-spa: - subdomain: dnslink-enabled-with-spa + domain: dnslink-enabled-with-spa.example.org # cid of ./redirects-spa.car path: /ipfs/bafybeib5lboymwd6p2eo4qb2lkueaine577flvsjjeuevmp2nlio72xv5q diff --git a/fixtures/subdomain_gateway/dnslink.yml b/fixtures/subdomain_gateway/dnslink.yml index 54c793b72..5e165dd0e 100644 --- a/fixtures/subdomain_gateway/dnslink.yml +++ b/fixtures/subdomain_gateway/dnslink.yml @@ -1,10 +1,10 @@ # yaml-language-server: $schema=../fixture.schema.json dnslinks: wikipedia: - domain: dnslink-subdomain-gw-test.example.org + domain: en.wikipedia-on-ipfs.example.org # Wikipedia CID path: /ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze test: - domain: dnslink-test.example.com + domain: dnslink-test.example.org # CIDv1=$(echo "hello" | ipfs add --cid-version 1 -Q) - path: /ipfs/bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am \ No newline at end of file + path: /ipfs/bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am diff --git a/tests/dnslink_gateway_test.go b/tests/dnslink_gateway_test.go index 110ec21c3..15c8614cd 100644 --- a/tests/dnslink_gateway_test.go +++ b/tests/dnslink_gateway_test.go @@ -20,25 +20,28 @@ func TestDNSLinkGatewayUnixFSDirectoryListing(t *testing.T) { fixture := car.MustOpenUnixfsCar("dir_listing/fixtures.car") file := fixture.MustGetNode("ą", "ę", "file-źł.txt") + // DNSLink domain and fixture we will be using for Host headerthis test + dnsLinkHostname := "dnslink-website.example.org" dnsLinks := dnslink.MustOpenDNSLink("dir_listing/dnslink.yml") - dnsLink := dnsLinks.MustGet("website") - - gatewayURL := SubdomainGatewayURL + dnsLink := dnsLinks.MustGet("dir-listing-website") tests := SugarTests{} - u, err := url.Parse(gatewayURL) + // Sent requests to endpoint defined by --gateway-url + u, err := url.Parse(GatewayURL) if err != nil { t.Fatal(err) } - dnsLinkHostname := tmpl.Fmt("{{dnslink}}.{{host}}", dnsLink, u.Host) + // Host header should use dnslink domain with the same scheme as --gateway-url + hostHdr := tmpl.Fmt("{{scheme}}://{{dnslink}}", u.Scheme, dnsLink) tests = append(tests, SugarTests{ { Name: "Backlink on root CID should be hidden (TODO: cleanup Kubo-specifics)", Request: Request(). - URL(`{{scheme}}://{{hostname}}/`, u.Scheme, dnsLinkHostname), + Header("Host", hostHdr). + URL(GatewayURL), Response: Expect(). Body( And( @@ -50,7 +53,8 @@ func TestDNSLinkGatewayUnixFSDirectoryListing(t *testing.T) { { Name: "Redirect dir listing to URL with trailing slash", Request: Request(). - URL(`{{scheme}}://{{hostname}}/ą/ę`, u.Scheme, dnsLinkHostname), + Header("Host", hostHdr). + URL(GatewayURL + "/ą/ę"), Response: Expect(). Status(301). Headers( @@ -60,7 +64,8 @@ func TestDNSLinkGatewayUnixFSDirectoryListing(t *testing.T) { { Name: "Regular dir listing (TODO: cleanup Kubo-specifics)", Request: Request(). - URL(`{{scheme}}://{{hostname}}/ą/ę/`, u.Scheme, dnsLinkHostname), + Header("Host", hostHdr). + URL(GatewayURL + "/ą/ę"), Response: Expect(). Headers( Header("Etag").Contains(`"DirIndex-`), diff --git a/tests/redirects_file_test.go b/tests/redirects_file_test.go index c9f7eca3e..20dca0048 100644 --- a/tests/redirects_file_test.go +++ b/tests/redirects_file_test.go @@ -26,214 +26,208 @@ func TestRedirectsFileSupport(t *testing.T) { tests := SugarTests{} - // We're going to run the same test against multiple gateways (localhost, and a subdomain gateway) - gatewayURLs := []string{ - SubdomainGatewayURL, - SubdomainLocalhostGatewayURL, + // Redirects require origin isolation (https://specs.ipfs.tech/http-gateways/web-redirects-file/) + // This means we only run these tests against origins explicitly passed via --subdomain-url + u, err := url.Parse(SubdomainGatewayURL) + if err != nil { + t.Fatal(err) } - for _, gatewayURL := range gatewayURLs { - u, err := url.Parse(gatewayURL) - if err != nil { - t.Fatal(err) - } + redirectDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, redirectDirCID, u.Host) - redirectDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, redirectDirCID, u.Host) - - tests = append(tests, SugarTests{ - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/redirect-one redirects with default of 301, per _redirects file", - Request: Request(). - Header("Host", u.Host). - URL("{{url}}/redirect-one", redirectDirBaseURL), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("/one.html"), - ), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/301-redirect-one redirects with 301, per _redirects file", - Request: Request(). - URL("{{url}}/301-redirect-one", redirectDirBaseURL), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("/one.html"), - ), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/302-redirect-two redirects with 302, per _redirects file", - Request: Request(). - URL("{{url}}/302-redirect-two", redirectDirBaseURL), - Response: Expect(). - Status(302). - Headers( - Header("Location").Equals("/two.html"), - ), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/200-index returns 200, per _redirects file", - Request: Request(). - URL("{{url}}/200-index", redirectDirBaseURL), - Response: Expect(). - Status(200). - Body(Contains("my index")), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/posts/:year/:month/:day/:title redirects with 301 and placeholders, per _redirects file", - Request: Request(). - URL("{{url}}/posts/2022/01/01/hello-world", redirectDirBaseURL), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("/articles/2022/01/01/hello-world"), - ), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/splat/one.html redirects with 301 and splat placeholder, per _redirects file", - Request: Request(). - URL("{{url}}/splat/one.html", redirectDirBaseURL), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("/redirected-splat/one.html"), - ), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/not-found/has-no-redirects-entry returns custom 404, per _redirects file", - Request: Request(). - URL("{{url}}/not-found/has-no-redirects-entry", redirectDirBaseURL), - Response: Expect(). - Status(404). - Headers( - Header("Cache-Control").Equals("public, max-age=29030400, immutable"), - Header("Etag").Equals(`"{{etag}}"`, custom404.Cid().String()), - ). - Body(Contains(custom404.ReadFile())), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/gone/has-no-redirects-entry returns custom 410, per _redirects file", - Request: Request(). - URL("{{url}}/gone/has-no-redirects-entry", redirectDirBaseURL), - Response: Expect(). - Status(410). - Headers( - Header("Cache-Control").Equals("public, max-age=29030400, immutable"), - Header("Etag").Equals(`"{{etag}}"`, custom410.Cid().String()), - ). - Body(Contains(custom410.ReadFile())), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/unavail/has-no-redirects-entry returns custom 451, per _redirects file", - Request: Request(). - URL("{{url}}/unavail/has-no-redirects-entry", redirectDirBaseURL), - Response: Expect(). - Status(451). - Headers( - Header("Cache-Control").Equals("public, max-age=29030400, immutable"), - Header("Etag").Equals(`"{{etag}}"`, custom451.Cid().String()), - ). - Body(Contains(custom451.ReadFile())), - }, - { - Name: "request for $REDIRECTS_DIR_HOSTNAME/catch-all returns 200, per _redirects file", - Request: Request(). - URL("{{url}}/catch-all", redirectDirBaseURL), - Response: Expect(). - Status(200). - Body(Contains("my index")), - }, - }...) + tests = append(tests, SugarTests{ + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/redirect-one redirects with default of 301, per _redirects file", + Request: Request(). + Header("Host", u.Host). + URL("{{url}}/redirect-one", redirectDirBaseURL), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("/one.html"), + ), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/301-redirect-one redirects with 301, per _redirects file", + Request: Request(). + URL("{{url}}/301-redirect-one", redirectDirBaseURL), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("/one.html"), + ), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/302-redirect-two redirects with 302, per _redirects file", + Request: Request(). + URL("{{url}}/302-redirect-two", redirectDirBaseURL), + Response: Expect(). + Status(302). + Headers( + Header("Location").Equals("/two.html"), + ), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/200-index returns 200, per _redirects file", + Request: Request(). + URL("{{url}}/200-index", redirectDirBaseURL), + Response: Expect(). + Status(200). + Body(Contains("my index")), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/posts/:year/:month/:day/:title redirects with 301 and placeholders, per _redirects file", + Request: Request(). + URL("{{url}}/posts/2022/01/01/hello-world", redirectDirBaseURL), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("/articles/2022/01/01/hello-world"), + ), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/splat/one.html redirects with 301 and splat placeholder, per _redirects file", + Request: Request(). + URL("{{url}}/splat/one.html", redirectDirBaseURL), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("/redirected-splat/one.html"), + ), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/not-found/has-no-redirects-entry returns custom 404, per _redirects file", + Request: Request(). + URL("{{url}}/not-found/has-no-redirects-entry", redirectDirBaseURL), + Response: Expect(). + Status(404). + Headers( + Header("Cache-Control").Equals("public, max-age=29030400, immutable"), + Header("Etag").Equals(`"{{etag}}"`, custom404.Cid().String()), + ). + Body(Contains(custom404.ReadFile())), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/gone/has-no-redirects-entry returns custom 410, per _redirects file", + Request: Request(). + URL("{{url}}/gone/has-no-redirects-entry", redirectDirBaseURL), + Response: Expect(). + Status(410). + Headers( + Header("Cache-Control").Equals("public, max-age=29030400, immutable"), + Header("Etag").Equals(`"{{etag}}"`, custom410.Cid().String()), + ). + Body(Contains(custom410.ReadFile())), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/unavail/has-no-redirects-entry returns custom 451, per _redirects file", + Request: Request(). + URL("{{url}}/unavail/has-no-redirects-entry", redirectDirBaseURL), + Response: Expect(). + Status(451). + Headers( + Header("Cache-Control").Equals("public, max-age=29030400, immutable"), + Header("Etag").Equals(`"{{etag}}"`, custom451.Cid().String()), + ). + Body(Contains(custom451.ReadFile())), + }, + { + Name: "request for $REDIRECTS_DIR_HOSTNAME/catch-all returns 200, per _redirects file", + Request: Request(). + URL("{{url}}/catch-all", redirectDirBaseURL), + Response: Expect(). + Status(200). + Body(Contains("my index")), + }, + }...) - // # Invalid file, containing forced redirect - invalidRedirectsDirCID := fixture.MustGetNode("forced").Base32Cid() - invalidDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, invalidRedirectsDirCID, u.Host) + // # Invalid file, containing forced redirect + invalidRedirectsDirCID := fixture.MustGetNode("forced").Base32Cid() + invalidDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, invalidRedirectsDirCID, u.Host) - tooLargeRedirectsDirCID := fixture.MustGetNode("too-large").Base32Cid() - tooLargeDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, tooLargeRedirectsDirCID, u.Host) + tooLargeRedirectsDirCID := fixture.MustGetNode("too-large").Base32Cid() + tooLargeDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, tooLargeRedirectsDirCID, u.Host) - tests = append(tests, SugarTests{ - { - Name: "invalid file: request for $INVALID_REDIRECTS_DIR_HOSTNAME/not-found returns error about invalid redirects file", - Hint: `if accessing a path that doesn't exist, read _redirects and fail parsing, and return error`, - Request: Request(). - URL("{{url}}/not-found", invalidDirBaseURL), - Response: Expect(). - Status(500). - Body( - And( - Contains("could not parse _redirects:"), - Contains(`forced redirects (or "shadowing") are not supported`), - ), - ).Spec("https://specs.ipfs.tech/http-gateways/web-redirects-file/#no-forced-redirects"), - Spec: "https://specs.ipfs.tech/http-gateways/web-redirects-file/#error-handling", - }, - { - Name: "invalid file: request for $TOO_LARGE_REDIRECTS_DIR_HOSTNAME/not-found returns error about too large redirects file", - Hint: `if accessing a path that doesn't exist and _redirects file is too large, return error`, - Request: Request(). - URL("{{url}}/not-found", tooLargeDirBaseURL), - Response: Expect(). - Status(500). - Body( - And( - Contains("could not parse _redirects:"), - Contains("redirects file size cannot exceed"), - ), + tests = append(tests, SugarTests{ + { + Name: "invalid file: request for $INVALID_REDIRECTS_DIR_HOSTNAME/not-found returns error about invalid redirects file", + Hint: `if accessing a path that doesn't exist, read _redirects and fail parsing, and return error`, + Request: Request(). + URL("{{url}}/not-found", invalidDirBaseURL), + Response: Expect(). + Status(500). + Body( + And( + Contains("could not parse _redirects:"), + Contains(`forced redirects (or "shadowing") are not supported`), + ), + ).Spec("https://specs.ipfs.tech/http-gateways/web-redirects-file/#no-forced-redirects"), + Spec: "https://specs.ipfs.tech/http-gateways/web-redirects-file/#error-handling", + }, + { + Name: "invalid file: request for $TOO_LARGE_REDIRECTS_DIR_HOSTNAME/not-found returns error about too large redirects file", + Hint: `if accessing a path that doesn't exist and _redirects file is too large, return error`, + Request: Request(). + URL("{{url}}/not-found", tooLargeDirBaseURL), + Response: Expect(). + Status(500). + Body( + And( + Contains("could not parse _redirects:"), + Contains("redirects file size cannot exceed"), ), - Spec: "https://specs.ipfs.tech/http-gateways/web-redirects-file/#max-file-size", - }, - }...) + ), + Spec: "https://specs.ipfs.tech/http-gateways/web-redirects-file/#max-file-size", + }, + }...) - // # With CRLF line terminator - newlineRedirectsDirCID := fixture.MustGetNode("newlines").Base32Cid() - newlineBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, newlineRedirectsDirCID, u.Host) + // # With CRLF line terminator + newlineRedirectsDirCID := fixture.MustGetNode("newlines").Base32Cid() + newlineBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, newlineRedirectsDirCID, u.Host) - // # Good codes - goodRedirectDirCID := fixture.MustGetNode("good-codes").Base32Cid() - goodRedirectDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, goodRedirectDirCID, u.Host) + // # Good codes + goodRedirectDirCID := fixture.MustGetNode("good-codes").Base32Cid() + goodRedirectDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, goodRedirectDirCID, u.Host) - // # Bad codes - badRedirectDirCID := fixture.MustGetNode("bad-codes").Base32Cid() - badRedirectDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, badRedirectDirCID, u.Host) + // # Bad codes + badRedirectDirCID := fixture.MustGetNode("bad-codes").Base32Cid() + badRedirectDirBaseURL := Fmt("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, badRedirectDirCID, u.Host) - tests = append(tests, SugarTests{ - { - Name: "newline: request for $NEWLINE_REDIRECTS_DIR_HOSTNAME/redirect-one redirects with default of 301, per _redirects file", - Request: Request(). - URL("{{url}}/redirect-one", newlineBaseURL), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("/one.html"), - ), - }, - { - Name: "good codes: request for $GOOD_REDIRECTS_DIR_HOSTNAME/redirect-one redirects with default of 301, per _redirects file", - Request: Request(). - URL("{{url}}/a301", goodRedirectDirBaseURL), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("/b301"), - ), - }, - { - Name: "bad codes: request for $BAD_REDIRECTS_DIR_HOSTNAME/found.html doesn't return error about bad code", - Request: Request(). - URL("{{url}}/found.html", badRedirectDirBaseURL), - Response: Expect(). - Status(200). - Body( - And( - Contains("my found"), - Not(Contains("unsupported redirect status")), - ), + tests = append(tests, SugarTests{ + { + Name: "newline: request for $NEWLINE_REDIRECTS_DIR_HOSTNAME/redirect-one redirects with default of 301, per _redirects file", + Request: Request(). + URL("{{url}}/redirect-one", newlineBaseURL), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("/one.html"), + ), + }, + { + Name: "good codes: request for $GOOD_REDIRECTS_DIR_HOSTNAME/redirect-one redirects with default of 301, per _redirects file", + Request: Request(). + URL("{{url}}/a301", goodRedirectDirBaseURL), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("/b301"), + ), + }, + { + Name: "bad codes: request for $BAD_REDIRECTS_DIR_HOSTNAME/found.html doesn't return error about bad code", + Request: Request(). + URL("{{url}}/found.html", badRedirectDirBaseURL), + Response: Expect(). + Status(200). + Body( + And( + Contains("my found"), + Not(Contains("unsupported redirect status")), ), - }, - }...) - } + ), + }, + }...) RunWithSpecs(t, helpers.UnwrapSubdomainTests(t, tests), specs.SubdomainGatewayIPFS, specs.RedirectsFile) } @@ -243,8 +237,7 @@ func TestRedirectsFileSupportWithDNSLink(t *testing.T) { dnsLinks := dnslink.MustOpenDNSLink("redirects_file/dnslink.yml") dnsLink := dnsLinks.MustGet("custom-dnslink") - gatewayURL := SubdomainGatewayURL - u, err := url.Parse(gatewayURL) + u, err := url.Parse(SubdomainGatewayURL) if err != nil { t.Fatal(err) } @@ -290,8 +283,7 @@ func TestRedirectsFileWithIfNoneMatchHeader(t *testing.T) { dnsLinks := dnslink.MustOpenDNSLink("redirects_file/dnslink.yml") dnsLink := dnsLinks.MustGet("dnslink-spa") - gatewayURL := SubdomainGatewayURL - u, err := url.Parse(gatewayURL) + u, err := url.Parse(SubdomainGatewayURL) if err != nil { t.Fatal(err) } diff --git a/tests/subdomain_gateway_ipfs_test.go b/tests/subdomain_gateway_ipfs_test.go index f0a61c69f..bca6cbedc 100644 --- a/tests/subdomain_gateway_ipfs_test.go +++ b/tests/subdomain_gateway_ipfs_test.go @@ -19,90 +19,83 @@ func TestUnixFSDirectoryListingOnSubdomainGateway(t *testing.T) { root := fixture.MustGetNode() file := fixture.MustGetNode("ą", "ę", "file-źł.txt") - // We're going to run the same test against multiple gateways (localhost, and a subdomain gateway) - gatewayURLs := []string{ - SubdomainGatewayURL, - SubdomainLocalhostGatewayURL, - } - tests := SugarTests{} - for _, gatewayURL := range gatewayURLs { - u, err := url.Parse(gatewayURL) - if err != nil { - t.Fatal(err) - } + // run against origins explicitly passed via --subdomain-url + u, err := url.Parse(SubdomainGatewayURL) + if err != nil { + t.Fatal(err) + } - tests = append(tests, SugarTests{ - { - Name: "backlink on root CID should be hidden (TODO: cleanup Kubo-specifics)", - Request: Request(). - URL( - "{{scheme}}://{{cid}}.ipfs.{{host}}/", - u.Scheme, - root.Cid(), - u.Host, - ), - Response: Expect(). - BodyWithHint("backlink on root CID should be hidden", - And( - Contains("Index of"), - Not(Contains(`..`)), - )), - }, - { - Name: "redirect dir listing to URL with trailing slash", - Request: Request(). - URL( - "{{scheme}}://{{cid}}.ipfs.{{host}}/ą/ę", - u.Scheme, - root.Cid(), - u.Host, - ), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals(`/%c4%85/%c4%99/`), - ), - }, - { - Name: "Regular dir listing HTML (TODO: cleanup Kubo-specifics)", - Request: Request().URL( - "{{scheme}}://{{cid}}.ipfs.{{host}}/ą/ę/", + tests = append(tests, SugarTests{ + { + Name: "backlink on root CID should be hidden (TODO: cleanup Kubo-specifics)", + Request: Request(). + URL( + "{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, root.Cid(), u.Host, ), - Response: Expect(). - Headers( - Header("Etag").Contains(`"DirIndex-`), - ).BodyWithHint(` + Response: Expect(). + BodyWithHint("backlink on root CID should be hidden", + And( + Contains("Index of"), + Not(Contains(`..`)), + )), + }, + { + Name: "redirect dir listing to URL with trailing slash", + Request: Request(). + URL( + "{{scheme}}://{{cid}}.ipfs.{{host}}/ą/ę", + u.Scheme, + root.Cid(), + u.Host, + ), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals(`/%c4%85/%c4%99/`), + ), + }, + { + Name: "Regular dir listing HTML (TODO: cleanup Kubo-specifics)", + Request: Request().URL( + "{{scheme}}://{{cid}}.ipfs.{{host}}/ą/ę/", + u.Scheme, + root.Cid(), + u.Host, + ), + Response: Expect(). + Headers( + Header("Etag").Contains(`"DirIndex-`), + ).BodyWithHint(` - backlink on subdirectory should point at parent directory (TODO: kubo-specific) - breadcrumbs should leverage path-based router mounted on the parent domain (TODO: kubo-specific) - name column should be a link to content root mounted at subdomain origin `, - And( - Contains("Index of"), - Contains( - `..`, - ), - Contains( - `/ipfs/{{cid}}/ą/ę`, - u.Host, // We don't have a subdomain here which prevents issues with normalization and cidv0 - root.Cid(), - ), - Contains( - `file-źł.txt`, - ), - Contains( - ``, - u.Host, // We don't have a subdomain here which prevents issues with normalization and cidv0 - file.Cid(), - ), - )), - }, - }...) - } + And( + Contains("Index of"), + Contains( + `..`, + ), + Contains( + `/ipfs/{{cid}}/ą/ę`, + u.Host, // We don't have a subdomain here which prevents issues with normalization and cidv0 + root.Cid(), + ), + Contains( + `file-źł.txt`, + ), + Contains( + ``, + u.Host, // We don't have a subdomain here which prevents issues with normalization and cidv0 + file.Cid(), + ), + )), + }, + }...) RunWithSpecs(t, helpers.UnwrapSubdomainTests(t, tests), specs.SubdomainGatewayIPFS) } @@ -125,242 +118,236 @@ func TestGatewaySubdomains(t *testing.T) { tests := SugarTests{} - // We're going to run the same test against multiple gateways (localhost, and a subdomain gateway) - gatewayURLs := []string{ - SubdomainGatewayURL, - SubdomainLocalhostGatewayURL, + // run against origins explicitly passed via --subdomain-url + gatewayURL := SubdomainGatewayURL + u, err := url.Parse(gatewayURL) + if err != nil { + t.Fatal(err) } - for _, gatewayURL := range gatewayURLs { - u, err := url.Parse(gatewayURL) - if err != nil { - t.Fatal(err) - } - - tests = append(tests, SugarTests{ - { - Name: "request for example.com/ipfs/{CIDv1} redirects to subdomain", - Hint: ` + tests = append(tests, SugarTests{ + { + Name: "request for example.com/ipfs/{CIDv1} redirects to subdomain", + Hint: ` path requests to gateways with subdomain support should not return payload directly, but redirect to URL with proper origin isolation `, - Request: Request().URL("{{url}}/ipfs/{{cid}}/", gatewayURL, CIDv1), - Response: Expect(). - Status(301). - Headers( - Header("Location"). - Hint("request for example.com/ipfs/{CIDv1} returns Location HTTP header for subdomain redirect in browsers"). - Contains("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv1, u.Host), - ), - }, - { - Name: "request for example.com/ipfs/{CIDv1}/{filename with percent encoding} redirects to subdomain", - Request: Request().URL("{{url}}/ipfs/{{cid}}/Portugal%252C+España=Peninsula%20Ibérica.txt", gatewayURL, dirWithPercentEncodedFilenameCID), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("{{scheme}}://{{cid}}.ipfs.{{host}}/Portugal%252C+Espa%C3%B1a=Peninsula%20Ib%C3%A9rica.txt", u.Scheme, dirWithPercentEncodedFilenameCID, u.Host), - ), - }, - { - Name: "request for example.com/ipfs/{DirCID} redirects to subdomain", - Hint: ` + Request: Request().URL("{{url}}/ipfs/{{cid}}/", gatewayURL, CIDv1), + Response: Expect(). + Status(301). + Headers( + Header("Location"). + Hint("request for example.com/ipfs/{CIDv1} returns Location HTTP header for subdomain redirect in browsers"). + Contains("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv1, u.Host), + ), + }, + { + Name: "request for example.com/ipfs/{CIDv1}/{filename with percent encoding} redirects to subdomain", + Request: Request().URL("{{url}}/ipfs/{{cid}}/Portugal%252C+España=Peninsula%20Ibérica.txt", gatewayURL, dirWithPercentEncodedFilenameCID), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("{{scheme}}://{{cid}}.ipfs.{{host}}/Portugal%252C+Espa%C3%B1a=Peninsula%20Ib%C3%A9rica.txt", u.Scheme, dirWithPercentEncodedFilenameCID, u.Host), + ), + }, + { + Name: "request for example.com/ipfs/{DirCID} redirects to subdomain", + Hint: ` path requests to gateways with subdomain support should not return payload directly, but redirect to URL with proper origin isolation `, - Request: Request().URL("{{url}}/ipfs/{{cid}}/", gatewayURL, DirCID), - Response: Expect(). - Status(301). - Headers( - Header("Location"). - Hint("request for example.com/ipfs/{DirCID} returns Location HTTP header for subdomain redirect in browsers"). - Contains("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, DirCID, u.Host), - ), - }, - { - Name: "request for example.com/ipfs/{CIDv0} redirects to CIDv1 representation in subdomain", - Request: Request().URL("{{url}}/ipfs/{{cid}}/", gatewayURL, CIDv0), - Response: Expect(). - Status(301). - Headers( - Header("Location"). - Hint("request for example.com/ipfs/{CIDv0to1} returns Location HTTP header for subdomain redirect in browsers"). - Contains("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv0to1, u.Host), - ), - }, - { - Name: "request for {CID}.ipfs.example.com should return expected payload", - Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, CIDv1, u.Host), - Response: Expect(). - Status(200). - Body(Contains(CIDVal)), - }, - { - Name: "request for {CID}.ipfs.example.com/ipfs/{CID} should return HTTP 404", - Hint: "ensure /ipfs/ namespace is not mounted on subdomain", - Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/{{cid}}", u.Scheme, CIDv1, u.Host), - Response: Expect(). - Status(404), - }, - { - Name: "request for {CID}.ipfs.example.com/ipfs/file.txt should return data from a file in CID content root", - Hint: "ensure requests to /ipfs/* are not blocked, if content root has such subdirectory", - Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/file.txt", u.Scheme, DirCID, u.Host), - Response: Expect(). - Status(200). - Body(Contains("I am a txt file")), - }, - { - Name: "valid file and subdirectory paths in directory listing at {cid}.ipfs.example.com", - Hint: "{CID}.ipfs.example.com/sub/dir (Directory Listing)", - Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, DirCID, u.Host), - Response: Expect(). - Status(200). - Body(And( - // TODO: implement html expectations - Contains(`hello`), - Contains(`ipfs`), - )), - }, - { - Name: "valid parent directory path in directory listing at {cid}.ipfs.example.com/sub/dir", - Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/ipns/", u.Scheme, DirCID, u.Host), - Response: Expect(). - Status(200). - Body(And( - // TODO: implement html expectations - Contains(`..`), - Contains(`bar`), - )), - }, - { - Name: "request for deep path resource at {cid}.ipfs.example.com/sub/dir/file", - Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/ipns/bar", u.Scheme, DirCID, u.Host), - Response: Expect(). - Status(200). - Body(Contains("text-file-content")), - }, - { - Name: "valid breadcrumb links in the header of directory listing at {cid}.ipfs.example.com/sub/dir (TODO: cleanup Kubo-specifics)", - Hint: ` + Request: Request().URL("{{url}}/ipfs/{{cid}}/", gatewayURL, DirCID), + Response: Expect(). + Status(301). + Headers( + Header("Location"). + Hint("request for example.com/ipfs/{DirCID} returns Location HTTP header for subdomain redirect in browsers"). + Contains("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, DirCID, u.Host), + ), + }, + { + Name: "request for example.com/ipfs/{CIDv0} redirects to CIDv1 representation in subdomain", + Request: Request().URL("{{url}}/ipfs/{{cid}}/", gatewayURL, CIDv0), + Response: Expect(). + Status(301). + Headers( + Header("Location"). + Hint("request for example.com/ipfs/{CIDv0to1} returns Location HTTP header for subdomain redirect in browsers"). + Contains("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv0to1, u.Host), + ), + }, + { + Name: "request for {CID}.ipfs.example.com should return expected payload", + Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}", u.Scheme, CIDv1, u.Host), + Response: Expect(). + Status(200). + Body(Contains(CIDVal)), + }, + { + Name: "request for {CID}.ipfs.example.com/ipfs/{CID} should return HTTP 404", + Hint: "ensure /ipfs/ namespace is not mounted on subdomain", + Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/{{cid}}", u.Scheme, CIDv1, u.Host), + Response: Expect(). + Status(404), + }, + { + Name: "request for {CID}.ipfs.example.com/ipfs/file.txt should return data from a file in CID content root", + Hint: "ensure requests to /ipfs/* are not blocked, if content root has such subdirectory", + Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/file.txt", u.Scheme, DirCID, u.Host), + Response: Expect(). + Status(200). + Body(Contains("I am a txt file")), + }, + { + Name: "valid file and subdirectory paths in directory listing at {cid}.ipfs.example.com", + Hint: "{CID}.ipfs.example.com/sub/dir (Directory Listing)", + Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, DirCID, u.Host), + Response: Expect(). + Status(200). + Body(And( + // TODO: implement html expectations + Contains(`hello`), + Contains(`ipfs`), + )), + }, + { + Name: "valid parent directory path in directory listing at {cid}.ipfs.example.com/sub/dir", + Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/ipns/", u.Scheme, DirCID, u.Host), + Response: Expect(). + Status(200). + Body(And( + // TODO: implement html expectations + Contains(`..`), + Contains(`bar`), + )), + }, + { + Name: "request for deep path resource at {cid}.ipfs.example.com/sub/dir/file", + Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/ipns/bar", u.Scheme, DirCID, u.Host), + Response: Expect(). + Status(200). + Body(Contains("text-file-content")), + }, + { + Name: "valid breadcrumb links in the header of directory listing at {cid}.ipfs.example.com/sub/dir (TODO: cleanup Kubo-specifics)", + Hint: ` Note 1: we test for sneaky subdir names {cid}.ipfs.example.com/ipfs/ipns/ :^) Note 2: example.com/ipfs/.. present in HTML will be redirected to subdomain, so this is expected behavior `, - Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/ipns/", u.Scheme, DirCID, u.Host), - Response: Expect(). - Status(200). - Body( - And( - Contains("Index of"), - Contains(`/ipfs/{{cid}}/ipfs/ipns`, - u.Host, DirCID), - ), - ), - }, - { - Name: "request for example.com/ipfs/{CIDv1} produces redirect to {CIDv1}.ipfs.example.com", - Hint: "path requests to the root hostname should redirect to a subdomain URL with proper origin isolation", - Request: Request().URL("{{scheme}}://{{host}}/ipfs/{{cid}}/", u.Scheme, u.Host, CIDv1), - Response: Expect(). - Headers( - Header("Location").Equals("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv1, u.Host), + Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/ipfs/ipns/", u.Scheme, DirCID, u.Host), + Response: Expect(). + Status(200). + Body( + And( + Contains("Index of"), + Contains(`/ipfs/{{cid}}/ipfs/ipns`, + u.Host, DirCID), ), - }, - { - Name: "request for example.com/ipfs/{InvalidCID} produces useful error before redirect", - Hint: "error message should include original CID (and it should be case-sensitive, as we can't assume everyone uses base32)", - Request: Request().URL("{{scheme}}://{{host}}/ipfs/QmInvalidCID", u.Scheme, u.Host), - Response: Expect(). - Body(Contains(`invalid path "/ipfs/QmInvalidCID"`)), - }, + ), + }, + { + Name: "request for example.com/ipfs/{CIDv1} produces redirect to {CIDv1}.ipfs.example.com", + Hint: "path requests to the root hostname should redirect to a subdomain URL with proper origin isolation", + Request: Request().URL("{{scheme}}://{{host}}/ipfs/{{cid}}/", u.Scheme, u.Host, CIDv1), + Response: Expect(). + Headers( + Header("Location").Equals("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv1, u.Host), + ), + }, + { + Name: "request for example.com/ipfs/{InvalidCID} produces useful error before redirect", + Hint: "error message should include original CID (and it should be case-sensitive, as we can't assume everyone uses base32)", + Request: Request().URL("{{scheme}}://{{host}}/ipfs/QmInvalidCID", u.Scheme, u.Host), + Response: Expect(). + Body(Contains(`invalid path "/ipfs/QmInvalidCID"`)), + }, - { - Name: "request for example.com/ipfs/{CIDv0} produces redirect to {CIDv1}.ipfs.example.com", - Request: Request().URL("{{scheme}}://{{host}}/ipfs/{{cid}}/", u.Scheme, u.Host, CIDv0), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv0to1, u.Host), - ), - }, + { + Name: "request for example.com/ipfs/{CIDv0} produces redirect to {CIDv1}.ipfs.example.com", + Request: Request().URL("{{scheme}}://{{host}}/ipfs/{{cid}}/", u.Scheme, u.Host, CIDv0), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv0to1, u.Host), + ), + }, - { - Name: "request for http://example.com/ipfs/{CID} with X-Forwarded-Proto: https produces redirect to HTTPS URL", - Hint: "Support X-Forwarded-Proto", - Request: Request().URL("{{scheme}}://{{host}}/ipfs/{{cid}}/", u.Scheme, u.Host, CIDv1). - Header("X-Forwarded-Proto", "https"), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("https://{{cid}}.ipfs.{{host}}/", CIDv1, u.Host), - ), - }, - { - Name: "request for example.com/ipfs/?uri=ipfs%3A%2F%2F.. produces redirect to /ipfs/.. content path", - Hint: "Support ipfs:// in https://developer.mozilla.org/en-US/docs/Web/API/Navigator/registerProtocolHandler", - Request: Request().URL("{{scheme}}://{{host}}/ipfs/", u.Scheme, u.Host). - Query( - "uri", "ipfs://{{host}}/wiki/Diego_Maradona.html", CIDWikipedia, - ), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("/ipfs/{{cid}}/wiki/Diego_Maradona.html", CIDWikipedia), - ), - }, - { - Name: "request for a too long CID at example.com/ipfs/{CIDv1} returns human readable error", - Hint: "router should not redirect to hostnames that could fail due to DNS limits", - Request: Request().URL("{{url}}/ipfs/{{cid}}", gatewayURL, CIDv1_TOO_LONG), - Response: Expect(). - Status(400). - Body(Contains("CID incompatible with DNS label length limit of 63")), - }, - { - Name: "request for a too long CID at {CIDv1}.ipfs.example.com returns expected payload", - Hint: "direct request should also fail (provides the same UX as router and avoids confusion)", - Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv1_TOO_LONG, u.Host), - Response: Expect(). - Status(400). - Body(Contains("CID incompatible with DNS label length limit of 63")), - }, - // ## ============================================================================ - // ## Test support for X-Forwarded-Host - // ## ============================================================================ - { - Name: "request for http://fake.domain.com/ipfs/{CID} doesn't match the example.com gateway", - Request: Request(). - URL("{{scheme}}://{{domain}}/ipfs/{{cid}}", u.Scheme, "fake.domain.com", CIDv1), - Response: Expect(). - Status(200), - }, - { - Name: "request for http://fake.domain.com/ipfs/{CID} with X-Forwarded-Host: example.com match the example.com gateway", - Request: Request(). - URL("{{scheme}}://{{domain}}/ipfs/{{cid}}", u.Scheme, "fake.domain.com", CIDv1). - Header("X-Forwarded-Host", u.Host), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv1, u.Host), - ), - }, - { - Name: "request for http://fake.domain.com/ipfs/{CID} with X-Forwarded-Host: example.com and X-Forwarded-Proto: https match the example.com gateway, redirect with https", - Request: Request(). - URL("{{scheme}}://{{domain}}/ipfs/{{cid}}", u.Scheme, "fake.domain.com", CIDv1). - Header("X-Forwarded-Host", u.Host). - Header("X-Forwarded-Proto", "https"), - Response: Expect(). - Status(301). - Headers( - Header("Location").Equals("https://{{cid}}.ipfs.{{host}}/", CIDv1, u.Host), - ), - }, - }...) - } + { + Name: "request for http://example.com/ipfs/{CID} with X-Forwarded-Proto: https produces redirect to HTTPS URL", + Hint: "Support X-Forwarded-Proto", + Request: Request().URL("{{scheme}}://{{host}}/ipfs/{{cid}}/", u.Scheme, u.Host, CIDv1). + Header("X-Forwarded-Proto", "https"), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("https://{{cid}}.ipfs.{{host}}/", CIDv1, u.Host), + ), + }, + { + Name: "request for example.com/ipfs/?uri=ipfs%3A%2F%2F.. produces redirect to /ipfs/.. content path", + Hint: "Support ipfs:// in https://developer.mozilla.org/en-US/docs/Web/API/Navigator/registerProtocolHandler", + Request: Request().URL("{{scheme}}://{{host}}/ipfs/", u.Scheme, u.Host). + Query( + "uri", "ipfs://{{host}}/wiki/Diego_Maradona.html", CIDWikipedia, + ), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("/ipfs/{{cid}}/wiki/Diego_Maradona.html", CIDWikipedia), + ), + }, + { + Name: "request for a too long CID at example.com/ipfs/{CIDv1} returns human readable error", + Hint: "router should not redirect to hostnames that could fail due to DNS limits", + Request: Request().URL("{{url}}/ipfs/{{cid}}", gatewayURL, CIDv1_TOO_LONG), + Response: Expect(). + Status(400). + Body(Contains("CID incompatible with DNS label length limit of 63")), + }, + { + Name: "request for a too long CID at {CIDv1}.ipfs.example.com returns expected payload", + Hint: "direct request should also fail (provides the same UX as router and avoids confusion)", + Request: Request().URL("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv1_TOO_LONG, u.Host), + Response: Expect(). + Status(400). + Body(Contains("CID incompatible with DNS label length limit of 63")), + }, + // ## ============================================================================ + // ## Test support for X-Forwarded-Host + // ## ============================================================================ + { + Name: "request for http://fake.domain.com/ipfs/{CID} doesn't match the example.com gateway", + Request: Request(). + URL("{{scheme}}://{{domain}}/ipfs/{{cid}}", u.Scheme, "fake.domain.com", CIDv1), + Response: Expect(). + Status(200), + }, + { + Name: "request for http://fake.domain.com/ipfs/{CID} with X-Forwarded-Host: example.com match the example.com gateway", + Request: Request(). + URL("{{scheme}}://{{domain}}/ipfs/{{cid}}", u.Scheme, "fake.domain.com", CIDv1). + Header("X-Forwarded-Host", u.Host), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("{{scheme}}://{{cid}}.ipfs.{{host}}/", u.Scheme, CIDv1, u.Host), + ), + }, + { + Name: "request for http://fake.domain.com/ipfs/{CID} with X-Forwarded-Host: example.com and X-Forwarded-Proto: https match the example.com gateway, redirect with https", + Request: Request(). + URL("{{scheme}}://{{domain}}/ipfs/{{cid}}", u.Scheme, "fake.domain.com", CIDv1). + Header("X-Forwarded-Host", u.Host). + Header("X-Forwarded-Proto", "https"), + Response: Expect(). + Status(301). + Headers( + Header("Location").Equals("https://{{cid}}.ipfs.{{host}}/", CIDv1, u.Host), + ), + }, + }...) RunWithSpecs(t, helpers.UnwrapSubdomainTests(t, tests), specs.SubdomainGatewayIPFS) } diff --git a/tests/subdomain_gateway_ipns_test.go b/tests/subdomain_gateway_ipns_test.go index 58211eacc..74c4ce769 100644 --- a/tests/subdomain_gateway_ipns_test.go +++ b/tests/subdomain_gateway_ipns_test.go @@ -26,138 +26,131 @@ func TestGatewaySubdomainAndIPNS(t *testing.T) { car := car.MustOpenUnixfsCar("subdomain_gateway/fixtures.car") payload := string(car.MustGetRawData("hello-CIDv1")) - // We're going to run the same test against multiple gateways (localhost, and a subdomain gateway) - gatewayURLs := []string{ - SubdomainGatewayURL, - SubdomainLocalhostGatewayURL, - } - ipnsRecords := []*ipns.IpnsRecord{ rsaFixture, ed25519Fixture, } - for _, gatewayURL := range gatewayURLs { - u, err := url.Parse(gatewayURL) - if err != nil { - t.Fatal(err) - } - - for _, record := range ipnsRecords { - tests = append(tests, SugarTests{ - { - Name: "request for /ipns/{CIDv0} redirects to CIDv1 with libp2p-key multicodec in subdomain", - Request: Request(). - URL("{{url}}/ipns/{{cid}}", gatewayURL, record.IdV0()), - Response: Expect(). - Status(301). - Headers( - Header("Location"). - Equals("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, record.IdV1(), u.Host), - ), - }, - { - Name: "request for {CIDv1-libp2p-key}.ipns.{gateway} returns expected payload", - Request: Request(). - URL("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, record.IdV1(), u.Host), - Response: Expect(). - Status(200). - BodyWithHint("Request for {{cid}}.ipns.{{host}} returns expected payload", payload), - }, - { - Name: "request for {CIDv1-dag-pb}.ipns.{gateway} redirects to CID with libp2p-key multicodec", - Request: Request(). - URL("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, record.ToCID(multicodec.DagPb, multibase.Base36), u.Host), - Response: Expect(). - Status(301). - Headers( - Header("Location"). - Equals("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, record.IdV1(), u.Host), - ), - }, - // # *.ipns.example.com - // # ============================================================================ - - // # .ipns.example.com - - // test_hostname_gateway_response_should_contain \ - // "request for {CIDv1-libp2p-key}.ipns.example.com returns expected payload" \ - // "${RSA_IPNS_IDv1}.ipns.example.com" \ - // "http://127.0.0.1:$GWAY_PORT" \ - // "$CID_VAL" - - // test_hostname_gateway_response_should_contain \ - // "request for {CIDv1-libp2p-key}.ipns.example.com returns expected payload" \ - // "${ED25519_IPNS_IDv1}.ipns.example.com" \ - // "http://127.0.0.1:$GWAY_PORT" \ - // "$CID_VAL" - - // test_hostname_gateway_response_should_contain \ - // "hostname request for {CIDv1-dag-pb}.ipns.localhost redirects to CID with libp2p-key multicodec" \ - // "${RSA_IPNS_IDv1_DAGPB}.ipns.example.com" \ - // "http://127.0.0.1:$GWAY_PORT" \ - // "Location: http://${RSA_IPNS_IDv1}.ipns.example.com/" - - // test_hostname_gateway_response_should_contain \ - // "hostname request for {CIDv1-dag-pb}.ipns.localhost redirects to CID with libp2p-key multicodec" \ - // "${ED25519_IPNS_IDv1_DAGPB}.ipns.example.com" \ - // "http://127.0.0.1:$GWAY_PORT" \ - // "Location: http://${ED25519_IPNS_IDv1}.ipns.example.com/" - // # disable /ipns for the hostname by not whitelisting it - // ipfs config --json Gateway.PublicGateways '{ - // "example.com": { - // "UseSubdomains": true, - // "Paths": ["/ipfs"] - // } - // }' || exit 1 - // # restart daemon to apply config changes - // test_kill_ipfs_daemon - // test_launch_ipfs_daemon_without_network - - // TODO: what to do with these? - // # refuse requests to Paths that were not explicitly whitelisted for the hostname - // test_hostname_gateway_response_should_contain \ - // "request for *.ipns.example.com returns HTTP 404 Not Found when /ipns is not on Paths whitelist" \ - // "${RSA_IPNS_IDv1}.ipns.example.com" \ - // "http://127.0.0.1:$GWAY_PORT" \ - // "404 Not Found" - - // test_hostname_gateway_response_should_contain \ - // "request for *.ipns.example.com returns HTTP 404 Not Found when /ipns is not on Paths whitelist" \ - // "${ED25519_IPNS_IDv1}.ipns.example.com" \ - // "http://127.0.0.1:$GWAY_PORT" \ - // "404 Not Found" - - // # refuse requests to Paths that were not explicitly whitelisted for the hostname - // test_hostname_gateway_response_should_contain \ - // "request for example.com/ipns/ returns HTTP 404 Not Found when /ipns is not on Paths whitelist" \ - // "example.com" \ - // "http://127.0.0.1:$GWAY_PORT/ipns/$RSA_IPNS_IDv1" \ - // "404 Not Found" - - // test_hostname_gateway_response_should_contain \ - // "request for example.com/ipns/ returns HTTP 404 Not Found when /ipns is not on Paths whitelist" \ - // "example.com" \ - // "http://127.0.0.1:$GWAY_PORT/ipns/$ED25519_IPNS_IDv1" \ - // "404 Not Found" - }...) - } + // run against origins passed via --subdomain-url (e.g. http://localhost:port) + gatewayURL := SubdomainGatewayURL + u, err := url.Parse(gatewayURL) + if err != nil { + t.Fatal(err) + } + for _, record := range ipnsRecords { tests = append(tests, SugarTests{ { - Name: "request for a ED25519 libp2p-key at example.com/ipns/{b58mh} returns Location HTTP header for DNS-safe subdomain redirect in browsers", + Name: "request for /ipns/{CIDv0} redirects to CIDv1 with libp2p-key multicodec in subdomain", Request: Request(). - URL("{{url}}/ipns/{{cid}}", gatewayURL, ed25519Fixture.B58MH()), + URL("{{url}}/ipns/{{cid}}", gatewayURL, record.IdV0()), Response: Expect(). + Status(301). Headers( Header("Location"). - Equals("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, ed25519Fixture.ToCID(multicodec.Libp2pKey, multibase.Base36), u.Host), + Equals("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, record.IdV1(), u.Host), ), }, + { + Name: "request for {CIDv1-libp2p-key}.ipns.{gateway} returns expected payload", + Request: Request(). + URL("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, record.IdV1(), u.Host), + Response: Expect(). + Status(200). + BodyWithHint("Request for {{cid}}.ipns.{{host}} returns expected payload", payload), + }, + { + Name: "request for {CIDv1-dag-pb}.ipns.{gateway} redirects to CID with libp2p-key multicodec", + Request: Request(). + URL("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, record.ToCID(multicodec.DagPb, multibase.Base36), u.Host), + Response: Expect(). + Status(301). + Headers( + Header("Location"). + Equals("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, record.IdV1(), u.Host), + ), + }, + // # *.ipns.example.com + // # ============================================================================ + + // # .ipns.example.com + + // test_hostname_gateway_response_should_contain \ + // "request for {CIDv1-libp2p-key}.ipns.example.com returns expected payload" \ + // "${RSA_IPNS_IDv1}.ipns.example.com" \ + // "http://127.0.0.1:$GWAY_PORT" \ + // "$CID_VAL" + + // test_hostname_gateway_response_should_contain \ + // "request for {CIDv1-libp2p-key}.ipns.example.com returns expected payload" \ + // "${ED25519_IPNS_IDv1}.ipns.example.com" \ + // "http://127.0.0.1:$GWAY_PORT" \ + // "$CID_VAL" + + // test_hostname_gateway_response_should_contain \ + // "hostname request for {CIDv1-dag-pb}.ipns.localhost redirects to CID with libp2p-key multicodec" \ + // "${RSA_IPNS_IDv1_DAGPB}.ipns.example.com" \ + // "http://127.0.0.1:$GWAY_PORT" \ + // "Location: http://${RSA_IPNS_IDv1}.ipns.example.com/" + + // test_hostname_gateway_response_should_contain \ + // "hostname request for {CIDv1-dag-pb}.ipns.localhost redirects to CID with libp2p-key multicodec" \ + // "${ED25519_IPNS_IDv1_DAGPB}.ipns.example.com" \ + // "http://127.0.0.1:$GWAY_PORT" \ + // "Location: http://${ED25519_IPNS_IDv1}.ipns.example.com/" + // # disable /ipns for the hostname by not whitelisting it + // ipfs config --json Gateway.PublicGateways '{ + // "example.com": { + // "UseSubdomains": true, + // "Paths": ["/ipfs"] + // } + // }' || exit 1 + // # restart daemon to apply config changes + // test_kill_ipfs_daemon + // test_launch_ipfs_daemon_without_network + + // TODO: what to do with these? + // # refuse requests to Paths that were not explicitly whitelisted for the hostname + // test_hostname_gateway_response_should_contain \ + // "request for *.ipns.example.com returns HTTP 404 Not Found when /ipns is not on Paths whitelist" \ + // "${RSA_IPNS_IDv1}.ipns.example.com" \ + // "http://127.0.0.1:$GWAY_PORT" \ + // "404 Not Found" + + // test_hostname_gateway_response_should_contain \ + // "request for *.ipns.example.com returns HTTP 404 Not Found when /ipns is not on Paths whitelist" \ + // "${ED25519_IPNS_IDv1}.ipns.example.com" \ + // "http://127.0.0.1:$GWAY_PORT" \ + // "404 Not Found" + + // # refuse requests to Paths that were not explicitly whitelisted for the hostname + // test_hostname_gateway_response_should_contain \ + // "request for example.com/ipns/ returns HTTP 404 Not Found when /ipns is not on Paths whitelist" \ + // "example.com" \ + // "http://127.0.0.1:$GWAY_PORT/ipns/$RSA_IPNS_IDv1" \ + // "404 Not Found" + + // test_hostname_gateway_response_should_contain \ + // "request for example.com/ipns/ returns HTTP 404 Not Found when /ipns is not on Paths whitelist" \ + // "example.com" \ + // "http://127.0.0.1:$GWAY_PORT/ipns/$ED25519_IPNS_IDv1" \ + // "404 Not Found" }...) - } + tests = append(tests, SugarTests{ + { + Name: "request for a ED25519 libp2p-key at example.com/ipns/{b58mh} returns Location HTTP header for DNS-safe subdomain redirect in browsers", + Request: Request(). + URL("{{url}}/ipns/{{cid}}", gatewayURL, ed25519Fixture.B58MH()), + Response: Expect(). + Headers( + Header("Location"). + Equals("{{scheme}}://{{cid}}.ipns.{{host}}/", u.Scheme, ed25519Fixture.ToCID(multicodec.Libp2pKey, multibase.Base36), u.Host), + ), + }, + }...) + RunWithSpecs(t, helpers.UnwrapSubdomainTests(t, tests), specs.SubdomainGatewayIPNS) } @@ -166,68 +159,62 @@ func TestSubdomainGatewayDNSLinkInlining(t *testing.T) { tests := SugarTests{} - // We're going to run the same test against multiple gateways (localhost, and a subdomain gateway) - gatewayURLs := []string{ - SubdomainGatewayURL, - SubdomainLocalhostGatewayURL, - } - dnsLinks := dnslink.MustOpenDNSLink("subdomain_gateway/dnslink.yml") wikipedia := dnsLinks.MustGet("wikipedia") dnsLinkTest := dnsLinks.MustGet("test") - for _, gatewayURL := range gatewayURLs { - u, err := url.Parse(gatewayURL) - if err != nil { - t.Fatal(err) - } + // run against origins passed via --subdomain-url + gatewayURL := SubdomainGatewayURL + u, err := url.Parse(gatewayURL) + if err != nil { + t.Fatal(err) + } - tests = append(tests, SugarTests{ - { - Name: "request for /ipns/{fqdn} redirects to DNSLink in subdomain", - Request: Request(). - URL("{{url}}/ipns/{{fqdn}}/wiki/", gatewayURL, wikipedia), - Response: Expect(). - Headers( - Header("Location"). - Equals("{{scheme}}://{{fqdn}}.ipns.{{host}}/wiki/", u.Scheme, dnslink.InlineDNS(wikipedia), u.Host), - ), - }, - { - Name: "request for {dnslink}.ipns.{gateway} returns expected payload", - Request: Request(). - URL("{{scheme}}://{{fqdn}}.ipns.{{host}}", u.Scheme, dnsLinkTest, u.Host), - Response: Expect(). - Body("hello\n"), - }, - { - Name: "request for example.com/ipns/{fqdn} with X-Forwarded-Proto redirects to TLS-safe label in subdomain", - Hint: ` + tests = append(tests, SugarTests{ + { + Name: "request for /ipns/{fqdn} redirects to DNSLink in subdomain", + Request: Request(). + URL("{{url}}/ipns/{{fqdn}}/wiki/", gatewayURL, wikipedia), + Response: Expect(). + Headers( + Header("Location"). + Equals("{{scheme}}://{{fqdn}}.ipns.{{host}}/wiki/", u.Scheme, dnslink.InlineDNS(wikipedia), u.Host), + ), + }, + { + Name: "request for {dnslink}.ipns.{gateway} returns expected payload", + Request: Request(). + URL("{{scheme}}://{{fqdn}}.ipns.{{host}}", u.Scheme, dnsLinkTest, u.Host), + Response: Expect(). + Body("hello\n"), + }, + { + Name: "request for example.com/ipns/{fqdn} with X-Forwarded-Proto redirects to TLS-safe label in subdomain", + Hint: ` DNSLink on Public gateway with a single-level wildcard TLS cert "Option C" from https://github.com/ipfs/in-web-browsers/issues/169 `, - Request: Request(). - Header("X-Forwarded-Proto", "https"). - URL("{{url}}/ipns/{{wikipedia}}/wiki/", gatewayURL, wikipedia), - Response: Expect(). - Headers( - Header("Location"). - Equals("https://{{inlined}}.ipns.{{host}}/wiki/", dnslink.InlineDNS(wikipedia), u.Host), - ), - }, - { - Name: `request for example.com/ipns/?uri=ipns%3A%2F%2F.. produces redirect to /ipns/.. content path`, - Hint: "Support ipns:// in https://developer.mozilla.org/en-US/docs/Web/API/Navigator/registerProtocolHandler", - Request: Request(). - // TODO: use Query or future QueryRaw here - URL(`{{url}}/ipns/?uri=ipns%3A%2F%2F{{dnslink}}`, gatewayURL, wikipedia), - Response: Expect(). - Headers( - Header("Location").Equals("/ipns/{{wikipedia}}", wikipedia), - ), - }, - }...) - } + Request: Request(). + Header("X-Forwarded-Proto", "https"). + URL("{{url}}/ipns/{{wikipedia}}/wiki/", gatewayURL, wikipedia), + Response: Expect(). + Headers( + Header("Location"). + Equals("https://{{inlined}}.ipns.{{host}}/wiki/", dnslink.InlineDNS(wikipedia), u.Host), + ), + }, + { + Name: `request for example.com/ipns/?uri=ipns%3A%2F%2F.. produces redirect to /ipns/.. content path`, + Hint: "Support ipns:// in https://developer.mozilla.org/en-US/docs/Web/API/Navigator/registerProtocolHandler", + Request: Request(). + // TODO: use Query or future QueryRaw here + URL(`{{url}}/ipns/?uri=ipns%3A%2F%2F{{dnslink}}`, gatewayURL, wikipedia), + Response: Expect(). + Headers( + Header("Location").Equals("/ipns/{{wikipedia}}", wikipedia), + ), + }, + }...) RunWithSpecs(t, helpers.UnwrapSubdomainTests(t, tests), specs.SubdomainGatewayIPNS) } diff --git a/tooling/helpers/subdomain.go b/tooling/helpers/subdomain.go index ae38bf25f..06d04f19f 100644 --- a/tooling/helpers/subdomain.go +++ b/tooling/helpers/subdomain.go @@ -39,12 +39,12 @@ func unwrapSubdomainTest(t *testing.T, unwraped test.SugarTest) test.SugarTests // // The test knows two addresses: // - GatewayURL: the URL we connect to, it might be "dweb.link", "127.0.0.1:8080", etc. - // - SubdomainGatewayURL: the URL we test for subdomain requests, it might be "dweb.link", "localhost", "example.com", etc. + // - SubdomainGatewayURL: the origin that informs value in Host HTTP header used for subdomain requests, it might be "dweb.link", "localhost", "example.com", etc. - // host is the hostname of the gateway we are testing, it might be `localhost` or `example.com` + // host is the subdomain ggateway origin we are testing, it might be `localhost` or `example.com` host := u.Host - // raw url is the url but we replace the host with our local url, it might be `http://127.0.0.1/ipfs/something` + // rawURL is the low level HTTP endpoint that is supposed to understand Host header (it might be `http://127.0.0.1/ipfs/something`) u.Host = test.GatewayHost rawURL := u.String() diff --git a/tooling/test/config.go b/tooling/test/config.go index 506c03785..3d501da83 100644 --- a/tooling/test/config.go +++ b/tooling/test/config.go @@ -22,15 +22,13 @@ var GatewayURL = strings.TrimRight( "/") var SubdomainGatewayURL = strings.TrimRight( - GetEnv("SUBDOMAIN_GATEWAY_URL", "http://example.com"), + GetEnv("SUBDOMAIN_GATEWAY_URL", "http://localhost:8080"), "/") var GatewayHost = "" var SubdomainGatewayHost = "" var SubdomainGatewayScheme = "" -var SubdomainLocalhostGatewayURL = "http://localhost" - func init() { parsed, err := url.Parse(GatewayURL) if err != nil {