feat: add `maxReturnSize` option #90

ledyba · 2024-11-16T14:58:14Z

Currently, this library tends to send entire file when 'acceptRanges': true,

In this PR, I added maxChunkSize to send.SendOptions to limit the max chunk size.

Test case: https://hexe.net/2022/09/17/17:04:55/
Souce code is here

I'm using firefox 132.0.

Request:

GET /entity/eb3467a204907295921f367ac8187d67 HTTP/2
Host: hexe.net
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0
Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5
Accept-Language: ja
Range: bytes=0-
DNT: 1
Connection: keep-alive
Referer: https://hexe.net/2022/09/17/17:04:55/
Sec-Fetch-Dest: video
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-origin
Accept-Encoding: identity
Priority: u=4
TE: trailers

Response:

HTTP/2 206 
server: nginx/1.26.1
date: Sat, 16 Nov 2024 14:48:38 GMT
content-type: video/mp4
content-length: 65347058
accept-ranges: bytes
cache-control: public, max-age=0
last-modified: Sat, 17 Sep 2022 08:04:49 GMT
etag: W/"3e51df2-1834a7a8ef4"
content-range: bytes 0-65347057/65347058
content-disposition: inline; filename="eb3467a204907295921f367ac8187d67.original.mp4"
X-Firefox-Spdy: h2

I also checked in MS Edge 131.0.2903.51.

Checklist

run npm run test and npm run benchmark
- This patch does not affect benchmark, and also, there is'nt npm run benchmark command
tests and/or benchmarks are included
documentation is changed or added
commit message and code follows the Developer's Certification of Origin
and the Code of conduct

types/index.d.ts

ledyba

Thanks for the review. I added tsd tests for maxChunkSize and acceptRanges.

Could you take another look?

Uzlopak · 2024-11-17T11:42:11Z

Is a maxChunkSize not basically a highWatermark? Should we name it like that?

ledyba · 2024-11-17T12:23:37Z

I don't think that maxChunkSize is the best name. I'm not an English native, so I'd appreciate any advice.

Copilot reviewed 6 out of 6 changed files in this pull request and generated no suggestions.

Comments skipped due to low confidence (2)

test/utils.js:12

The function name should be 'shouldHaveHeader' instead of 'shouldNotHaveHeader'.

module.exports.shouldHaveHeader = function shouldNotHaveHeader (header, t) {

README.md:52

The heading should be 'maxChunkSize' instead of 'acceptRanges'.

##### acceptRanges

test/send.1.test.js

README.md

test/utils.js

Co-authored-by: Frazer Smith <[email protected]> Signed-off-by: Kaede Fujisaki <[email protected]>

ledyba

Thank you for the review and typo suggestion!

Could you take another look?

README.md

lib/send.js

Uzlopak · 2024-11-21T13:26:20Z

The name is not that nice. maxChunkSize is imho a highWatermark . @fastify/send only supports bytes for range. So it is de facto a highWaterMark

Uzlopak · 2024-11-21T13:26:59Z

lib/send.js

@@ -546,6 +551,9 @@ function sendFileDirectly (request, path, stat, options) {

          // Content-Range
          statusCode = 206
+          if (options.maxChunkSize) {
+            ranges[0].end = Math.min(ranges[0].end, ranges[0].start + options.maxChunkSize - 1)


It doesnt seem right.

There are some unit tests that success, so I think you are pointing out some edge cases that cause bug. If so, could you tell me the condition or something?

Uzlopak

preemptive blocker

climba03003

I don't see a problem here.
In your request headers Range: bytes=0- means send me a file starting with bytes 0 to the end of the file.
And the response is actually does what it is requested.

Partial content return should be controlled by the request (client), not the response (server) to limit.

ledyba

Thanks for the review. I renamed to highWatermark.

lib/send.js

ledyba · 2024-11-22T08:51:59Z

lib/send.js

@@ -546,6 +551,9 @@ function sendFileDirectly (request, path, stat, options) {

          // Content-Range
          statusCode = 206
+          if (options.maxChunkSize) {
+            ranges[0].end = Math.min(ranges[0].end, ranges[0].start + options.maxChunkSize - 1)


There are some unit tests that success, so I think you are pointing out some edge cases that cause bug. If so, could you tell me the condition or something?

ledyba · 2024-11-22T09:25:34Z

@climba03003 Thanks for the comment.

Here is the latest rfc(9110).

Here is a from 15.3.7. 206 Partial Content:

A server that supports range requests (Section 14) will usually attempt to satisfy all of the requested ranges, since sending less data will likely result in another client request for the remainder. However, a server might want to send only a subset of the data requested for reasons of its own, such as temporary unavailability, cache efficiency, load balancing, etc. Since a 206 response is self-descriptive, the client can still understand a response that only partially satisfies its range request.

So, I think this PR's behavior is, at least, compatible with RFC.

climba03003 · 2024-11-22T10:06:55Z

So, I think this PR's behavior is, at least, compatible with RFC.

Yes, a very rare use-case.

Is a maxChunkSize not basically a highWatermark? Should we name it like that?

highWaterMark is easily to mix with the stream highWaterMark option, especially when we return a stream currently.
From the operator description, it is an option to limit the whole payload size.
I am more toward something like maxReturnSize, maxStreamSize.

...

ledyba · 2024-11-22T11:18:54Z

Thanks for understanding and naming recommendations!

I choiced maxReturnSize, because I thought that maxStreamSize might be confusing or difficult to understand unless you know that the underlying implementation uses fs.createReadStream.

mcollina

lgtm

README.md

Signed-off-by: Frazer Smith <[email protected]>

gurgunday

I think this is a bug fix no? Shouldn't it be on by default?

gurgunday · 2024-11-27T08:45:29Z

Cc @climba03003

climba03003 · 2024-11-27T09:03:17Z

I think this is a bug fix no? Shouldn't it be on by default?

It is not a bugfix, but a feature request. By default, it should return as requested by the client.

gurgunday · 2024-11-27T09:49:05Z

Oh I misunderstood

Currently, this library tends to send entire file when 'acceptRanges': true

I thought the server wasn't respecting the range headers or sth

ledyba · 2024-11-28T04:28:24Z

Hi. I'm the author. I think this is not a bug fix. (Please see the title: it starts with feat: )

#90 (comment)
In a nutshell, even if the client requests all entire file range, the server can limit the response size. It seems allowed in RFC.
So, this feature is disabled by default.
It can be helpful for some rare cases:

However, a server might want to send only a subset of the data requested for reasons of its own, such as temporary unavailability, cache efficiency, load balancing, etc.

ledyba added 6 commits November 16, 2024 13:57

Add maxChunkSize to SendOptions

9749bec

Revert unused change

dd9df28

Fix for the case when maxChunkSize is null

88eef47

Add maxChunkSize related tests

be9c75f

Fix README.md

fb2be90

typo

51927ad

mcollina reviewed Nov 17, 2024

View reviewed changes

types/index.d.ts Outdated Show resolved Hide resolved

Add tsd tests

096c2ca

ledyba commented Nov 17, 2024

View reviewed changes

ledyba requested a review from mcollina November 17, 2024 10:42

Fdawgs requested a review from Copilot November 20, 2024 10:33

Copilot AI reviewed Nov 20, 2024

View reviewed changes

Fdawgs reviewed Nov 20, 2024

View reviewed changes

test/send.1.test.js Outdated Show resolved Hide resolved

Fdawgs reviewed Nov 20, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

Fdawgs reviewed Nov 20, 2024

View reviewed changes

test/utils.js Outdated Show resolved Hide resolved

Fdawgs changed the title ~~Add maxChunkSize option to SendOptions~~ feat: add maxChunkSize option Nov 20, 2024

ledyba and others added 2 commits November 21, 2024 21:38

Apply suggestions from code review

8108c8a

Co-authored-by: Frazer Smith <[email protected]> Signed-off-by: Kaede Fujisaki <[email protected]>

All options must be sorted alphabetically

9e182ac

ledyba commented Nov 21, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

ledyba requested a review from Fdawgs November 21, 2024 12:41

Fdawgs reviewed Nov 21, 2024

View reviewed changes

lib/send.js Outdated Show resolved Hide resolved

Uzlopak reviewed Nov 21, 2024

View reviewed changes

Uzlopak requested changes Nov 21, 2024

View reviewed changes

Uzlopak added 3 commits November 21, 2024 15:13

add test cases

8d98c3a

add more test cases

41454ea

add more test cases

027649c

climba03003 previously requested changes Nov 22, 2024

View reviewed changes

Renamed to highWatarmark

fedeb61

ledyba commented Nov 22, 2024

View reviewed changes

ledyba changed the title ~~feat: add maxChunkSize option~~ feat: add highWatermark option Nov 22, 2024

ledyba requested review from Uzlopak, climba03003 and Fdawgs November 22, 2024 09:25

nits: Fix doc

dbe83ff

Renamed to maxReturnSize

635adcd

ledyba changed the title ~~feat: add highWatermark option~~ feat: add maxReturnSize option Nov 22, 2024

mcollina approved these changes Nov 22, 2024

View reviewed changes

gurgunday approved these changes Nov 22, 2024

View reviewed changes

Fdawgs reviewed Nov 22, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

Update README.md

4055e8f

Signed-off-by: Frazer Smith <[email protected]>

Fdawgs approved these changes Nov 22, 2024

View reviewed changes

gurgunday reviewed Nov 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `maxReturnSize` option #90

feat: add `maxReturnSize` option #90

ledyba commented Nov 16, 2024 •

edited

Loading

ledyba left a comment

Uzlopak commented Nov 17, 2024

ledyba commented Nov 17, 2024 •

edited

Loading

ledyba left a comment

Uzlopak commented Nov 21, 2024

Uzlopak Nov 21, 2024

ledyba Nov 22, 2024

Uzlopak left a comment

climba03003 left a comment •

edited

Loading

ledyba left a comment

ledyba Nov 22, 2024

ledyba commented Nov 22, 2024 •

edited

Loading

climba03003 commented Nov 22, 2024

ledyba commented Nov 22, 2024

mcollina left a comment

gurgunday left a comment

gurgunday commented Nov 27, 2024

climba03003 commented Nov 27, 2024

gurgunday commented Nov 27, 2024

ledyba commented Nov 28, 2024 •

edited

Loading

feat: add maxReturnSize option #90

Are you sure you want to change the base?

feat: add maxReturnSize option #90

Conversation

ledyba commented Nov 16, 2024 • edited Loading

Checklist

ledyba left a comment

Choose a reason for hiding this comment

Uzlopak commented Nov 17, 2024

ledyba commented Nov 17, 2024 • edited Loading

Choose a reason for hiding this comment

ledyba left a comment

Choose a reason for hiding this comment

Uzlopak commented Nov 21, 2024

Uzlopak Nov 21, 2024

Choose a reason for hiding this comment

ledyba Nov 22, 2024

Choose a reason for hiding this comment

Uzlopak left a comment

Choose a reason for hiding this comment

climba03003 left a comment • edited Loading

Choose a reason for hiding this comment

ledyba left a comment

Choose a reason for hiding this comment

ledyba Nov 22, 2024

Choose a reason for hiding this comment

ledyba commented Nov 22, 2024 • edited Loading

climba03003 commented Nov 22, 2024

ledyba commented Nov 22, 2024

mcollina left a comment

Choose a reason for hiding this comment

gurgunday left a comment

Choose a reason for hiding this comment

gurgunday commented Nov 27, 2024

climba03003 commented Nov 27, 2024

gurgunday commented Nov 27, 2024

ledyba commented Nov 28, 2024 • edited Loading

feat: add `maxReturnSize` option #90

feat: add `maxReturnSize` option #90

ledyba commented Nov 16, 2024 •

edited

Loading

ledyba commented Nov 17, 2024 •

edited

Loading

climba03003 left a comment •

edited

Loading

ledyba commented Nov 22, 2024 •

edited

Loading

ledyba commented Nov 28, 2024 •

edited

Loading