Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unidentified bug causing short read of chunked data #108

Open
ghost opened this issue Dec 9, 2020 · 10 comments
Open

unidentified bug causing short read of chunked data #108

ghost opened this issue Dec 9, 2020 · 10 comments

Comments

@ghost
Copy link

ghost commented Dec 9, 2020

The problem: (drakma:http-request "http://digidb.io/digimon-list/") got an input-chunking-unexpected-end-of-file error.

Details:
After I poking around the code, I find that it is indeed a short read of (read-sequence input-buffer inner-stream :start 0 :end chunk-size) in chunga::fill-buffer. (https://github.com/edicl/chunga/blob/cb333cdba178e99b03fa60e2caa8c5d3654201d8/input.lisp#L145)

The number of bytes read is consistent most of time, but sometimes it might vary.

The bytes read is valid, and another (read-sequence big-buffer chunga::inner-stream) in the debugger can successfully read the following content left in the stream.

How to reproduce:
This error is only reproducible on Android, where only ecl is available. Can both be reproduced in cl-repl app (which is available in Play Store), and ecl compiled in Termux (a terminal for android). This error is not reproducible by curl or other tools on the same device.

This error is only observed with remote server - but I only tested with "http://digidb.io/digimon-list/" so I am uncertain about other servers. On the other hand, replaying the response on localhost does not reproduce the error.

This error is only reproducible when using drakma:http-request without :want-stream t. Even if use want-stream,

  1. use read-sequence directly with the returned stream
  2. use drakma:read-body on returned http-stream, or
  3. use drakma::%read-body on the returned stream
    does not reproduce the error (and the request is successful).
@stassats
Copy link
Member

stassats commented Dec 9, 2020

The android requirement makes this harder to diagnose (don't have an android device).

@ghost
Copy link
Author

ghost commented Dec 9, 2020

Even though I have one, the diagnose process so far is really painful.

I am currently run out of idea. I cannot think a single reason why drakma:read-body with :want-stream work, but directly use of drakma:http-request don't.

Do you have any suggestion how can I further investigate this problem?

Thanks

edit: wording

@stassats
Copy link
Member

stassats commented Dec 9, 2020

Why does it do only one read-sequence and then give up?

@ghost
Copy link
Author

ghost commented Dec 9, 2020 via email

@stassats
Copy link
Member

stassats commented Dec 9, 2020

read() is not guaranteed to read everything up to EOF. I'll make change to chunga for you to try.

@stassats
Copy link
Member

stassats commented Dec 9, 2020

Although I'm not sure how READ-SEQUENCE is implemented on ECL.

@stassats
Copy link
Member

stassats commented Dec 9, 2020

Can you try edicl/chunga@e83c804

And I hope I got the logic right, as I don't have anything to test it on.

@ghost
Copy link
Author

ghost commented Dec 10, 2020 via email

@stassats
Copy link
Member

I'm not sure it's the right fix, maybe ECL implements read-sequence incorrectly. CLHS says that read-sequence returns fewer elements only upon reaching EOF.

@ghost
Copy link
Author

ghost commented Dec 10, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant