phrase_from_stream/2 nontermination (Stream from http_open/3) #2437

rodrigosetti · 2024-07-03T21:26:48Z

rodrigosetti
Jul 3, 2024

I'm building an application that has a HTTP Client component.

It would be nice to streaming DCG to parse the data but phrase_from_stream/2 does not terminate when using the instantiated stream from http_open/3.

Steps to reproduce in top-level scryer-prolog:

(Started a simple local HTTP server using python -m http.server 8000).

?- use_module(library(charsio)).
   true.
?- use_module(library(dcgs)).
   true.
?- use_module(library(http/http_open)).
   true.

% reading the entire stream before parsing works as expected:
?- http_open("http://127.0.0.1:8000", Stream, []), get_n_chars(Stream, N, Chars), phrase(seq(Data), Chars).
   Stream = '$stream'(0x600000636908), N = 726, Chars = "<!DOCTYPE HTML>\n<ht ...", Data = "<!DOCTYPE HTML>\n<ht ...".

?- use_module(library(pio)).
   true.

% phrase_from_stream/2 does not terminate in this case:
?- http_open("http://127.0.0.1:8000", Stream, []), phrase_from_stream(seq(Data), Stream).
% nontermination

This seems to happen with other grammar rules as well (I'm originally trying to parse JSON data), but I'm able to reproduce with any rule. I also attempted to unify with content size and set Connection: close header (using options) without success.

Similar code using SWI-Prolog works as expected:

?- use_module(library(http/http_open)).
true.
?- use_module(library(dcg/basics)).
true.
?- use_module(library(pio)).
true.
?- http_open("http://127.0.0.1:8000", Stream, []), phrase_from_stream(string(Data), Stream).
Stream = <stream>(0x600002954400,0x600002974b00),
Data = [60, 33, 68, 79, 67, 84, 89, 80, 69|...] ;
false.

I was able to isolate the issue further. it looks like at_end_of_stream/1 is failing when it should succeed.

Scryer:

?- http_open("http://127.0.0.1:8000", S, []), get_n_chars(S, N, Chars), at_end_of_stream(S).
   false.
?- http_open("http://127.0.0.1:8000", S, []), get_n_chars(S, N, Chars), stream_property(S, end_of_stream(X)).
   S = '$stream'(0x600002108888), N = 724, Chars = "<!DOCTYPE HTML>\n<ht ...", X = not.

SWI:

?- http_open("http://127.0.0.1:8000", Stream, []), read_string(Stream, N, String), at_end_of_stream(Stream).
Stream = <stream>(0x600002b38900,0x600002b38300),
N = 724,
String = "<!DOCTYPE HTML>..."

I debugged the library pio, and it looks like it gets stuck into a infinite loop in the internal (tail) recursive buffer_prepare_for_n/5 predicate - it uses at_end_of_stream/1 as the recursive base condition but it fails (like above) when it should succeed when it has read all the characters.

Answered by aarroyoc

Jul 6, 2024

The implementation of a method in the streams code was missing, as a result it always reported that there were more bytes to read even if it wasn't the case. It's fixed here: #2442

View full answer

haijinSk · 2024-07-04T09:52:35Z

haijinSk
Jul 4, 2024

I think, it's a bug. And I've encountered some crashes, for example this I can easily reproduce:

:- use_module(library(http/http_open)).

crash :- http_open("", _, []).

?- crash.
thread 'main' panicked at src/machine/system_calls.rs:4301:55:
called `Result::unwrap()` on an `Err` value: RelativeUrlWithoutBase
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And another example of "abnormal termination" using Ctrl-D:

:- use_module(library(http/http_open)).

okForNow :- http_open("https://www.scryer.pl/", _, []).

?- okForNow.
   true.
?- 
Segmentation fault (core dumped)

And even without using Ctrl+D, I can "compose" them:

:- use_module(library(http/http_open)).

crash :- http_open("", _, []).

okForNow :- http_open("https://www.scryer.pl/", _, []).

?- okForNow, crash.
thread 'main' panicked at src/machine/system_calls.rs:4301:55:
called `Result::unwrap()` on an `Err` value: RelativeUrlWithoutBase
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Segmentation fault (core dumped)

2 replies

rodrigosetti Jul 4, 2024
Author

I was not able to reproduce the okForNow error (scryer-prolog --version = df41eb1); but the crash example seems like a separate issue (should throw a domain_error for malformed URLs?).

haijinSk Jul 4, 2024

My version is from May 27, 2024, compiled using cargo build --release, 9672045

$ scryer-prolog -f
?- use_module(library(http/http_open)).
   true.
?- http_open("https://www.scryer.pl/", _, []).
   true.
?- % Here I'm pressing Ctrl-D (on Ubuntu) to go back to the OS command line
Segmentation fault (core dumped)

Yes, my observations are very informal, associative... perhaps not directly related and not directly useful for this particular case. But, for the record... notes for the maintainer/creator and contributors.... Maybe...

In other words, I'm not the right Prolog person here, but, yes, I think, the "malformed URL" should not terminate abnormally/crash... but throw some related/informative error, like the domain_error you propose. Only a beginner, but I like Scryer; thank you and thank you for reporting this issue.

haijinSk · 2024-07-04T14:40:26Z

haijinSk
Jul 4, 2024

Non-termination: 7x true., 8th is ("a hard") non-termination:

?- http_open("https://www.scryer.pl/", _, []).
   true.
?- http_open("https://www.scryer.pl/", _, []).
   true.
?- http_open("https://www.scryer.pl/", _, []).
   true.
?- http_open("https://www.scryer.pl/", _, []).
   true.
?- http_open("https://www.scryer.pl/", _, []).
   true.
?- http_open("https://www.scryer.pl/", _, []).
   true.
?- http_open("https://www.scryer.pl/", _, []).
   true.
?- http_open("https://www.scryer.pl/", _, []).
% non-termination

1 reply

rodrigosetti Jul 4, 2024
Author

I get non-termination reliably on the 7th attempt.

I also attempted to close the stream (maybe it's a keep-alive connection pool deadlock), but still doesn't terminate:

?- use_module(library(http/http_open)).
   true.
?- http_open("https://www.scryer.pl/", S, []), close(S, [force(true)]).
   S = '$dropped_value'.
?- http_open("https://www.scryer.pl/", S, []), close(S, [force(true)]).
   S = '$dropped_value'.
?- http_open("https://www.scryer.pl/", S, []), close(S, [force(true)]).
   S = '$dropped_value'.
?- http_open("https://www.scryer.pl/", S, []), close(S, [force(true)]).
   S = '$dropped_value'.
?- http_open("https://www.scryer.pl/", S, []), close(S, [force(true)]).
   S = '$dropped_value'.
?- http_open("https://www.scryer.pl/", S, []), close(S, [force(true)]).
   S = '$dropped_value'.
?- http_open("https://www.scryer.pl/", S, []), close(S, [force(true)]).
% non-termination

haijinSk · 2024-07-04T19:30:15Z

haijinSk
Jul 4, 2024

Again, maybe not directly related, and again a "Ctrl-D example": (from my naive point of view) unexpected unexpected_end_of_file:

?- phrase_from_stream(seq(Str),user_input).
qwertzuiop
% Here (after "Enter/newline") I'm pressing Ctrl-D to end my input...
   error(syntax_error(unexpected_end_of_file),'$get_n_chars'/3).

And using Ctrl+C:

?- phrase_from_stream(seq(Str),user_input).
qwertzuiop
% Here (after "Enter/newline") I'm pressing Ctrl-C...
   error(syntax_error(input_output_error),'$get_n_chars'/3).

0 replies

triska · 2024-07-06T10:43:51Z

triska
Jul 6, 2024

~~@bakaq: I would greatly appreciate if you could take a look at this: Is this an issue that can be addressed in phrase_from_stream/2, or rather in the underlying functionality?~~ Update: @aarroyoc has addressed the underlying issue in #2442, thank you a lot!

0 replies

aarroyoc · 2024-07-06T14:07:50Z

aarroyoc
Jul 6, 2024
Sponsor

The implementation of a method in the streams code was missing, as a result it always reported that there were more bytes to read even if it wasn't the case. It's fixed here: #2442

0 replies

haijinSk · 2024-07-08T07:26:12Z

haijinSk
Jul 8, 2024

@aarroyoc Thank you for your knowledge, skills, talent, time and work, thank you very much for fixing the phrase_from_stream/2 non-termination in this context of http_open/3.

I don't want to write more here, but I think, I have to add a margin note, that my (not directly related to the use case) "bad examples" continue working, as a separate issue.

2 replies

aarroyoc Jul 8, 2024
Sponsor

Yes, but those examples are showing a different issue. I think @razetime also reported something similar some time ago when lots of http calls are issued but I didn't investigate it too much.

haijinSk Jul 8, 2024

Yes, my examples are a separate thing (use case, or "affordance" of Scryer). I'm not complaining here, I thank you. I wanted to add my note for the record only (for a future reader), so to speak.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

phrase_from_stream/2 nontermination (Stream from http_open/3) #2437

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

phrase_from_stream/2 nontermination (Stream from http_open/3) #2437

Replies: 6 comments · 5 replies

rodrigosetti Jul 4, 2024 Author

rodrigosetti Jul 4, 2024 Author

aarroyoc Jul 6, 2024 Sponsor

aarroyoc Jul 8, 2024 Sponsor

Replies: 6 comments 5 replies

rodrigosetti Jul 4, 2024
Author

rodrigosetti Jul 4, 2024
Author

aarroyoc
Jul 6, 2024
Sponsor

aarroyoc Jul 8, 2024
Sponsor