[core][compiled-graph] Should we support leaf nodes?

[kevin85421]

Description

#47757 raises an exception if it detects any leaf nodes. However, asking users to add all leaf nodes to MultiOutputNode is not the best UX. Some possible solutions:

1. Launch another thread in the driver process to watch exceptions from leaf nodes.
2. Provide another way for leaf nodes to notify the driver process when they throw exceptions. For example, leaf nodes can talk to a "supervisor" actor on the head node. (input from @rynewang)

@ruisearch42's comment: #47757 (review)

Use case

No response

[rynewang]

I think it's good to let compiled graphs return a Tuple[Output, Exceptions from leaf nodes]. When compiling a graph this means an edge to pass exceptions from each leaf node to the output (driver?).

An actor to receive messages is a good way on the user side to do, e.g. even when there's no exceptions user can send control messages and events to an actor like a message queue. But Ray can't do that just for exception passing.

One side note is on ordinary Ray, if you make a remote call and don't ray.get it, the exception can go away silently (when the obj ref goes out of scope). Maybe we can come up with something to help that case as well.

[stephanie-wang]

In ordinary Ray, you are supposed to get a log message on the driver if an ObjectRef that contained an exception goes out of scope and no one called ray.get on it. See message here.

So regarding @ruisearch42's comment, there is a difference in with the new API, because now when you drop the error's reference, you should get a log message about it.

If the logging isn't working for ordinary Ray or compiled Ray, it is a bug in either case and needs to be fixed.

[stephanie-wang]

Seems like @rkooo567 is already looking into this for compiled Ray in #47689.