What's the proper way to implement syscalls that involve blocking host function calls or interactions between emulated threads?

[willowypanda]

I am not against the current design of using gevent greenlet to emulate threads in principle, but I am not sure what's the best way to implement syscalls that involve interactions between multiple emulated threads or some blocking host function calls.

A case I would like to highlight is the socket recv system call which happens to involve two major limitations or problems of the current implementation.
link: https://github.com/qilingframework/qiling/blob/d921cb21f71fa8b3be58611294db2e3a968e05d1/qiling/os/posix/syscall/socket.py#LL672C5-L672C5

in the current implementation, I believe an emulated thread may be blocked by the recv system call and never yield control to other emulated threads. In a real system, other threads should get the chance to run when some threads are blocked by certain actions.

So problem one: We need to find a way to make blocking host function calls cooperative (in gevent terms) or use the cooperative version provided by gevent (in this particular case, use gevent.socket as the "backend"). However, we cannot just yield the control in the syscall callback function because it's in the unicorn context according to this comment.

One way I can think of is to start another gevent greenlet and use this greenlet to wait for the return of the blocking host function call, let's call this greenlet "wait_greenlet". A gevent Event e will be created and the sched_cb function would call e.wait() to wait for the wait_greenlet to call e.set() after the blocking host function call returns. And the blocked emulated thread will be unblocked and get the chance to be rescheduled again after e.wait() returns.

However, I don't know if the solution I described above is the best solution, and it also comes with the second problem: how to set the return value of the system call according the result of the blocking host function call? What I can think of, is to retrieve the data from the wait_greenlet after e.wait() returns in the sched_cb function, and then manually set the corresponding register value (OS and Arch dependent) and write to the memory (if needed) according to the retrieved data. But again, I don't know if this is the best solution or if there's a more elegant way.

Another question I would like to ask is that if I call ql.emu_stop() in the syscall callback function, when this emulated thread is rescheduled and emu_start is being called, is it guaranteed that the first instruction to be executed will be the instruction next to the "int xx" instruction that triggers the system call? I am not very familiar with the Unicorn Engine and I find the documentation for Unicorn Engine is a little bit lacking.

[willowypanda]

This post was started in the qiling telegram group and later copied / migrated to here as @wtdcode suggested.
@wtdcode also asked in the telegram group:

Regarding recv, why not just call gevent equivalent functions and give up control in sched_cb?

My reply / follow-up question:
I think in this particular case, we can use the gevent equivalent function. Since we should not give up control in the Unicorn context, I believe we need to call the corresponding gevent function inside sched_cb? The problem remaining is how to set the return value for the system call and write the received data into the buffer inside the emulated memory? Can we just manually change the register value and modify the memory in sched_cb after the corresponding gevent function returns? Is this the proper way to do it?

Also, in case of functions without a gevent equivalent, is starting a "wait greenlet" the best option?

[wtdcode]

Also, in case of functions without a gevent equivalent, is starting a "wait greenlet" the best option?

No, you have to create a future somehow.

Can we just manually change the register value and modify the memory in sched_cb after the corresponding gevent function returns? Is this the proper way to do it?

Yes.

[willowypanda]

Thanks for your reply. By future, do you mean the abstract concept of future or some specific future implementation in a specific library like concurrent.futures.Future (which uses threading.Condition.wait() under the hood and can probably work with gevent monkey-patched version of threading). What's on my mind is to use gevent.event.AsyncResult and do something like

def sched_cb():
    r = AsyncResult()
    gevent.spawn(lambda: blocking_host_call()).link(r)
    try:
        r.get()
        ql.os.set_syscall_return(0)
    execpt:
        ql.os.set_syscall_return(-1)

We can create a new function wrapper somewhere if this becomes a very common pattern.