Worker Thread Attrition

[J-W-C]

Some months ago I started trying to figure out why my Internet-facing CherryPy web server would, after a few weeks, stop responding to HTTPS requests.

The failures seemed to be associated with security researchers / miscreants probing the TLS implementation, evidenced by bursts of SSL exceptions - NO_SHARED_CIPHER, WRONG_VERSION_NUMBER, etc.

Another thing I noticed was these episodes would sometimes result in the termination of a cheroot worker thread, which leads to my first question:

Q: If a cheroot workers WorkerThread dies does it ever get replaced? [The only uses of self.min I see in class ThreadPool are in start() and shrink()]

Looking a logged stack trace, I think I'm seeing the following code path leading to an unhandled exception in a worker thread:

• HTTPConnection.communicate() calls req.parse_request() at cheroot/server.py:1277
• parse_request() throws a socket.error
• The handler goes on to call self._conditional_error(), either at server.py:1300 or :1306
• _conditional_error() calls req.simple_response() and it throws an exception which is neither errors.FatalSSLAlert nor errors.NoSSLError [maybe SSL: UNKNOWN_STATE]
• The unhandled exception pops all the way up to WorkerThread.run() and takes out the thread

Q: Does that make sense?

[J-W-C]

I do think I have an issue here.

I've been running with local changes to cheroot/server.py function HTTPConnection:_conditional_error() to catch and log what would otherwise be uncaught exceptions coming out of its invocation of req.simple_response(response) for a while now and have seen at least one occurrence of this happening on my Internet-facing CherryPy server.

Further, I confirmed (by contriving to lose worker threads with my own exceptions raised in HTTPConnection:communicate()) that, after losing a certain number of worker threads, my server stops responding to HTTPS requests (but will still respond to HTTP),that being the symptom that originally got me looking at this.

I can create a pull request to address this but I'd like some guidance as to how you'd like it done--just more exception catching in exception handling code, or something further up the call stack?

[webknjaz]

I think there's a number of open issues and PRs with similar symptoms. Have you checked them? If there's something that already addresses the problem, maybe it needs addressing some reviews.

[J-W-C]

OK, good. I looked at some before; I'll make a more systematic review now.

[webknjaz]

In #649, I fixed the thread crashes. It's available in v11.0.0b0. A stable release is upcoming but that beta is enough for testing.

I missed the original question about replacing the threads and the answer is that Cheroot doesn't do it by itself. However, there's https://pypi.org/p/cherrypy-dynpool that is targeting CherryPy so it should be possible to pull some of its internals to use for the pool management. #190 brought this up but the discussion went stale and didn't get traction. Somebody would have to lead the effort if we were to build that into the library.

[J-W-C]

Thanks! We'll give that version a try.

(Not too concerned about a dynamic pool now as my servers have been faultless for quite a while just by addressing the exceptions coming out of _conditional_error())