Best way to measure percentage of time spent doing IO

[AlliBalliBaba]

I've been doing some testing with automatically scaling threads at runtime. This generally works pretty well by just creating a new thread if a request has been stalled for long enough.

The only problem with that approach is that it can not distinguish if requests are stalled because threads are doing IO or because all CPU cores are already busy. If CPU's are busy, we obviously don't want to spawn additional threads.

My question then would be: What's the best (cross platform compatible) method to measure how much CPU is being used? Does prometheus expose this metric somehow reliably (@withinboredom)? Or would it be better to manually calculate the cpu time of each thread via clock()?

[withinboredom]

Maybe we should do something like this: https://www.usenix.org/conference/nsdi24/presentation/wydrowski?

Basically, we can know the average rif and duration. Scale accordingly to maximize rif and minimize duration.

[withinboredom]

We are loadbalancing between servers -- er, threads in a "first available" kinda way. In our case though, that is simply the most easiest way to load balance them: channels and first-available. If we wanted to change that, we would probably select the thread on the least active core or least recently active core -- in the paper these are more like hot servers vs. cold servers. Adding more threads on the same hardware may increase cpu utilization, etc, etc.

Anyway, that's probably more complex than it needs to be, but I was inspired by the paper when I saw your message. Instead of monitoring cpu load, instead we monitor rif and latency to determine scaling decisions. Except since we are dealing with a single server and a thread can only handle one request at a time, we really only need to know the number of "busy" servers processing a request in relation to the number of threads we have + latency.

Basically:

1. Start with a minimum number of threads (num_threads).
2. Define thresholds
   • Idle threshold.
   • Scale threshold.
3. On incoming request:
   1. send to channel
   2. start latency timer
   3. mark thread as busy in metrics
4. After request
   1. end latency timer and track in metrics
   2. mark thread as free in metrics
5. On timer:
   1. check len(request_chan) -- assuming it is buffered -- if len >= 1, scale up by len (e.g., all threads are busy and we need more threads) and return. Fun fact, this is basically how I determine the limits of my servers. We might want to make it at least 2-3 times in a row to avoid hysteresis.
   2. if total threads is greater than or equal to max threads, return
   3. if 90% latency is rising or busy threads is over the threshold, scale up.
   4. if total threads is less than or equal to num_threads, return.
   5. if number of idle threads is less than idle threshold, scale down.

graph TD
    A[Incoming Request] -->|Add to Channel| B[Thread Pool]
    B -->|Mark Thread Busy| C[Process Request]
    C -->|Complete| D[Mark Thread Idle]

    E[Scaling Timer] -->|Check len of request_chan| F{len > 1?}
    F -->|Yes| G[Scale Up Threads]
    F -->|No| H{Latency Rising?}
    H -->|Yes| G
    H -->|No| I{Idle Threads < Threshold?}
    I -->|Yes| J[Scale Down Threads]
    I -->|No| K[No Action]

    G -->|Spawn Threads| B
    J -->|Terminate Threads| B

Loading

This means I can set my scaling threshold to 0.8 and num_threads = 3 and max_threads = 10 and once 2 out of 3 threads are being utilized, another thread will be added. Additionally, if my tail latency starts to increase, new threads will be spun up to handle more requests until it stabilizes again. The latency is likely increased by i/o (such as databases) and by scaling here, we can beat a traditional autoscaler in bringing more resources online.

Going by cpu metrics introduces an issue in that threads waiting on i/o (eg, a database) will be invisible, and threads doing processing may cause unnecessary scaling.

That was my idea at the time anyway. I see you have a PR now and I'll take a look.

[AlliBalliBaba]

I also like the idea of both doing the up- and down-scaling in a loop. Finding the best algorithm will probably include a bit of trial and error, I'll create a k6 test that covers some of the most likely scenarios to compare.
In my PR I additionally ended up measuring CPU, since latency alone doesn't necessarily tell us that the server is doing IO. I also added an admin API endpoint to debug and test threads. That info could also be useful in metrics (or could be migrated there?), but I'm not familiar enough with the current prometheus metrics to know if that's a good choice.

[dunglas]

Very exciting!

[withinboredom]

In thinking about it a bit more, I would expect CPU usage and latency to behave counter-intuitively. High CPU usage might mean that scaling down is the right answer -- too many threads sharing a limited set of CPUs, or it might mean that scaling up is the right answer, and making that decision point will be hard or even impossible.

Honestly, after thinking about it, I'd be willing to bet we can ignore those two metrics. In other words: how many requests are in-flight and how many requests can we currently handle? We can attempt to achieve a ratio after accounting for any currently enqueued requests waiting for a thread, for example:

Desired ratio: 120% of current requests in-flight
Current requests: 20
Desired threads: 24

Current requests: 0
Desired threads: 0

Most devs have a pretty good idea what their traffic spikes look like, but ultimately, you also have to constrain it because it is running on a limited amount of resources. Thus I expect most people will be using this to scale to zero for denser serving of low-traffic sites, and when there is unpredictable traffic. I suspect high-traffic and high-performance sites would get the same advice given for fpm: set a static number of workers.

[AlliBalliBaba]

I think we need the downward pressure from high CPU usage for the best efficiency. To which degree that makes a difference, I still have to test.

I agree that if you know exactly how your traffic looks like and have all latencies under control, it's probably best to set a fixed amount of threads. I disagree though that most devs fine-tune their servers that way, I think very few people actually bother (it's very hard to get right). High traffic sites usually have ways to scale horizontally and should also benefit from better latencies under pressure.