Maximize performance for Jobs plugin

[Kaspiman]

Hello! I use Jobs plugin with AMQP driver (RabbitMQ protocol) and move daemon from separate processes to Roadrunner workers.

Roadrunner version: 2.12.3

I expect performance improvement but degradation occurs. I'm trying to keep the performance up by changing the values in the config. So far nothing works, I'm looking for help.

More detailed information about queue and load:

• queue http_requests has rate 60 msg/sec with long http requests (working like a lazy bear)
• 4 performance servers with 12 cores

Old daemons:

• heavy legacy PHP-exemplar with AMQP-read by one message
• http_requests queue daemon executes synchronous http request by curl to external api's, average time is about to 850 ms
• 75 daemons per server * 4 = 300 exemplars

New worker:

• lightweight worker with minimal overhead, Jobs plugin + spiral/roadrunner-jobs usages
• 40-80-100-200 workers on every server = 160-320-400-800 exemplars (experimented with different values)
• http_requests has the same http execution as before, but average time is about 10 seconds

I assume that the workers block the execution of requests and wait for each other. Apparently this does not happen with separately running daemons?

I tried to increase or reduce: the number of workers, prefetch size and pollers number. The best result was shown by changing the prefetch size. But this is still very far from the previous result with separately running daemons.

My rr.yaml config looks like this:

version: '2.7'
rpc:
    listen: 'tcp://127.0.0.1:6001'
server:
    command: 'php public/worker.php'
amqp:
    addr: 'dsn here'
logs:
    level: warn
    mode: production
    output: stdout
jobs:
    num_pollers: 32 # 12 (cores?) 32, 40, etc values
    pool:
        num_workers: 40 # 40, 80, 100, 200 values
        max_jobs: 1000
    consume:
        - http_requests
    pipelines:
        http_requests:
            driver: amqp
            config:
                queue: http_requests
                prefetch: 200                      # 1, 20, 100, 200 etc values
                priority: 10
                durable: true
                exchange_durable: true
                consume_all: true

[rustatian]

Hey @embargo2710 👋🏻

I assume that the workers block the execution of requests and wait for each other.

Workers execute the request in an asynchronous way. This means that if you have 100 workers, 100 messages would be sent to all of them.

I tested locally (using local rabbitmq instances in docker) and if you use a simple worker, you can consume up to 100k messages per second. For sure, that depends on disk IOPS, CPU and many other factors, but 10s average is too much.

General advices here is to profile your code. You can try buggregator to find a slow spots in your code.

num_pollers

You may set the number of concurrent pollers to the CPU_CORES*2. No need to set it to super high numbers, like, e.g. 200.

prefetch

This is a rabbitmq prefetch. That is, how many messages to pull and store locally in the RR to save some network and increase speed (like cache) for the round trip to the rabbitmq.

num_workers + num_jobs

Do not use num_jobs if your application is not leaking memory. It forces the whole pool of workers to be restarted every 1000 requests. Before profiling, try to remove this option.

As for num_workers, it's better to increase the number carefully. You don't need 200 workers, I think that's too much (this is not a problem for RR, but still). Start with 10 and slowly increase the number.

[Kaspiman]

Hello! Sorry for the long silence, I was on vacation.

I continue to study worker performance. There is a hypothesis that workers cannot get enough jobs.

I am guided by the metric named rr_jobs_workers_working and rr_jobs_workers_ready. I expect that with a large task queue, all workers will be working, but this does not happen.

For example, i have this params for jobs plugin - num_workers: 64 and num_pollers: 8. Also I have a simplest worker with a random sleep function (from 50 to 1000 ms).

Next, I fill the queue with tasks and observe the "rr_jobs_workers_working" metric value. it's maximum is 8 like num_pollers: 8. 8 out of 64 workers are working, the rest are ready.
When I increase the "num_pollers" value, the value of the metric also increases. I have prepared a test project to demonstrate this problem and can share it if you want.

As for num_workers, it's better to increase the number carefully. You don't need 200 workers

I have already integrated Roadrunner on a working project. I'm slowly porting the old daemons to the new workers. The old daemons ran in about 450 instances X 4 servers and handled large volumes of tasks of varying duration (from 5ms to 5 min). There are performance issues when migrating heavily loaded daemons. The server cannot cope and cannot distribute the jobs among all the workers.

[rustatian]

Hey @embargo2710 👋🏻

Next, I fill the queue with tasks and observe the "rr_jobs_workers_working" metric value. it's maximum is 8 like num_pollers: 8. 8 out of 64 workers are working, the rest are ready.

Number of pollers should be 1.5x (at least) more than number of workers to correctly load all workers.

[Kaspiman]

I experimented with different values for "num_pollers" and "num_workers=64", the results are:

• num_pollers=64 => max rr_jobs_workers_working=50
• num_pollers=96 => max rr_jobs_workers_working=50
• num_pollers=128 => max rr_jobs_workers_working=50

A little strange. I will prepare a demo project to clarify the problem.

[Kaspiman]

https://github.com/embargo2710/roadrunner-problem-demo

I run this demo at WSL2 on Windows laptop. A similar situation in production on a powerful server with 12 cores.

[Kaspiman]

Today I experimented in production with the number of workers. Initially, there were 128 workers and a large number of tasks, all workers were provided with work. After the increase in the number of workers from 128 to 256, an unusual situation occurred. The number of employed workers dropped sharply to 10-15. This went on for a long time until I rolled back the release. There seems to be some problem with the "Jobs" plugin, what information can I provide?

[rustatian]

Let me try to summarise: you have an empty worker with a random sleep function inside. Then you just bombard the rabbitmq with the messages that RR reads and sends to the worker. And the problem is that you have too big latency? (how do you measure it? )
Keep in mind that checking state may return state in that particular millisecond, and state is valid for that millisecond. Also, getting state too often will slow down the execution (stop the world).

[Kaspiman]

Let me summarize.

There are performance issues in production under load. I've been watching the production metrics and found that the workers often don't work. The number of workers worked is often less than the maximum possible. I'm already checked ./rr workers -i -c command and workers don't really work.

To confirm this, I made a simple project that: "you have an empty worker with a random sleep function inside....". Yes, this is confirmed if you believe the "rr_jobs_workers_working" metric and the "./rr workers" command.

[rustatian]

The number of workers worked is often less than the maximum possible

All of these commands are informational; when you see the information about the worker's status, the status may already have changed. In the next millisecond after you get the state. You shouldn't use this as a basis for your theory, nor should you use a random sleep inside your test.

To verify your assumption, use sleep for 10 seconds, send 10 requests (with number of workers equals to 10) and after that, check the state. All workers should be in the working state. If you wanted to perform a test under load, use counter inside your worker (or execs from the ./rr workers command) and check the number requests you sent with the actual handled requests.

[Kaspiman]

Hello! I continued to study the problem and saw interesting results.

I tried changing the sleep time to 10 seconds in worker.

With 64 workers and 64 tasks in the queue, I only got 50 active workers. Why 50? Because I configured 5 queues with prefetch parameter equal to 10. 5 х 10 = 50. Bingo!

When I increased the prefetch parameter to 100, when I repeated the experiment, I got 64 active workers. Yeah!

I experimented with the prefetch parameter, for example, a value of 7 for all queues allows only 35 workers to be involved.

It turns out that the prefetch parameter for each queue affects the overall throughput. It seems that the distribution of new tasks is blocked until the old batch is processed. As a result, a situation may arise when workers do not receive enough tasks.

[rustatian]

Hm, I thought you were using prefetch 200 as in the original post (prefetch: 200)... Although, good catch 👍🏻
Just FYI, what does that mean: prefetch is how many messages RR reads and caches from the AMQP before ACKing them. It seems that in your case your prefetch number was small, but the workers number was large. In this case, RR read the messages but waited for the ACK to read more (leaving more workers not loaded). When you specified a larger prefetch number, RR was able to read more messages before ACK-ing them. Now you may play with these parameters to achieve the optimal for you performance 👍🏻

[Kaspiman]

The documentation needs a clearer explanation of the num_pollers, num_workers and prefetch parameters.

The documentation suggests that the value num_pollers should be twice the number of server cores and these values are related.

per second within 64 pollers (on a 32 core CPU).

However, it turns out that the value ofnum_pollersmust be at least one and a half times greater than the value of num_workers. And this still requires verification and selection.

Moreover, the prefetch value for queues can create problems and level out the carefully chosen parameters from the previous paragraph. It doesn't matter how many workers you have ready, if the sum of the prefetch parameters is not enough. Strange, isn't it? I think that the driver should poll the RabbitMQ and distribute tasks as efficiently as possible, excluding ready (idle) state.

It seems that there is some kind of implicit mechanics of getting tasks in batches. Batch block pooling, the server is forced to wait for the execution of all tasks from this batch.

[rustatian]

However, it turns out that the value ofnum_pollersmust be at least one and a half times greater than the value of num_workers. And this still requires verification and selection.

This depends on what your tasks are. How fast your code runs inside a worker, and should be chosen based on your application.

Strange, isn't it?

That is not strange, because prefech is well-known parameter of many queues (sqs, rabbitmq, kafka, etc). Literally the first statement:

AMQP 0-9-1 specifies the basic.qos method to make it possible to [limit the number of unacknowledged messages](https://www.rabbitmq.com/confirms.html) on a channel (or connection) when consuming (aka "prefetch count")

I think that the driver should poll the server and distribute tasks as efficiently as possible, excluding ready (idle) state.
It seems that there is some kind of implicit mechanics of getting tasks in batches. Batch block execution, the server is forced to wait for the execution of all tasks from this batch.

I don't agree with this statement. You don't seem to read the RabbitMQ documentation (sorry for pointing on that). Batch blocks execution by design, because you have to ACK all messages. If you just want to read everything without acknowledging - you may use AutoAck option or multiply ack.

However, I see, that I need to include more verbose expanation about that in the docs, thanks for pointing me on that 👍🏻

[rustatian]

I've updated the docs and inclue more clear info about num_pollers and prefetch with the links to the rabbitmq docs.