The pipeline queue can grow over time when a client maintains a continuous connection

[wckao]

Describe the bug
The pipeline queue can grow over time when a client maintains a continuous connection to a server and sends frequent/infrequent requests. This occurs even if the requests complete quickly and long lasts even after requests. However, once the client disconnects, the pipeline queue is promptly cleared, and its length returns to zero.

Expected behavior
The queue length should not grow with time.

Screenshots

At 15:00, we disconnected the client that is idle at the time.

Environment (please complete the following information):

• OS: [ubuntu 24.04]
• Kernel: # Command: 6.8.0-1018-gcp MULTI / EXEC basics crash the server #20-Ubuntu SMP Thu Nov 7 17:04:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Containerized?: Docker Compose
• Dragonfly Version: docker.dragonflydb.io/dragonflydb/dragonfly:v1.25.2

[BagritsevichStepan]

Hi @wckao. Thank you for reporting the bug

[BagritsevichStepan]

Please provide more information on how you use the pipeline, how many connections you have, and how you send data.

Because it's not entirely clear why this would be considered a bug. Typically, Prometheus aggregates all connections together and displays their total size. Therefore, these spikes might represent data transmitted over other connections.

[wckao]

The problems are not the spikes around 12:00. It is that if we disconnect one single idle client at 15:00, the length of the pipeline drop from 10k to 0. We use dragonfly(Redis) as our Celery result backend. And uses celerybeat to schedule tasks. The persistent connection to the dragonfly server are from the celery beat server and we can see the length of queue grows with time. Once we stop the celery beat process, the length of queue drops back to 0. What celery beat does is to periodically put some tasks(messages) into rabbitmq. So it shouldn't be anything complex. I did found that when Celery uses Redis as result backend, it uses pipeline but nothing complex (ref: https://github.com/celery/celery/blob/main/celery/backends/redis.py)

[romange]

can you please run "client list" during the period when the queue length is high and post here the results?