dnsdist handle more than 1 million concurrent connection

[ahmedhkhalil]

Problem:
We are experiencing over 1 million concurrent requests across all VMs, but despite this high query load:

The CPU and memory utilization remain very low.
The maximum QPS (Queries Per Second) for dnsdist is around 10k.
Question:
How can we:

Increase the QPS (queries per second) handled by dnsdist?
Improve the utilization of CPU resources to handle more requests faster and ensure the system can handle this high traffic load effectively?
Environment:
dnsdist setup for load balancing across multiple VMs.
PowerDNS configured as the backend.
Public DNS authoritative server.
We are seeing low CPU/memory utilization, but the DNS query rate is capped at 10k QPS.
What We've Tried:
We have checked and optimized configurations like buffer sizes, worker threads, and query timeouts.
The system scales well vertically in terms of available resources, but QPS still remains limited to ~10k.
Tried adjusting various dnsdist performance and load-balancing settings.
What We're Looking For:
Suggestions for tuning dnsdist to handle more QPS.
CPU optimizations to ensure that we are fully utilizing available CPU cores.
Best practices for configuring dnsdist in high-traffic environments to ensure scalability and improved throughput.

[rgacogne]

Hi,

You are not giving us a lot to work with, we are going to need much more information to be able to help you:

• dnsdist version
• OS version
• hardware details (CPU, memory, network, ...)
• current dnsdist configuration

10k QPS is very, very low, to be clear, dnsdist can handle a lot more than that on decent hardware.

[paddg]

Hi, Can you rule out the Powerdns Auth and it's (Database?) backend as a bottleneck? Winfried Am 14. November 2024 11:02:02 MEZ schrieb ahmedhkhalil ***@***.***>:

…

Problem: We are experiencing over 1 million concurrent requests across all VMs, but despite this high query load: The CPU and memory utilization remain very low. The maximum QPS (Queries Per Second) for dnsdist is around 10k. Question: How can we: Increase the QPS (queries per second) handled by dnsdist? Improve the utilization of CPU resources to handle more requests faster and ensure the system can handle this high traffic load effectively? Environment: dnsdist setup for load balancing across multiple VMs. PowerDNS configured as the backend. Public DNS authoritative server. We are seeing low CPU/memory utilization, but the DNS query rate is capped at 10k QPS. What We've Tried: We have checked and optimized configurations like buffer sizes, worker threads, and query timeouts. The system scales well vertically in terms of available resources, but QPS still remains limited to ~10k. Tried adjusting various dnsdist performance and load-balancing settings. What We're Looking For: Suggestions for tuning dnsdist to handle more QPS. CPU optimizations to ensure that we are fully utilizing available CPU cores. Best practices for configuring dnsdist in high-traffic environments to ensure scalability and improved throughput. -- Reply to this email directly or view it on GitHub: #14845 You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[ahmedhkhalil]

Database is scale out and not bottleneck it's mysql
OS Version for Servers : ubuntu 22.04
hardware : 16 / 48 G RAM
current dnsdist : i can't show them exactly but we are using cache and NS as apool and some blocking roles related to NXdomain

[rgacogne]

Would you mind running a pdnsutil bench-db in the PowerDNS Auth server just to rule it out?

[ahmedhkhalil]

ov 14 14:12:12 Lookup for 'ANY' of '1908609430.powerdns.com' within zoneID -1
Nov 14 14:12:12 Found no authoritative zone for '1908609430.powerdns.com' and/or id -1
Nov 14 14:12:12 Query 94745787559552: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name=?
Nov 14 14:12:12 Query 94745787559552: 289 us to execute
Nov 14 14:12:12 Query 94745787559552: 294 us total to last row
1.54967 millisecond/lookup
Retrieved 0 records, did 10000 queries which should have no match
Packet cache reports: 0 hits (should be 0) and 0 misses

[rgacogne]

1.54967 millisecond/lookup

While not bad, it means that this backend can only do 645 fully uncached lookups per second. In practice it will be better than that thanks to the record and packet caches, but you need to be aware of this potential bottleneck.

[rgacogne]

To put this into perspective, this is what I get from a LMDB backend:

0.005831 millisecond/lookup

which roughly translates to ~170k fully uncached lookups per second.

[ahmedhkhalil]

how to enhance , should i increase mysql specs or use other backend , or apply some kind of indexing

[ahmedhkhalil]

could i migrate to LMDB from mysql

[rgacogne]

I don't know enough about your use-case to tell you for sure, but if the LMDB backend supports the features you need (see https://doc.powerdns.com/authoritative/backends/index.html) then yes: https://doc.powerdns.com/authoritative/migration.html#migrating-data-from-one-backend-to-another-backend

[ahmedhkhalil]

how can i increase number of threads , as i suspect issue not only in database because i test database from several servers simultaneals and give me same response time

[klaus-nicat]

First, you need to debug what kind of traffic is your problem. If it is high volume of "normal" traffic, then your system should be fast due to caching. If the queries are random subdomain attacks, the caches will not help and every single query will trigger one or more DB queries. And this is the bottleneck.
If you want to answer 1mio random subdomain queries you can not use a SQL-based backend. Either change the Power-DNS backend to LMDB, or change the nameserver. The decision is a compromise: dnsdist + powerdns gives you great flexibility and means to interact with DNS requests, other software like Knot/NSD is unflexible but faster.

[klaus-nicat]

For customers which have 24x7 random subdomain attacks we opted for both: We replaced dnsdist with Knot. Knot gets the zone data (AXFR) from PowerDNS. So we can still use PowerDNS for zone provisioning and doing plenty of tricks, but on the public side we have Knot answering the queries.