DRSS : The Hidden Performance Lever in the FastLinQ ESXi Driver
In a restaurant, does one waiter take all the orders while others wait? Obviously not. All those available share the work.
In the early days of computing, only one CPU was available to perform processing. Now we have systems having multiple CPU cores. Undoubtedly, one core should not do all the processing while others stay idle. And today's software should be capable of implementing this. In the servers operating in data centers, the Network Interface Card (NIC) is the component that sends and receives the network traffic, in terms of packets. The packets need to be processed in the server. In the days when only one CPU was available, there was no question of which CPU would process the packets. Today, in a multi-core system, the software is expected to facilitate an even distribution of the workload. This is the situation we faced with the FastLinQ ESXi device driver before DRSS (Default queue Receive side scaling) was enabled by default.
"The default queue is inherently the busiest queue on the NIC — not because netqueues are idle by design, but because a large portion of traffic is structurally unmatched."
Understanding the Default Queue and NetQueues
ESXi uses a feature called NetQueue to distribute incoming network traffic across multiple receive (RX) queues on a NIC. Each queue is associated with a CPU core, allowing parallel processing of network traffic. This improves throughput and reduces latency. To understand the problem, it is important to first understand how the default queue and netqueues work together.
The default queue is always present and operates in promiscuous mode — it receives all incoming packets that do not match any existing netqueue filter. This includes broadcast traffic (ARP, DHCP), multicast traffic, control plane traffic, and any unicast traffic for which no netqueue filter has yet been programmed. NetQueues, on the other hand, require an explicit unicast RX filter to be applied before they can receive traffic. A filter matches traffic based on criteria such as MAC address, VLAN ID, or tunnel VNI, and directs matching traffic to a specific dedicated queue. This means every new network flow starts on the default queue first. The vmkernel observes the traffic pattern on the default queue, learns the MAC address or flow identifier, and only then allocates a dedicated netqueue and programs the corresponding RX filter. Until that filter is in place, all traffic for that flow continues to arrive on the default queue. As a result, the default queue carries a significant and continuous load:
- All new and unrecognized flows before netqueue assignment
- All broadcast and multicast traffic — permanently, since netqueues only handle unicast filters
- All control and management plane traffic (ARP, DHCP, IGMP, netdump capture, etc.)
- All overflow traffic when netqueue or filter limits are reached This is why the default queue is inherently the busiest queue on the NIC — not because netqueues are idle by design, but because a large portion of traffic is structurally unmatched and must flow through the default queue.
This is why the default queue is inherently the busiest queue on the NIC — not because netqueues are idle by design, but because a large portion of traffic is structurally unmatched and must flow through the default queue.
The Four-Filter Limitation
Marvell FastLinQ ESXi driver (qedentv) had a design constraint of having a maximum of four filters per RX queue. When a queue had four filters assigned to it, no additional flows could be assigned to that queue. In practice, this means that many RX flows cannot be moved from the default queue to netqueues. All traffic that does not match an existing filter ends up on the default queue, regardless of how many other queues are available. The result was uneven load distribution:
- The default queue is overloaded
- Other queues are underutilized
- The CPU core handling the default queue becomes a bottleneck
- Throughput drops and latency increases
This was the root cause of numerous customer support requests reporting RX drops and performance degradation on FastLinQ E4 CNAs.
What Is DRSS?
Default queue RSS (DRSS) is a feature that addresses this limitation. Instead of relying on individual queue filters, DRSS distributes incoming flows across a pool of DRSS queues using a hash-based algorithm.
When DRSS is enabled:
- Flows that cannot be assigned to a specific netqueue via filters are distributed across the DRSS queue pool
- The load is spread across multiple CPU cores
- The default queue is no longer a bottleneck
- Throughput improves and RX drops decreases
The Fix: Enabling DRSS by Default
Starting with qedentv device driver version 3.71.90.2, DRSS is enabled by default for non-ENS (Enhanced Networking Stack) mode. Analysis of multiple customer cases showed that the root cause was consistently the same: DRSS was not activated. As a result, DRSS is now enabled by default.
The fix is straightforward in principle but required careful validation:
- DRSS must not interfere with ENS mode, which has its own traffic handling path
- DRSS must not affect NPAR (NIC Partitioning) mode behavior
- The change must not introduce regressions in existing deployments
How to Verify if DRSS is Working
After upgrading to a device driver version with DRSS enabled by default, you can verify the configuration using the following steps:
Check DRSS status: esxcli network nic drss get -n vmnicX
Monitor per-queue RX statistics: vsish -e get /net/pNics/vmnicX/stats
If DRSS is working correctly, you should see traffic distributed across multiple queues rather than concentrated on the default queue.
Check for RX errors: esxcli network nic stats get -n vmnicX
RX error counters should not be advancing if DRSS is properly distributing the load.
When to Consider Manual DRSS Configuration -
While DRSS is now enabled by default, there are scenarios where manual configuration may be needed:
- High-density VM environments: If you have many VMs with high network traffic, you may want to tune the DRSS queue pool size
- NPAR mode: In NPAR mode, traffic distribution behavior is different. Review the NPAR documentation before enabling DRSS
- ENS mode: DRSS is not applicable in ENS mode. ENS has its own optimized traffic path
Conclusion
A seemingly benign driver constraint — limited RX filters per queue — can significantly impact system performance at scale. The default queue, by its very nature, carries the heaviest and most diverse traffic load: new flows, broadcast, multicast, control plane traffic, and overflow from netqueue filter limits. A single queue and a single CPU core handling all of this is simply not sufficient at scale.
DRSS removes this bottleneck by enabling efficient workload distribution across CPU cores, improving throughput and reducing packet drops. This case underscores a key principle: unlocking hardware performance often depends on ensuring effective workload distribution, not just raw capability.
Got an idea?
Ready to transform your idea into a functioning product ?
