DRSS : The Hidden Performance Lever in the FastLinQ ESXi Driver
In a restaurant, does one waiter take all the orders while others wait? Obviously not. All those available share the work.
In the early days of computing, only one CPU was available to perform processing. Now we have systems having multiple CPU cores. Today's software should facilitate even distribution of workload. This was the situation faced with the FastLinQ ESXi device driver before DRSS was enabled by default.
"The default queue is inherently the busiest queue on the NIC — not because netqueues are idle by design, but because a large portion of traffic is structurally unmatched."
Understanding the Default Queue and NetQueues
ESXi uses NetQueue to distribute incoming network traffic across multiple receive (RX) queues on a NIC. Each queue is associated with a CPU core, allowing parallel processing.
The default queue operates in promiscuous mode — it receives all packets not matching any existing netqueue filter: broadcast traffic (ARP, DHCP), multicast traffic, control plane traffic, and unicast traffic with no netqueue filter filter has yet been programmed.
NetQueues, on the other hand, require an explicit unicast RX filter to be applied before they can receive traffic. A filter matches traffic based on criteria such as MAC address, VLAN ID, or tunnel VNI, and directs matching traffic to a specific dedicated queue.
This means every new network flow starts on the default queue first. The vmkernel observes the traffic pattern on the default queue, learns the MAC address or flow identifier, and only then allocates a dedicated netqueue and programs the corresponding RX filter. Until that filter is in place, all traffic for that flow continues to arrive on the default queue.
As a result, the default queue carries a significant and continuous load:
- All new and unrecognized flows before netqueue assignment
- All broadcast and multicast traffic — permanently, since netqueues only handle unicast filters
- All control and management plane traffic (ARP, DHCP, IGMP, netdump capture, etc.)
- All overflow traffic when netqueue or filter limits are reached.
The Four-Filter Limitation
Marvell FastLinQ ESXi driver (qedentv) had a design constraint of having a maximum of four filters per RX queue. When a queue had four filters assigned to it, no additional flows could be assigned to that queue.
In practice, this means that many RX flows cannot be moved from the default queue to netqueues. All traffic that does not match an existing filter ends up on the default queue, regardless of how many other queues are available.
The result was uneven load distribution:
- The default queue is overloaded
- Other queues are underutilized
- The CPU core handling the default queue becomes a bottleneck
- Throughput drops and latency increases
What Is DRSS?
Default queue RSS (DRSS) is a feature that addresses this limitation. Instead of relying on individual queue filters, DRSS distributes incoming flows across a pool of DRSS queues using a hash-based algorithm.
When DRSS is enabled:
- Flows that cannot be assigned to a specific netqueue via filters are distributed across the DRSS queue pool,
- The load is spread across multiple CPU cores
- The default queue is no longer a bottleneck
- Throughput improves and RX drops decreases
The Fix: Enabling DRSS by Default
Starting with qedentv device driver version 3.71.90.2, DRSS is enabled by default for non-ENS (Enhanced Networking Stack) mode. Analysis of multiple customer cases showed that the root cause was consistently the same: DRSS was not activated. As a result, DRSS is now enabled by default.
The fix is straightforward in principle but required careful validation:
- DRSS must not interfere with ENS mode, which has its own traffic handling path
- DRSS must not affect NPAR (NIC Partitioning) mode behavior
- The change must not introduce regressions in existing deployments
How to Verify if DRSS is Working
After upgrading to a device driver version with DRSS enabled by default, you can verify the configuration using the following steps:
Check DRSS status: esxcli network nic drss get -n vmnicX
Monitor per-queue RX statistics: vsish -e get /net/pNics/vmnicX/stats
If DRSS is working correctly, you should see traffic distributed across multiple queues rather than concentrated on the default queue.
Check for RX errors: esxcli network nic stats get -n vmnicX
RX error counters should not be advancing if DRSS is properly distributing the load.
When to Consider Manual DRSS Configuration:
While DRSS is now enabled by default, there are scenarios where manual configuration may be needed:
- High-density VM environments: If you have many VMs with high network traffic, you may want to tune the DRSS queue pool size
- NPAR mode: In NPAR mode, traffic distribution behavior is different. Review the NPAR documentation before enabling DRSS
- ENS mode: DRSS is not applicable in ENS mode. ENS has its own optimized traffic path.
Conclusion
A seemingly benign driver constraint — limited RX filters per queue — can significantly impact system performance at scale. The default queue carries the heaviest and most diverse traffic load. A single queue and single CPU core is simply not sufficient at scale.
DRSS removes this bottleneck by enabling efficient workload distribution across CPU cores, improving throughput and reducing packet drops.
Got an idea?
Ready to transform your idea into a functioning product ?
