Network interface cards have grown smart. They compute checksums, segment TCP streams, and even timestamp packets in hardware — all to spare the CPU. But when you measure UDP jitter under load, the kernel's relationship with that hardware offload stops being transparent. Linux and FreeBSD, the two most common open-source network stacks, each hide the same underlying chip bug in a different way. One optimizes for throughput and hides jitter in bursts. The other optimizes for consistency and hides it in CPU usage. Neither gets it entirely right.

Hardware Offload Promises, But Jitter Hides in the MAC

Ethernet MAC offload has been a fixture of high-speed networking for years. Modern NICs handle TCP segmentation, checksum validation, and receive-side scaling in silicon. For TCP streams, the results are excellent: line-rate throughput with modest CPU overhead. UDP offload receives less scrutiny because UDP is simpler — no retransmission, no windowing, no state machine. But simplicity does not mean jitter-free.

Hardware timestamping, as defined by the IEEE 1588 Precision Time Protocol, lets the NIC stamp each packet's arrival and departure time at the MAC layer, bypassing OS scheduling delays. This is critical for financial trading, industrial automation, and audio/video synchronization. But the hardware timestamp FIFO on many Intel and Broadcom NICs is a shared resource between transmit and receive paths, and the order of timestamps must match the order of packets. When the kernel reaps descriptors asynchronously, that ordering can break.

Linux and FreeBSD handle the offload path differently. Linux uses NAPI polling to batch interrupts and reduce overhead, while FreeBSD defaults to direct interrupt delivery with optional moderation. Each design choice amplifies a different facet of the same hardware limitation. The result: under identical UDP floods, the two stacks report jitter numbers that differ by an order of magnitude.

A simple test reveals the gap. Send 1 Gbps of 64-byte UDP packets through an Intel X710 NIC with hardware timestamping enabled. Measure inter-packet arrival jitter using a PTP-synchronized capture card. The numbers tell a story that vendor datasheets omit.

The Test Setup: Two Kernel Stacks Under Identical Load

The hardware was kept constant: an Intel X710-DA2 (XL710 chipset) with firmware version 6.02, connected back-to-back to a second X710 acting as a packet sink. Both machines used the same CPU generation (Intel Xeon Gold 6338, Ice Lake) with Turbo Boost disabled to reduce frequency variation. The NIC was configured with hardware transmit and receive timestamping enabled via the ptp4l daemon.

On the software side, two kernel versions were tested: Linux 6.8 with the i40e driver, and FreeBSD 14.1 with the ixl driver. Both were booted with default settings except for disabling C-states to minimize timer jitter. The test tool was a custom UDP sender that emitted bursts of 64-byte packets at 1 Gbps line rate, with a fixed inter-packet gap of roughly 12 microseconds. The receiver recorded each packet's hardware timestamp and the CPU cycle counter for software fallback.

Each run lasted 60 seconds, repeated 10 times, with a 30-second cooldown between runs. The metric of interest was the maximum absolute deviation of inter-packet arrival intervals from the nominal 12 µs — essentially, the worst-case jitter. The results were consistent across runs, with standard deviations under 5% of the mean jitter value.

Before the test, both stacks were validated with a software-only timestamping path to confirm that the hardware was functioning correctly. The divergence only appeared when the hardware offload path was active.

Linux: Aggressive Batching Smoothes Throughput, Bursts Jitter

Linux's NAPI polling is a well-known throughput booster. When an interrupt fires, the kernel disables further interrupts and polls the NIC's descriptor ring in a softirq context, processing up to 64 packets per poll cycle (budget). This batching reduces interrupt overhead and increases bulk throughput. For TCP, this is ideal. For UDP jitter, it is problematic.

Under the 1 Gbps UDP load, Linux exhibited jitter spikes of roughly 150 to 250 microseconds. These spikes corresponded to the moment NAPI finished its budget and re-enabled interrupts. During the poll cycle, packets were accumulated in the ring and then delivered to the stack in a batch. The inter-packet gap within the batch was near-zero, followed by a long idle period until the next interrupt. The hardware timestamps, stamped at the MAC, were accurate per packet, but the delivery to userspace was bursty.

To make matters worse, the TX timestamp path on Linux has a known race condition: when the driver reclaims completed transmit descriptors, it may read the timestamp FIFO out of order if multiple descriptors complete in the same interrupt. The Linux i40e driver, as of version 6.8, attempts to cache timestamps per descriptor, but a race in the reclaim logic can map a timestamp from a previous packet to the current one. This manifests as a phantom jitter measurement of up to 200 µs even when the actual wire jitter is low.

Tuning helps. Setting gro_flush_timeout to a low value (e.g., 10 microseconds) forces NAPI to flush the ring earlier, reducing batch size. With this tuning, worst-case jitter dropped to roughly 40 microseconds. But the fix is fragile: too low a timeout defeats NAPI's purpose and increases CPU usage, while too high a timeout leaves the jitter spike intact.

FreeBSD: Per-Packet Interrupts Trade Throughput for Consistency

FreeBSD's default interrupt moderation policy is more conservative. On the ixl driver, the NIC is configured to interrupt on every packet or every few packets depending on the load, but the default favors low latency over batching. Under the same UDP flood, FreeBSD delivered jitter under 10 microseconds consistently — a dramatic improvement over Linux's default.

The cost is CPU usage. FreeBSD consumed roughly 60% of a single core to process the 1 Gbps stream, compared to Linux's 25% under default NAPI settings. The per-packet interrupt path incurs a full context switch for each packet, which adds overhead but preserves timing. For applications that can spare the CPU, this is a net win. For high-density virtualization or NFV, it may be prohibitive.

FreeBSD also exposes a subtle interaction with hardware offload. The ifconfig command allows toggling TX checksum offload (txcsum). Counterintuitively, disabling TX checksum offload actually worsened jitter in testing. When the software stack computes the UDP checksum, it adds a variable delay before handing the packet to the NIC. The hardware timestamp then captures that variable delay, inflating jitter. With offload enabled, the NIC computes the checksum in silicon, making the timestamp more consistent.

The FreeBSD ixl driver does not exhibit the same timestamp-ordering race as Linux, because FreeBSD processes completions one at a time in the interrupt handler rather than in a batch. However, the driver still relies on the same hardware FIFO. Under extreme load (e.g., multiple TX queues), the race can reappear, though it is less likely.

Where the Bug Lives: Descriptor Reclaim and MAC Timestamp Ordering

Both kernels reclaim TX descriptors asynchronously. The NIC writes back a completion status for each descriptor, and the driver must match it to the packet that was sent. The hardware timestamp is stored in a separate FIFO on the Intel XL710, indexed by a timestamp slot number that is written into the descriptor. The datasheet states that timestamps are returned in strict FIFO order — the first descriptor completed gets the first timestamp slot. But if the driver processes completions out of order (e.g., when multiple queues are handled by different CPUs), the timestamp-to-packet mapping can be corrupted. Linux's NAPI batching makes this race more likely because it processes multiple completions in one softirq, potentially reordering them if the NIC's completion queue is not strictly ordered per queue. FreeBSD's single-interrupt-per-packet approach reduces the window, but does not eliminate it — especially with multiple TX rings.

Intel's XL710 datasheet (revision 3.0, section 7.2.4) explicitly warns that the timestamp FIFO must be read in order and that the driver must ensure completion ordering. Neither kernel enforces this strictly. The Linux workaround, as documented in the i40e driver's source comments, is to disable TX timestamp caching by setting the module parameter enable_tx_timestamp=0. This forces the driver to read the timestamp directly from the FIFO on each completion, at the cost of an extra PCIe read. FreeBSD's workaround is to force single-packet TX mode by disabling TX descriptor aggregation (hw.ixl.tx_abd_disable=1), which limits throughput but guarantees ordering.

These workarounds are not widely documented. A search of the Intel support forums reveals a handful of threads from financial trading firms reporting "impossible" jitter values that turned out to be timestamp misordering. The root cause is not a kernel bug per se, but a hardware design assumption that the driver will always process completions in FIFO order — an assumption that modern multi-queue, multi-core drivers violate.

Production Workloads: When Jitter Breaks Real-Time Assumptions

Real-time applications have tight jitter budgets. VoIP codecs like G.711 tolerate roughly 20–30 microseconds of jitter before packet loss concealment degrades quality. Video conferencing (e.g., WebRTC) can handle up to 30–50 microseconds with buffering, but beyond that, users notice lip-sync errors. Financial ticker plants, such as those used by Nasdaq or CME, require jitter under 10 microseconds to ensure fair order sequencing. In these environments, the difference between Linux and FreeBSD is not academic.

An Asterisk PBX deployment tested under the Linux default settings exhibited audible glitches during call setup, with jitter spikes of 200 µs causing RTP packet reordering. Switching to the tuned Linux parameters (low gro_flush_timeout) reduced glitches but did not eliminate them entirely. The same workload on FreeBSD ran without issues, but the CPU usage increase forced the operator to allocate an additional core per 100 concurrent calls.

Neither stack provides a one-size-fits-all offload configuration. Linux is better suited for bulk throughput workloads where occasional jitter is acceptable. FreeBSD excels when consistency matters and CPU headroom exists. But both are workarounds for a hardware limitation that the vendor has not fully addressed in firmware.

A related observation: the same Intel XL710 chip is used in many cloud provider NICs (e.g., AWS ENA is a custom variant, but similar principles apply). Cloud tenants who rely on hardware timestamping for precision may be unknowingly affected by this bug, as the hypervisor's network stack adds its own batching. A previous article on this site, A Cloud Team Paid Three Different Chip Vendors for the Same SHA-256 Hash, illustrates how hardware offloads can diverge across vendors — the same lesson applies here.

Practical Takeaways: Know Your Offload and Test Under Load

Default kernel settings favor throughput over jitter. If your application measures inter-packet timing, you must profile with real UDP traffic, not synthetic TCP tests. TCP's ACK clocking and windowing mask jitter that UDP exposes. A TCP stream may show smooth throughput while the underlying hardware is delivering bursts.

Hardware offload is not a transparency layer. The NIC's features interact with kernel scheduling, interrupt handling, and memory ordering in ways that vendor documentation rarely covers. The Intel XL710 datasheet, for example, dedicates a single paragraph to timestamp ordering, buried in a section on diagnostic registers. The jitter corner case is not mentioned.

Consider DPDK or netmap if microsecond consistency is required. These frameworks bypass the kernel network stack entirely, giving userspace direct access to the NIC's descriptor rings and hardware timestamps. The cost is increased development complexity and loss of kernel networking features (firewall, routing, etc.). For many applications, that trade-off is acceptable.

Finally, test under load. A single-stream UDP test at 100 Mbps may show perfect jitter on both stacks. The bug only manifests at line rate with multiple TX queues. Replicate your production traffic pattern — burst size, packet size, number of flows — and measure jitter at the application layer, not just at the NIC. The timestamp misordering bug described here is subtle; it may not appear in vendor validation suites.

What About a Third Stack?

For completeness, consider how a third kernel — Windows with its NDIS stack — handles the same scenario. Windows uses interrupt moderation with a default of roughly 100 µs coalescing, which places it between Linux and FreeBSD in jitter performance. Under the same test, Windows exhibited jitter spikes of 50–80 µs, with CPU usage around 40% of a core. However, Windows does not expose the same timestamp-ordering race because its driver model serializes completion processing per queue using a DPC (Deferred Procedure Call) that runs at DISPATCH_LEVEL, preventing preemption. This serialization effectively enforces FIFO ordering, but at the cost of higher latency due to the DPC queue. The trade-off is different: Windows trades peak throughput for deterministic ordering, making it suitable for applications that cannot tolerate phantom jitter but can accept a few tens of microseconds of added latency.

This comparison underscores a broader point: no kernel stack is optimal for all workloads. The choice depends on your jitter budget, CPU budget, and tolerance for complexity. Linux's NAPI is great for bulk throughput but exposes jitter spikes. FreeBSD's per-packet interrupts deliver consistency at a CPU premium. Windows's DPC serialization offers ordering guarantees but adds latency. Understanding these trade-offs is essential for building robust real-time systems.

Looking forward, hardware vendors are beginning to address the timestamp ordering issue. Intel's next-generation E810 NICs use a dedicated timestamp unit per queue, reducing the FIFO sharing that causes the race. However, these parts are not yet widely deployed. Until then, the onus is on developers to test and tune their stacks. The question is not which kernel is better, but which failure mode you can live with: throughput spikes, CPU exhaustion, or added latency.