Scott Hanselman

I've been moving my home storage over to a UniFi UNAS Pro 8 as part of a larger homelab moderization. My main IRONHEART (the ultimate PC from a few years back) desktop now has an Intel E610-XT2 10GbE card, the NAS is on 10GbE, and there's a Minisforum MS-01 miniPC on the same network with a 10GbE SFP+ connection running Immich and Portainer and a few other things.

Everything says 10 gigabit. Windows says 10 gigabit. UniFi says 10 gigabit. SMB copies are using the correct NIC but my file copies are running at around 100 to 200 megabits per second which is sad making.

Naturally, I blamed the NAS, and the spinning rust within. The UNAS has six 16 TB spinning disks in RAID 6 and a pair of NVMe SSDs being used as cache. I'm also running Immich on the MS-01, with its photo library living on the UNAS, so there are lots of thumbnails, metadata reads, and little background writes happening. All seem like reasonable suspects.

I switched the UNAS SSD cache from read-write to read-only. No meaningful difference. I stopped Immich completely. No difference. I looked at iostat; the disks weren't saturated. We looked at SMB signing and Windows Defender network scanning. Still slow.

Then I stopped testing the NAS and ran iperf3 directly between the Windows desktop and the MS-01:

iperf3 -c 192.168.1.222 -P 4

133 Mbit/sec

Oops. The reverse test was better, but still wrong:

iperf3 -c 192.168.1.222 -P 4 -R

1.33 Gbit/sec

That's weird. Now the disks, SMB, Immich, RAID, and the NAS itself were completely out of the equation. This was a Windows/NIC problem and it's weirdly asymmetrical.

Looking at the Intel adapter statistics shows me...

Get-NetAdapterStatistics -Name "Ethernet - 10 Gig Intel"

There were nearly a million ReceivedDiscardedPackets. During one ten-second iperf3 test, the counter increased by another 268. Why?

The E610 driver had its receive buffers at the default 512, although it supported up to 4096. I increased them. I love an increased buffer.

Set-NetAdapterAdvancedProperty `
  -Name "Ethernet - 10 Gig Intel" `
  -DisplayName "Receive Buffers" `
  -DisplayValue "4096"

The discarded-packet count during the next test went from 268 to zero, and receive throughput jumped from 1.33 Gbit/sec to 5.15 Gbit/sec. The transmit direction was still terrible, basically 313 Mbit/sec. The next experiment was disabling Large Send Offload (LSO) V2 for IPv4:

Set-NetAdapterAdvancedProperty `
  -Name "Ethernet - 10 Gig Intel" `
  -DisplayName "Large Send Offload V2 (IPv4)" `
  -DisplayValue "Disabled"

Then I ran the same iperf3 test again.

7.03 Gbit/sec

That's not a typo. 313 Mbit/sec to 7.03 Gbit/sec by changing one NIC setting. Sweet sassy molassey.

LSO exists for a good reason: Windows can hand large TCP buffers to the NIC and let the adapter/driver segment them into network-sized packets, reducing CPU work. Microsoft does, however, explicitly point out that segmentation offload can reduce maximum sustainable throughput with some network adapters/configurations. LSO is usually useful, but not in this case.

In my particular combination of Windows and the Intel E610-XT2, something in the IPv4 LSO path was very, very sad. I don't yet know whether this is an Intel driver bug, firmware issue, Windows interaction, or something particular to this machine, so I wouldn't turn this into random tech blogger advice that everyone should disable LSO. Measure first, cut once. Er, twice. Just stay woke.

Finally I went back to the test that started all this and copied the same large file to the UNAS and Robocopy reported:

Speed : 350,201,354 Bytes/sec.
Speed : 20,038.682 MegaBytes/min.

About 350 MB/sec, or 2.8 Gbit/sec of sustained real-world SMB writes to a six-disk RAID 6 NAS.

That's much more like it. The useful lesson isn't "disable LSO." It was that when storage is mysteriously slow, eventually you have to stop testing storage. iperf3 removed the NAS, filesystem, RAID, cache, SMB and disks from the experiment in one move. Once the raw network was also slow, the problem became dramatically smaller. And, sometimes the little checkbox labeled Large Send Offload is capable of making your 10 gigabit card run like it's 2004.

TL;DR - with LSO V2 for IPv4 enabled, Windows-to-Linux iperf3 managed about 313 Mbit/sec. Turning off that single offload took the exact same test to 7.03 Gbit/sec. I’m deliberately saying on this machine because LSO is normally useful and this isn’t blanket advice to disable it everywhere. 

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By