UPDATE: Here's a Bit.ly Bundled Link of the complete "Wiring your house for Gigabit Ethernet 5 PART SERIES."
I posted earlier about copying files across my new Gigabit home network. I was getting about 10 MB/s (that's MegaBytes) between two laptops and I got responses like these in the comments that shook my confidence. Here's my thoughts after each interesting comment that appeared in the previous post.
"You should be pushing at least 100 megabytes per second with gigabit ethernet."
Hm..that doesn't sound right. Given bits/bytes math and some Maximum speed given overhead, etc, is usually 80% of spec, so 1000Mb/s means about 80MB/s. But, still, the point is taken.
"His stats only caught my attention because just today I was profiling the speeds of a new hard drive array for one of the programmers and copied 3.2gb in 7 minutes over our 100mbit ethernet connection."
That's about 8-10 megs a second, so that's on par which what I was seeing. This makes me wonder about the hard drive speed.
"10Meg/sec is well below what I'd expect even for a laptop to laptop exchange over a gigabit connection where the drives probably top out at 30-50Meg/sec"
I checked my hard drive throughput and it was 70MB/s sequential writes and 40MB/s sequential large file reads on the big machine and 30/20 on the laptop. Certainly fast enough that I’d want to see similar network speeds when copying a single large VM, etc. However, I was copying hundreds of files on a medium fragmented drive, so 10MB/s isn't unreasonable, IMHO. I'll look at disks in a second.
"You should be getting about ~32meg per second with standard 5400rpm notebook hard drives copying a single large file."
OK, sure I can believe that, assuming there is nothing else going on and there's a nice contiguous spot to lay it down, absolutely. So, let's break this down step by step.
Starting Point
Here's where we start, as possibly useful background, I recently built a house and put some amount of effort into the Home Network.
The house is all Cat 6 wiring, and the main switch is a NetGear 24-port with 20 gigabits of total managed bandwidth. For this test machines are OFF the corporate network and I’ve physically turned their wireless cards OFF. They are single-homed on a basic flat 192.168.x.x network. Nothing fancy.
I brought VS2008 down from the cloud, the files, not the single file large ISO and copied the 3 gigs of files via Explorer Drag Drop between two Vista 32-bit RTM machines and got disappointing results.
No music was playing. The machines are all dual or quad core machines, one with a 10,000 RPM hard drive and the others are pretty fast laptops, so they are beefy machines.
Why mention this? First we eliminate slow or crappy hardware. In this instance, we know it's not CPU that's causing this. That doesn't mean the network cards aren't poo or something, but we gotta start somewhere. Point is, they are more than fast enough to handle the traffic.
The Initial Sub-Optimal Results
So, after the first run, I was getting 10MB/s between a Vista RTM machine (a Mac) and an SP1 machine. When Vista SP1 talks to Vista RTM it'll talk SMB 1.0, the common protocol between them (the one that's been used forever).
Of course this isn't a BAD result, as one of the commenters pointed out. I mean, 10 MB/s is more than reasonable most often and this is the kind of thing one would only come up against when copying giant files. Still...now it's a mystery...
Disk Speed
Perhaps Disk Speed is the problem? Commenters on previous posts say Nay, Nay. SATA disks should get 40 to 70MB/s, and that's consistent with my test. The Hanselman-Atwood Ultimate Developer Rig gets 70Mb/s for sequential writes. Of course this assume non-fragmented disks and that nothing else is moving the disk heads around. The MacBookPro and Lenovo t60p get between 20 and 30MB/s, so they are not slouches either. All this points away from Disk Speed.
Here's disk throughput on a laptop:
C:\>disktest.exe -file "Random" -iobytes 8388608 -pipeline 2 -totalbytes 1073741824
---
File name: Random
Total size: 1073741824 bytes
I/O type: read
I/O size: 8388608 bytes
I/O concurrency: 2 (asynchronous)
Buffered: no
---Running the test...---
App. Throughput: 33.2 MBytes/sec (30.9 seconds)
Disk Throughput: 33.2 MBytes/sec (27 samples, 1 high variance)
Min. I/O latency: 250 ms
Avg. I/O latency: 480 ms
Max. I/O latency: 811 ms
So, disk throughput is fine.
Network Cards
In cases like these, it's always a good idea to step back and read the manual. I checked out the release notes for each of my Network Cards and confirmed that they do work on Gigagbit and on Vista and had no issues that smelled like what I am seeing.
One useful tip from Ed Briggs:
"One thing to watch. On some motherboards, the gigibit NIC (usually a Marvell Yukon 88E8001) is wired to the PCI-33 bus, even if there is a PCI express bus. I find this on a number of boards where they advertise multiple Gig Eth NICs. So say, you'll have an NVidia on the PCI-E or the southbridge, and a Marvell on the PCI-33, and your Intel Pro1000 on the PCI-E. So you'll see some differences there because PCI-33 runs out of steam. "
This wasn't the case for me, but it is very good to know.
Cabling
Even though it shouldn't matter I checked all the cables and confirmed that the house was Cat6. I had been using Cat5 (not 5e) patch cables, so I swapped those out for Cat 5e, and ordered some Cat6 ones just to have.
Switch/Router
Next, I confirmed that my Switch saw these different machines connecting as Gigabit. Here's a screenshot from the Administration Interface of my switch.
I'm going between g18, g19, and g21. The administration interface for the switch also shows the length of the cables.
Everything is reasonable from a cabling perspective.
Does the Center Hold?
I determined that I must be making some terrible mistake. Rather than pulling and proding, I just started testing with traffic. Here’s a NTttcp run. Note the CPU and MBits. You can get NTttcp in the Windows 2000 Resource Kit.
Total Bytes(MEG) Realtime(s) Average Frame Size Total Throughput(Mbit/s)
================ =========== ================== ========================
1342.177280 20.649 1424.681 519.997
Packets Sent Packets Received Total Retransmits Total Errors Avg. CPU %
============ ================ ================= ============ ==========
942090 83137 0 0 7.27
Here's a result with two threads. Kind of weird that it's more, but it is over 800 megabits. This implies that the network is generally OK.
Total Bytes(MEG) Realtime(s) Average Frame Size Total Throughput(Mbit/s)
================ =========== ================== ========================
2684.354560 26.500 18753.481 810.371
Packets Sent Packets Received Total Retransmits Total Errors Avg. CPU %
============ ================ ================= ============ ==========
143139 175153 3 0 15.46
Then I tried
I tried iPerf, the standard cross-platform network throughput tester.
Z:\ntttcp\iperf>iperf -c 192.168.1.14
------------------------------------------------------------
Client connecting to 192.168.1.14, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[132] local 192.168.1.9 port 49642 connected with 192.168.1.14 port 5001
[ ID] Interval Transfer Bandwidth
[132] 0.0-10.0 sec 360 MBytes 302 Mbits/sec
Notice the small TCP window size. I'll try again with a larger one.
Z:\ntttcp\iperf>iperf -c 192.168.1.14 -w 64k
------------------------------------------------------------
Client connecting to 192.168.1.14, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[132] local 192.168.1.9 port 49654 connected with 192.168.1.14 port 5001
[ ID] Interval Transfer Bandwidth
[132] 0.0-10.0 sec 1.08 GBytes 926 Mbits/sec
Ok, so that's good and implies again that the network doesn't totally suck. Note also that this very good test happened between a Vista RTM machine and a Vista SP1 machine.
Local TCP Settings
Next, I checked the window size on my MSFT Corporate t60p Laptop with Vista SP1 and was shocked to see the tuning level set to highlyrestricted. Not sure who set that but it wasn’t me (that I remember). I switched it back to “normal” but it didn’t affect any of the tests – I ran them again. Everyone is set to normal.
C:\Users\Scott\Desktop\ntttcp>netsh interface tcp show global
Querying active state...
TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State : enabled
Chimney Offload State : enabled
Receive Window Auto-Tuning Level : highlyrestricted (should be normal)
Add-On Congestion Control Provider : none
ECN Capability : disabled
RFC 1323 Timestamps : disabled
The standard setting of "normal" should work for nearly everyone although some folks swear by "disabled."
Gigabit Jumbo Frames
Marc suggested enabling Jumbo Frames, usually a NIC Driver Level settings. If all your NICs and router/switch support it, you can have frame sizes up to 9014 bytes. However, only to of my NICs supported it, so I didn't bother.
UPDATE: Turns out that Jumbo Frames are obscenely useful. If your network card and router supports them, go for it. More on this at the bottom.
Reintroducing The Players
Right now here’s the machines:
- Quadpower – The Good Lord's own machine. Truly. nForce chipset. Vista x64 RTM. WEI 5.8
- MacBookPro – Vista x86 SP1 Beta from last week. WEI 4.9. This machine is no slouch.
- Lenovo T60p – Vista x64 SP1 Beta from last week. WEI 4.2. MSFT Domain joined, but not right now.
- Server – Windows Home Server machine, based on Windows 2k3
The Test Matrix
Here are the four machines in question.
| |
Quadpower x64RTM |
MacBookPro x86SP1 |
T60p x64SP1 |
WHS 2k3 |
| Quadpower x64RTM |
X |
<10Mb/s |
<10Mb/s |
<10Mb/s |
| MacBookPro x86SP1 |
X |
X |
33Mb/s |
33Mb/s |
| T60p x64SP1 |
X |
X |
X |
33Mb/s |
| WHS 2k3 |
X |
X |
X |
X |
So it appears the problem is on/around/near/adjacent to Quadpower. When stuff goes into or out of it, it's slower.
Here’s SP1 to WHS (W2k3) copying a 1 gig file of random data.
![clip_image002[6]](https://images.hanselman.com/blog/WindowsLiveWriter/34655e3b6f66_D1F1/clip_image002%5B6%5D_thumb.jpg)
Here’s SP1 to RTM:
Here’s the Lenovo t60p with Vista SP1 to a MacBookPro running Vista SP1
This is starting to sound familiar. I'm mixing my network protocols, specifically SMB2 vs. SMB1.
"These are some of the key enhancements in SMB 2.0:
- SMB 2.0 supports an arbitrary, extensible way of compounding operations to reduce round trips. This makes the protocol less chatty as compared to SMB 1.0. Chattiness of SMB 1.0 has often been a major pain point.
- SMB 2.0 supports much larger buffer sizes compared to SMB 1.0.
- SMB 2.0 greatly grows the restrictive constants in the protocol, so we never need to worry about the protocol itself being the limiting factor for scalability. This includes increasing the number of concurrent open file handles on the server, and the number of shares that a server can share out, among other things.
- SMB 2.0 supports durable handles that can withstand short network glitches.
- SMB2.0 has support for symbolic links. "
NetMon 3.1 confirms that I'm talking SMB1 between my RTM and SP1 machines and SMB2 between SP1 machines.
Conclusion
One thing worth pointing out that slowed down my analysis was that I only saw this slowdown when doing a File Copy. I was assuming this was a Network Hardware or TCP issue, perhaps between Vista RTM and Vista SP1. However, the tests above that showed 900+ Mbits/s were actually between and RTM and SP1 machine. Vista Networking works rather fine it seems. It was the older SMB1 protocol being negotiated between my RTM and SP1 machines that was the bottleneck.
I was also prodded to look more into Jumbo Frames. (Thanks Robert G!) Seems my Windows Home Server machine's network card has a Marvell Yukon that supports Jumbo Frames, as does the Marvell Yukon in the MacBookPro. Even though the machines will be using SMB1 as their protocol, the 9104byte frame size makes a massive difference. How much? With a 1 gig file, twice as fast, so 65+ Megabytes a second, even over SMB1. Cool!
The NVidia nForce hardware chipset supports Jumbo Frames, but the drivers do not yet. NVidia is mum on the issue, which is lame. This the the chipset in my primary machine, the one I'd really like to support Jumbo.
In the Lenovo T60p there is a Intel Pro/1000 PL with an Intel 82573L chipset which apparently doesn't support Jumbo Frames when Active State Power Management (ASPM) is disabled, and since it's in a laptop, it's disabled. No Jumbo Frames on a Lenovo t60p, even with the latest Intel Network Adapter Driver for Vista 64-bit.
"Intel does not plan to resolve this erratum in the 82573 Gigabit Ethernet
Controller. Jumbo frames is not supported in 82573E/V & is supported with the
workaround above in 82573L."
Regardless, it turns out SMB2 is a way better protocol than SMB1. As far as I'm concerned, problem solved, I'm upgraded to Vista SP1 everywhere and am enjoying Disk Speed on the wire. Now I've gotten speeds up to 33MB/s between machines when copying large files depending on the disk of the machines involved, etc, without Jumbo Frames, and up to 66MB/s with Jumbo. I'll be sure to only get network cards that support Jumbo Frames in the future. I'm happy I went gigabit and I'm happy I went Vista SP1 and I'm happy I went Cat6.
Whew. I was worried I'd have to tear my walls open. ;)
Hosting By
![clip_image002[6]](https://images.hanselman.com/blog/WindowsLiveWriter/34655e3b6f66_D1F1/clip_image002%5B6%5D.jpg)
