Scott Hanselman

Wiring the house for a Home Network - Part 5 - Gigabit Throughput and Vista

December 1, '07 Comments [26] Posted in ASP.NET | Microsoft | Musings | Programming
Sponsored By

UPDATE: Here's a Bit.ly Bundled Link of the complete "Wiring your house for Gigabit Ethernet 5 PART SERIES."

I posted earlier about copying files across my new Gigabit home network. I was getting about 10 MB/s (that's MegaBytes) between two laptops and I got responses like these in the comments that shook my confidence. Here's my thoughts after each interesting comment that appeared in the previous post.

"You should be pushing at least 100 megabytes per second with gigabit ethernet."

Hm..that doesn't sound right. Given bits/bytes math and some Maximum speed given overhead, etc, is usually 80% of spec, so 1000Mb/s  means about 80MB/s. But, still, the point is taken.

"His stats only caught my attention because just today I was profiling the speeds of a new hard drive array for one of the programmers and copied 3.2gb in 7 minutes over our 100mbit ethernet connection."

That's about 8-10 megs a second, so that's on par which what I was seeing. This makes me wonder about the hard drive speed.

"10Meg/sec is well below what I'd expect even for a laptop to laptop exchange over a gigabit connection where the drives probably top out at 30-50Meg/sec"

I checked my hard drive throughput and it was 70MB/s sequential writes and 40MB/s sequential large file reads on the big machine and 30/20 on the laptop. Certainly fast enough that I’d want to see similar network speeds when copying a single large VM, etc. However, I was copying hundreds of files on a medium fragmented drive, so 10MB/s isn't unreasonable, IMHO. I'll look at disks in a second.

"You should be getting about ~32meg per second with standard 5400rpm notebook hard drives copying a single large file."

OK, sure I can believe that, assuming there is nothing else going on and there's a nice contiguous spot to lay it down, absolutely. So, let's break this down step by step.

Starting Point

Here's where we start, as possibly useful background, I recently built a house and put some amount of effort into the Home Network.

The house is all Cat 6 wiring, and the main switch is a NetGear 24-port with 20 gigabits of total managed bandwidth. For this test machines are OFF the corporate network and I’ve physically turned their wireless cards OFF. They are single-homed on a basic flat 192.168.x.x network. Nothing fancy.

I brought VS2008 down from the cloud, the files, not the single file large ISO and copied the 3 gigs of files via Explorer Drag Drop between two Vista 32-bit RTM machines and got disappointing results.

No music was playing. The machines are all dual or quad core machines, one with a 10,000 RPM hard drive and the others are pretty fast laptops, so they are beefy machines.

clip_image002

Why mention this? First we eliminate slow or crappy hardware. In this instance, we know it's not CPU that's causing this. That doesn't mean the network cards aren't poo or something, but we gotta start somewhere. Point is, they are more than fast enough to handle the traffic.

The Initial Sub-Optimal Results

So, after the first run, I was getting 10MB/s between a Vista RTM machine (a Mac) and an SP1 machine. When Vista SP1 talks to Vista RTM it'll talk SMB 1.0, the common protocol between them (the one that's been used forever). 

badresults

Of course this isn't a BAD result, as one of the commenters pointed out. I mean, 10 MB/s is more than reasonable most often and this is the kind of thing one would only come up against when copying giant files. Still...now it's a mystery...

Disk Speed

Perhaps Disk Speed is the problem? Commenters on previous posts say Nay, Nay. SATA disks should get 40 to 70MB/s, and that's consistent with my test. The Hanselman-Atwood Ultimate Developer Rig gets 70Mb/s for sequential writes. Of course this assume non-fragmented disks and that nothing else is moving the disk heads around. The MacBookPro and Lenovo t60p get between 20 and 30MB/s, so they are not slouches either. All this points away from Disk Speed.

Here's disk throughput on a laptop:

C:\>disktest.exe -file "Random" -iobytes 8388608 -pipeline 2 -totalbytes 1073741824

---
File name: Random
Total size: 1073741824 bytes
I/O type: read
I/O size: 8388608 bytes
I/O concurrency: 2 (asynchronous)
Buffered: no

---Running the test...---

App. Throughput: 33.2 MBytes/sec (30.9 seconds)
Disk Throughput: 33.2 MBytes/sec (27 samples, 1 high variance)
Min. I/O latency: 250 ms
Avg. I/O latency: 480 ms
Max. I/O latency: 811 ms

So, disk throughput is fine.

Network Cards

In cases like these, it's always a good idea to step back and read the manual. I checked out the release notes for each of my Network Cards and confirmed that they do work on Gigagbit and on Vista and had no issues that smelled like what I am seeing.

One useful tip from Ed Briggs:

"One thing to watch.  On some motherboards, the gigibit NIC (usually a Marvell Yukon 88E8001) is wired to the PCI-33 bus, even if there is a PCI express bus.  I find this on a number of boards where they advertise multiple Gig Eth NICs.  So say, you'll have an NVidia on the PCI-E or the southbridge, and a Marvell on the PCI-33, and your Intel Pro1000 on the PCI-E.  So you'll see some differences there because PCI-33 runs out of steam. "

This wasn't the case for me, but it is very good to know.

Cabling

Even though it shouldn't matter I checked all the cables and confirmed that the house was Cat6. I had been using Cat5 (not 5e) patch cables, so I swapped those out for Cat 5e, and ordered some Cat6 ones just to have.

Switch/Router

Next, I confirmed that my Switch saw these different machines connecting as Gigabit. Here's a screenshot from the Administration Interface of my switch.

NETGEAR Web Smart Switch - Windows Internet Explorer

I'm going between g18, g19, and g21. The administration interface for the switch also shows the length of the cables.

image

Everything is reasonable from a cabling perspective.

Does the Center Hold?

I determined that I must be making some terrible mistake. Rather than pulling and proding, I just started testing with traffic. Here’s a NTttcp run. Note the CPU and MBits. You can get NTttcp in the Windows 2000 Resource Kit.

Total Bytes(MEG) Realtime(s) Average Frame Size Total Throughput(Mbit/s)
================ =========== ================== ========================
     1342.177280      20.649           1424.681                  519.997

Packets Sent Packets Received Total Retransmits Total Errors Avg. CPU %
============ ================ ================= ============ ==========
      942090            83137                 0            0       7.27

Here's a result with two threads. Kind of weird that it's more, but it is over 800 megabits. This implies that the network is generally OK.

Total Bytes(MEG) Realtime(s) Average Frame Size Total Throughput(Mbit/s)
================ =========== ================== ========================
     2684.354560      26.500          18753.481                  810.371

Packets Sent Packets Received Total Retransmits Total Errors Avg. CPU %
============ ================ ================= ============ ==========
      143139           175153                 3            0      15.46

Then I tried

I tried iPerf, the standard cross-platform network throughput tester.

Z:\ntttcp\iperf>iperf -c 192.168.1.14
------------------------------------------------------------
Client connecting to 192.168.1.14, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[132] local 192.168.1.9 port 49642 connected with 192.168.1.14 port 5001
[ ID] Interval       Transfer     Bandwidth
[132]  0.0-10.0 sec   360 MBytes   302 Mbits/sec

Notice the small TCP window size. I'll try again with a larger one.

Z:\ntttcp\iperf>iperf -c 192.168.1.14 -w 64k
------------------------------------------------------------
Client connecting to 192.168.1.14, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[132] local 192.168.1.9 port 49654 connected with 192.168.1.14 port 5001
[ ID] Interval       Transfer     Bandwidth
[132]  0.0-10.0 sec  1.08 GBytes   926 Mbits/sec

Ok, so that's good and implies again that the network doesn't totally suck. Note also that this very good test happened between a Vista RTM machine and a Vista SP1 machine.

Local TCP Settings

Next, I checked the window size on my MSFT Corporate t60p Laptop with Vista SP1 and was shocked to see the tuning level set to highlyrestricted. Not sure who set that but it wasn’t me (that I remember). I switched it back to “normal” but it didn’t affect any of the tests – I ran them again. Everyone is set to normal.

C:\Users\Scott\Desktop\ntttcp>netsh interface tcp show global
Querying active state...
TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Chimney Offload State               : enabled
Receive Window Auto-Tuning Level    : highlyrestricted (should be normal)
Add-On Congestion Control Provider  : none
ECN Capability                      : disabled
RFC 1323 Timestamps                 : disabled

The standard setting of "normal" should work for nearly everyone although some folks swear by "disabled."

Gigabit Jumbo Frames

Marc suggested enabling Jumbo Frames, usually a NIC Driver Level settings. If all your NICs and router/switch support it, you can have frame sizes up to 9014 bytes. However, only to of my NICs supported it, so I didn't bother.

UPDATE: Turns out that Jumbo Frames are obscenely useful. If your network card and router supports them, go for it. More on this at the bottom.

Reintroducing The Players

Right now here’s the machines:

  • Quadpower – The Good Lord's own machine. Truly. nForce chipset. Vista x64 RTM. WEI 5.8
  • MacBookPro – Vista x86 SP1 Beta from last week. WEI 4.9. This machine is no slouch.
  • Lenovo T60p – Vista x64 SP1 Beta from last week. WEI 4.2.  MSFT Domain joined, but not right now.
  • Server – Windows Home Server machine, based on Windows 2k3

The Test Matrix

Here are the four machines in question.

  Quadpower x64RTM MacBookPro x86SP1 T60p x64SP1 WHS 2k3
Quadpower x64RTM

X

<10Mb/s

<10Mb/s

<10Mb/s

MacBookPro x86SP1

X

X

33Mb/s

33Mb/s

T60p x64SP1

X

X

X

33Mb/s

WHS 2k3

X

X

X

X

So it appears the problem is on/around/near/adjacent to Quadpower. When stuff goes into or out of it, it's slower.

Here’s SP1 to WHS (W2k3) copying a 1 gig file of random data.

clip_image002[6]

Here’s SP1 to RTM:

clip_image004

Here’s the Lenovo t60p with Vista SP1 to a MacBookPro running Vista SP1

clip_image006

This is starting to sound familiar. I'm mixing my network protocols, specifically SMB2 vs. SMB1.

"These are some of the key enhancements in SMB 2.0:

  • SMB 2.0 supports an arbitrary, extensible way of compounding operations to reduce round trips. This makes the protocol less chatty as compared to SMB 1.0. Chattiness of SMB 1.0 has often been a major pain point. 
  • SMB 2.0 supports much larger buffer sizes compared to SMB 1.0. 
  • SMB 2.0 greatly grows the restrictive constants in the protocol, so we never need to worry about the protocol itself being the limiting factor for scalability. This includes increasing the number of concurrent open file handles on the server, and the number of shares that a server can share out, among other things. 
  • SMB 2.0 supports durable handles that can withstand short network glitches. 
  • SMB2.0 has support for symbolic links. "

NetMon 3.1 confirms that I'm talking SMB1 between my RTM and SP1 machines and SMB2 between SP1 machines.

Microsoft Network Monitor 3.1

Conclusion

One thing worth pointing out that slowed down my analysis was that I only saw this slowdown when doing a File Copy. I was assuming this was a Network Hardware or TCP issue, perhaps between Vista RTM and Vista SP1. However, the tests above that showed 900+ Mbits/s were actually between and RTM and SP1 machine. Vista Networking works rather fine it seems. It was the older SMB1 protocol being negotiated between my RTM and SP1 machines that was the bottleneck.

I was also prodded to look more into Jumbo Frames. (Thanks Robert G!) Seems my Windows Home Server machine's network card has a Marvell Yukon that supports Jumbo Frames, as does the Marvell Yukon in the MacBookPro. Even though the machines will be using SMB1 as their protocol, the 9104byte frame size makes a massive difference. How much? With a 1 gig file, twice as fast, so 65+ Megabytes a second, even over SMB1. Cool!

10 Seconds remaining

The NVidia nForce hardware chipset supports Jumbo Frames, but the drivers do not yet. NVidia is mum on the issue, which is lame. This the the chipset in my primary machine, the one I'd really like to support Jumbo.

In the Lenovo T60p there is a Intel Pro/1000 PL with an Intel 82573L chipset which apparently doesn't support Jumbo Frames when Active State Power Management (ASPM) is disabled, and since it's in a laptop, it's disabled. No Jumbo Frames on a Lenovo t60p, even with the latest Intel Network Adapter Driver for Vista 64-bit.

"Intel does not plan to resolve this erratum in the 82573 Gigabit Ethernet
Controller.
Jumbo frames is not supported in 82573E/V & is supported with the
workaround above in 82573L."

Regardless, it turns out SMB2 is a way better protocol than SMB1. As far as I'm concerned, problem solved, I'm upgraded to Vista SP1 everywhere and am enjoying Disk Speed on the wire. Now I've gotten speeds up to 33MB/s between machines when copying large files depending on the disk of the machines involved, etc, without Jumbo Frames, and up to 66MB/s with Jumbo. I'll be sure to only get network cards that support Jumbo Frames in the future. I'm happy I went gigabit and I'm happy I went Vista SP1 and I'm happy I went Cat6.

Whew. I was worried I'd have to tear my walls open. ;)

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Saturday, December 01, 2007 2:48:28 AM UTC
Glad you solved it, Scott! That 100MB/sec comment was pretty off the cuff and I should have did some more math/thinking before stating it so confidently. I just took theoretical and lopped 25MB off the speed and came up with 100MB which happens to be 80%. :)
Saturday, December 01, 2007 5:48:15 AM UTC
I was under the impression that Vista RTM supported SMB2. Is this not the case? Is it turned off by default? Is upgrading to SP1 the only solution?

--
Jason
Saturday, December 01, 2007 5:56:28 AM UTC
Jason - I'm looking into it...I'll know soon.
Saturday, December 01, 2007 6:03:41 AM UTC
I think I missed something. In the screen shot for the Lenovo t60p with Vista SP1 to a MacBookPro running Vista SP1 (using presumably SMB2) , you are copying at 33MB/s. When going from Vista SP1 to Windows Home Server (presumably SMB1), it's also 33MB/s. Does not sound like the protocol is the bottleneck.

In the final screen shot, it's 65 MB/s between 2 copies of Vista SP1. How did you from 33 to 65?

Regards
Saturday, December 01, 2007 6:12:43 AM UTC
Where do you get that "DiskCheck" program?
Saturday, December 01, 2007 6:58:20 AM UTC
Where can I find/download the disktest.exe utility that you are using to measure the throughtput?
Krishnan
Saturday, December 01, 2007 8:05:42 AM UTC
Robert...hm...good question...actually those two are using Jumbo Frames (server and the mac), but I'll check again.
Saturday, December 01, 2007 8:47:51 AM UTC
Thanks, Robert - Your were right, it's Jumbo Frames that doubled my last score even though it was using SMB1.
Saturday, December 01, 2007 9:17:10 AM UTC
Regarding slow network issue on Vista, check this out:
http://cs.rthand.com/blogs/blog_with_righthand/archive/2007/10/03/Playing-video-or-audio-in-Vista-is-a-bobby-trap.aspx
Saturday, December 01, 2007 12:59:56 PM UTC
80% of a gigabit network is actually 100 megabytes/second.... ?
Ibleif
Saturday, December 01, 2007 3:31:36 PM UTC
One gigabit = 1 billion bits (1,000,000,000)
Eight bits in a byte
1 billion bits = 125 million bytes (125,000,000)
80% of 125 million bytes = 100 million bytes

So 80% of one gigabit per second is 100 megabytes per second (note that these are powers of 10 megabytes, meaning 1,000,000 bits rather than powers of 2 megabytes, which would be 1,048,576 bits).
Chris
Saturday, December 01, 2007 5:40:49 PM UTC
this is awesome. I too would like to know where to get the disktest program, or is that something you wrote up yourself?
Saturday, December 01, 2007 6:42:14 PM UTC
Scott,

So at the end of the day SMB 2.0 is NOT a speed improvement over SMB 1.0?
Saturday, December 01, 2007 8:02:44 PM UTC
Robert - Oh, no, SMB2 is a *huge* improvement over SMB1. It's reason enough to upgrade to SP1 when available. I got 3x speed over SMB1, so no question. However, Jumbo Frames on fast machines with supported hardware (2 nics + router) was a 6x increase. So, that means the ideal is SMB2 PLUS Jumbo Frames. That'll get me true "disk speed over the wire" performance, IMHO.
Saturday, December 01, 2007 8:48:13 PM UTC
According to the blog entry you referenced regarding SMB2 changes it explicitly lists Vista <-> Vista as using SMB2.

So is the original blog entry written in early 2006 not accurate, or do you have some configuration issue on your Vista RTM machine that is preventing the use of SMB2?

Cheers
Sean McLeod
Saturday, December 01, 2007 9:40:48 PM UTC
Sean - good question! I will ask the team, because I'm definitely seeing SMB1 when talking RTM-RTM.
Sunday, December 02, 2007 3:46:45 PM UTC
Scott,

I went through similar gyrations of testing when I upgraded my home from 100baseT to Gigabit ethernet and was not happy with my first "tests". I also came to similar conclusions as you have, however, an additional "conclusion" of mine was that moving to gigabit was not really "worth it" - certaintly not 10x improvement over 100baseT - more like 2x or 3x. My original 100MB/s network was actually operating pretty darn optimized as it turned out.

On the other hand, as more devices on the network are upgraded over the next few years (faster disks, new OS versions, etc.), I should benefit from the added infrastructure. In addition, all of my switch-to-switch links are now gigabit, which is providing a benefit right now and I did not test/measure. In other words, when the kids are streaming their multi-media around the house, it does not affect as much my own throughput to the source code control server. :-) So I am by no means sorry I upgraded.

Still a mere 1-2Mbit/s out of the house and to the internet for me, however :-( .. No Fios here (yet?), Verizon? Where are you?

Bill T.
Monday, December 03, 2007 5:03:51 AM UTC
Hey Scott, would you mind sharing where you found the disktest.exe application? Great article and congrads on the new house :)
Joe S.
Monday, December 03, 2007 9:33:16 AM UTC
> The standard setting of "normal" should work for nearly everyone although some folks swear by "disabled."

It depends on the network hardware that you are talking to. "normal" works most places I've been to, but I travel a fair bit and I run into the odd hotel or corporate environment where whatever router I am talking to it just stops working after a couple of packets. Switching to "disabled" fixes me. You could argue that it is just working around a bug in the router - which it is - but you can see why some people therefore swear by disabled because that is what works for them. Anyway, as a result, I often find that I'm set to "disabled" most of the time because I will change it while on the road and forget to put it back. Interestingly, when I travel I often go with a bunch of chaps who are very cross-platform. The routers that exhibit this behaviour with the window scaling feature affect the implementations on most modern linux distributions but doesn't effect the Mac - showing (I guess) which Operating System have the most modern standards compliant networking stacks ;-)
Tuesday, December 04, 2007 12:37:47 AM UTC
I get peak rates 65MB/sec also on Gbit. I'm copying from 8 SATA disk RAID 6 to 8 SATA disk RAID 6.
Ben
Saturday, December 08, 2007 7:08:58 AM UTC
Are you signed up for FiOS TV yet? Instalations start on Monday the 10th!
Saturday, December 08, 2007 7:22:52 AM UTC
I called today and they said they start in Portland on January 6th...where do you live? I have an appt that day for two HD DVRs.
Saturday, December 08, 2007 1:10:24 PM UTC
Have you heard back yet about SMB2 with Vista RTM-RTM?
Sean McLeod
Wednesday, December 12, 2007 12:55:13 PM UTC
Thanks for the info. After taking a class for my job put on by the Siemens company I wired my entire house with CAT6 and was quite disappointed in the performance 'jump' from 10/100. Reading this makes more sense. I actually found a bigger performance boost when I went from PATA to SATA drives then when I went from 100 to 1000 network equipment. I will check my intel proset cards for jumbo support, but I don't think it will be forthcoming. Using the terrific NetStat Live (analogx.com) tool with bit settings, I see burst speeds peaked @ 865 Mb/sec. during DVD transfers; with PATA on gigabit, I saw 1/3 of that. This is on an inexpensive NetGear 16 port unmanaged Gbit switch with individual 40ft homeruns to five or six WinXP boxes scattered about my house.

Thanks again.
Don Beal
Sunday, December 30, 2007 6:01:01 PM UTC
Also see these fine articles on gigabit Ethernet, jumbo frames and flow control;

http://www.smallnetbuilder.com/content/view/30212/54/
http://www.smallnetbuilder.com/content/view/30201/54/
http://www.smallnetbuilder.com/content/view/24445/54/
Robert H
Friday, January 04, 2008 11:23:31 PM UTC
excellent network troubleshooting, Scott! I'm in Dundee and getting FiOS TV installed on the 16th - can't wait :)
tomf
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.