Scott Hanselman

There were a number of interesting comments from my post on Faster Builds with MSBuild using Parallel Builds and Multicore CPUs.

Here's a few good ones, and some answers. First,

Is the developer that sits inside of VS all day debugging able to take advantage of the same build performance improvements?

Yes, I think. This post is about my own hack to make it work.

@Scott, there are still people using VS2005 and .Net 2.0 (our employers are either too cheap to upgrade or we work in non-profit[my case]) So it would be nice for posts like this to clearly denote that these are things only available in the 3.0/3.5+ world. Or should I just assume that .Net 2.0 is dead?

Certainly 2.0 is not dead, considering that 3.5 is still built on the the 2.0 CLR. Additionally, if you're a 2.0 shop, there's really no reason not to upgrade to VS2008 and get all the IDE improvements while still building for .NET 2.0SP1 as MSBUild supports multitargeting.

However, the underlying question here is, "why are you teasing me?" as this gentleman clearly wants faster builds on .NET 2.0. Well, this trick under VS2008 still works when building 2.0 stuff.

If you're using a Solution File (SLN) it's automatic, and if you're using custom MSBuild files, you can build .NET 2.0 stuff with the 3.5 MSBuild by putting

<TargetFrameworkVersion>v2.0</TargetFrameworkVersion>

under a <PropertyGroup> in your C# project. Then you can use /m and get multiproc builds. Not much changes though, because either way your build output is running on CLR2.0. (.NET 3.0 and .NET 3.5 use CLR2.0 underneath.) You will get some warnings if you try to use v3.0/v3.5 types.

It's true that parallel managed code compilation lags behind unmanaged.

Welcome, msbuild, to the 21st century. ;-) GNU make has the -j switch (the equivalent to /m here) _at least_ since the early-to-mid 1990's. We use its mingw port to drive Visual Studio 2003's commandline interface to build vcproj files and to handle dependencies, and since we got multi-core build machines, our build times have improved tremendously (we're talking about roughly 2,5 million SLOCs).

There's a number of reasons for that. For example, Visual Basic is compiling in the background all the time while you're typing in VS using a non-reentrant in-proc compiler. The in-proc C# compiler is the same way.

The Multicore MSBuild with VS Hack

This trick is unsupported because it hasn’t been tested. Off the top of our collective head, I can’t think of why it wouldn’t work OK, at least if your solution is all VB/C#. Having VC projects in your solution, especially if there are project references with the VB/C# projects, would probably not work so well. That’s because VC projects are not native MSBuild format in VS2008.

To start with, this is a totally obvious hack. We'll add an External Tool to Visual Studio to call MSBuild, then add some Toolbar buttons.

First, Tools | External Tools. We'll add MSBuild, pointing it to the v3.5 .NET Framework folder. Then, we'll add /m to the arguments for multi-proc and /v:m (where the v is verbosity and the m is minimal) to tone down MSBuild's chattiness. Finally, we'll check the Use Output Window box and set the initial directory to the current Solution's Directory.

Next, we'll right click on the Toolbar, and hit Customize.

Then, from the Commands tab, drag the number of the newly added External Tool (mine is #3) to the Toolbar. I also added a little Lode Runner icon to the button. Additionally, I added a Clean Solution option, just because I like it handy.

You can also re-map Ctrl-Shift-B if it makes you happy, but I used Ctrl-Shift-M, mapping it to the Tools.ExternalCommand, again, in my case it is #3.

At this point, I can now run MSBuild out-of-proc from inside Visual Studio.

What DOESN'T Work

The only thing that doesn't work for me so far is that the errors reported by MSBuild don't appear neatly parsed in the Error List. For some, this may be a deal breaker. However, the good news is that if you double click on the errors in the Output Window you WILL get taken automatically to the right file and line number which is all I personally care about. Still, sucks.

Gotchas for this Hack

If you're building against a SLN file you can use Project References or manually specified project dependencies otherwise MSBuild has no way to figure out the dependencies and you might get "file locked" or other weird concurrency errors.

Also, even if Project to Project references are set up, sometimes project authors are a bit sloppy (I've done this) and inadvertently access the same file in multiple projects, which can cause spurious file locking/access errors. If you see this behavior, turn off multiproc.

Note also that while you can do builds with MSBuild like this, if you get F5 and start a debug session the VS in-process compilers will do a quick build. It WILL figure out that it doesn't need to do anything, but it will check.

Hey, it's NOT faster for me?

If you have a small solution (<10 projects) and/or your solution is designed such that each project "lines up" in a dependency chain rather than a dependency tree, then there will be less parallelism possible.

If you're using a small number of projects, or you're using Visual Basic on small projects, which is compiling in the background anyway, you might not get big changes. Additionally, remember that we're shelling out n new instances of MSBuild, while VS compiles using the in-process versions of the languages compilers. With a small number of projects, in-proc single-core might be faster.

I've got 900 Projects and Visual Studio is Slow, what's the deal?

"Doctor, doctor, it's hurts what I do that!"

"Then don't do that"

Seriously, yes, there are limits. Personally, if you have more than 50 or so projects, it's time to break them out into sub-system SLN files, and keep the über-solution just for the build server.

The team is aware that if you have hundreds of projects in a solution VS totally breaks down. I, and many others, have made suggestions of things like a "tree of projects" where you could build from one point down, and there's ways you can simulate this with multiple-configurations, but it's kinda lame right now if you have a buttload of projects.

Here's a post I did three years ago called How do you organize your code? and I think it still mostly works today. I was using NAnt at the time, but the same concepts apply to MSBuild. Forgive me now, as I quote myself:

Every 'subsystem' has a folder, and usually (not always) a Subsystem is a bigger namespace. In this picture there's "Common" and it might have 5 projects under it. There's a peer Common.Test. Under CodeGeneration, for example, there's 14 projects. Seven are sub-projects within the CodeGeneration subsystem, and seven are the NUnit tests for each. The reason that you see Common and Common.Test at this stuff is that they are the 'highest." But the pattern follows as you traverse the depths. Each subsystem has it's own NAnt build file, and these are used by the primary default.build.

This is not only a nicer way to lay things out for the project, but also for the people on the project, most of whom are not making cross-subsystem changes.

Action Item for you, Dear Reader

If you have feedback then email the MSBuild team at msbuild@ (you know the company name) and they will listen. I warned them you might be emailing, so be nice. If you DEEPLY want multi-proc builds in the next version of Visual Studio, flood them with your (kind) requests. They are aware of ALL these issues and really want to make things better.

It's interesting as I've been at MSFT about 7 months now to realize that for something like multi-proc builds to work it involves (all) the Compiler Teams, the VS Tooling Teams, and the MSBuild folks. Each has to agree it's important and work towards the goal. Sometimes this is obvious and easy, and other times, less so.

Thanks for Dan Moseley, Cliff Hudson and Cullen Waters for their help!

Related Links

• How do you organize your code?
• Faster Builds with MSBuild using Parallel Builds and Multicore CPUs

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By

My one-hundred-and-ninth podcast is up. Recorded at the Microsoft MVP Summit, this show is about the new ASP.NET Dynamic Data subsystem that's being added to ASP.NET. I sit down with Scott Hunter, a Program Manager for ASP.NET (everyone in the division is required to be named Scott, BTW ;) ) and dig deeper into Dynamic Data.

• Dynamic Data announced on The Gu's Blog
• David Ebbo's Dynamic Data Screencast
• Putting ASP.NET Dynamic Data into Context
• Brad Abrams on a new build of Dynamic Data

Subscribe:

• Download: MP3 Full Show
• Torrent: Download via µtorrent
• ACTION: Please vote for us on Podcast Alley! Digg us at Digg Podcasts!

If you have trouble downloading, or your download is slow, do try the torrent with µtorrent or another BitTorrent Downloader.

Do also remember the complete archives are always up and they have PDF Transcripts, a little known feature that show up a few weeks after each show.

Telerik is our sponsor for this show.

Check out their UI Suite of controls for ASP.NET. It's very hardcore stuff. One of the things I appreciate about Telerik is their commitment to completeness. For example, they have a page about their Right-to-Left support while some vendors have zero support, or don't bother testing. They also are committed to XHTML compliance and publish their roadmap. It's nice when your controls vendor is very transparent.

As I've said before this show comes to you with the audio expertise and stewardship of Carl Franklin. The name comes from Travis Illig, but the goal of the show is simple. Avoid wasting the listener's time. (and make the commute less boring)

Enjoy. Who knows what'll happen in the next show?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By

UPDATE: I've written an UPDATE on how to get MSBuild building using multiple cores from within Visual Studio. You might check that out when you're done here.

Jeff asked the question "Should All Developers Have Manycore CPUs?" this week. There are number of things in his post I disagree with.

First, "dual-core CPUs protect you from badly written software" is just a specious statement as it's the OS's job to protect you, it shouldn't matter how many cores there are.

Second, "In my opinion, quad-core CPUs are still a waste of electricity unless you're putting them in a server" is also silly. Considering that modern CPUs slow down when not being used, and use minimal electricity when compared to your desk lamp and monitors, I can't see not buying the best (and most) processors) that I can afford. The same goes with memory. Buy as much as you can comfortably afford. No one ever regretted having more memory, a faster CPU and a large hard drive.

Third he says,"there are only a handful of applications that can truly benefit from more than 2 CPU cores" but of course, if you're running a handful of applications, you can benefit even if they are not multi-threaded. Just yesterday I was rendering a DVD, watching Arrested Development, compiling an application, reading email while my system was being backed up by Home Server. This isn't an unreasonable amount of multitasking, IMHO, and this is why I have a quad-proc machine.

That said, the limits to which a machine can multi-task are often limited to the bottleneck that sits between the chair and keyboard. Jeff, of course, must realize this, so I'm just taking issue with his phrasing more than anything.

He does add the disclaimer, which is totally valid: "All I wanted to do here is encourage people to make an informed decision in selecting a CPU" and that can't be a bad thing.

MSBuild

Now, enough picking on Jeff, let's talk about my reality as a .NET Developer and a concrete reason I care about multi-core CPUs. Jeff compiled SharpDevelop using 2 cores and said "I see nothing here that indicates any kind of possible managed code compilation time performance improvement from moving to more than 2 cores."

When I compiled SharpDevelop via "MSBuild SharpDevelop.sln" (which uses one core) it took 11 seconds:

TotalMilliseconds : 11207.7979

Adding the /m:2 parameter to MSBuild yielded a 35% speed up:

TotalMilliseconds : 7190.3041

And adding /m:4 yielded (from 1 core) a a 59% speed up:

TotalMilliseconds : 4581.4157

Certainly when doing a command line build, why WOULDN'T I want to use all my CPUs? I can detect how many there are using an Environment Variable that is set automatically:

C:>echo %NUMBER_OF_PROCESSORS%
4

But if I just say /m to MSBuild like

MSBuild /m

It will automatically use all the cores on the system to create that many MSBuild processes in a pool as seen in this Task Manager screenshot:

The MSBuild team calls these "nodes" because they are cooperating and act as a pool, building projects as fast as they can to the point of being I/O bound. You'll notice that their PIDs (Process IDs) won't change while they are living in memory. This means they are recycled, saving startup time over running MSBuild over and over (which you wouldn't want to do, but I've seen in the wild.)

You might wonder, why do we not just use one multithreaded process for MSBuild? Because each building project wants its own current directory (and potentially custom tasks expect this) and each PROCESS can only have one current directory, no matter how many threads exist.

When you run MSBuild on a SLN (Solution File) (which is NOT an MSBuild file) then MSBuild will create a "sln.cache" file that IS an MSBuild file.

Some folks like to custom craft their MSBuild files and others like to get the auto-generate one. Regardless, when you're calling an MSBuild task, one of the options that gets set is (from an auto-generated file):

<MSBuild Condition="@(BuildLevel1) != ''" Projects="@(BuildLevel1)" Properties="Configuration=%(Configuration); Platform=%(Platform); ...snip... BuildInParallel="true" UnloadProjectsOnCompletion="$(UnloadProjectsOnCompletion)" UseResultsCache="$(UseResultsCache)"> ...

When you indicate BuildInParallel you're asking for parallelism in building your Projects. It doesn't cause Task-level parallelism as that would require a task dependency tree and you could get some tricky problems as copies, etc, happened simultaneously.

However, Projects DO often have dependency on each other and the SLN file captures that. If you're using a Visual Studio Solution and you've used Project References, you've already given the system enough information to know which projects to build first, and which to wait on.

More Granularity (if needed)

If you are custom-crafting your MSBuild files, you could turn off parallelism on just certain MSBuild tasks by adding:

BuildInParallel=$(BuildInParallel)

to specific MSBuild Tasks and then just those sub-projects wouldn't build in parallel if you passed in a property from the command line:

MSBuild /m:4 /p:BuildInParallel=false

But this an edge case as far as I'm concerned.

How does BuildInParallel relate to the MSBuild /m Switch?

Certainly, if you've got a lot of projects that are mostly independent of each other, you'll get more speed up than if your solution's dependency graph is just a queue of one project depending on another all the way down the line.

In conclusion, BuildInParallel allows the MSBuild task to process the list of projects which were passed to it in a parallel fashion, while /m tells MSBuild how many processes it is allowed to start.

If you have multiple cores, you should be using this feature on big builds from the command line and on your build servers.

Thanks to Chris Mann and Dan Mosley for their help and corrections.

Related Links

• MSBuild Blog - The MSBuild team's blog.
• MSBuild Sidekick v2 - A visual editor, ala NantPad, for MSBuild files.
• MSBuild Visualizer - An old project of Mitch Denny's trying to make large MSBuild files easier to visualize.
• How to get MSBuild building using multiple cores from within Visual Studio

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By

Have you ever looked at a word you use and see every day, like the word "What," and suddenly start doubting yourself and wonder if it's spelled correctly? There you go. What. Is THAT how it's spelled? That doesn't look right. Why would such a simple thing suddenly be called into question?

Anyway, there's a glut of technical books out there, and I'm starting to realize that I take their style and their existence for granted. They usually describe what some product is and what it does.

I really enjoy books/blogs and writing that spend less time on the What and more time on the So What? and Now What? and What For? I'd like to see more books that put technologies into a larger context and less on just What.

It'd be interesting to take any book title and add one of these phrases to the end, like "Professional ASP.NET 2.0, So What?" or "Ruby on Rails, Now What?" or "SQL Server, What For?"

What For?

The Ruby Way does a good job explaining What Ruby is Good For. It answers the Why, to me at least, very well. It's the best book I've found that doesn't just teach Ruby syntax, but also Ruby idioms; it's mastering idiomatic structures that brings fluency in any language, human or computer.

Programming Pearls - I used this book while teaching my C# Class at OIT even though it had nothing to do with C# because it's just that good. It's a series of collected articles from Communications of the ACM. It helped me understand what a number of things were for. I better understand computer problem solving, what Math is for (its relationship to programing, which isn't always clear to everyone)

How to be a Programmer (free) by Robert Read - This is a fun, short read that is general enough that it makes sense in most languages. If you want a CS101 (or 201) practical refresher, this is a great book that answers mostly "how to" questions, but ultimately explains what a number of technique are for.

So What?

Dissecting a C# Application: Inside SharpDevelop is a fantastic book and one of my favorites because it's a story that's full of So What? answers. They talk about what they tried, what failed, what didn't and how they moved forward. The Reviews on Amazon aren't all favorable as one reviewer says this is "an extremely difficult book from which to learn" but if you learn, as I do, from reading source with explanations of design decisions, then this (slightly old, but still useful) book is worth your time.

Programming WPF by Chris Sells and Ian Griffiths (his blog is a technical joy) is great because the authors are always trying to answer these "so what" questions. Chris is like this in real life. "So What?" is, I think, his number one question as he takes little for granted. If you read Ian's blog, you know his knowledge is deep to the point of being obscene (that's a good thing) and when you put these two guys together you get a great book that answers, for me, WPF, So What?

Now What?

OK, after I've learned a technology, Now What? What's the next step in my learning and what should I do?

The Pragmatic Programmer by Andrew Hunt and Dave Thomas is a book that answers a lot of Now What questions for me. It shows a complete, pragmatic attitude towards the software development lifecycle. I wouldn't say it's complete in an Enterprise sense, but good for medium-size projects, and most of the general kinds of things you're going to bump into.

In May, my friend and long-time compatriot Patrick Cauldwell will release Code Leader: Using People, Tools, and Processes to Build Successful Software which I read, enjoyed and happily wrote a foreword for. This is a book based on Patrick's "This I believe: The Developer Edition" post of last year. He takes many different tools and processes and answers the Now What? question.

Making Things Happen by Scott Berkun is an update to The Art of Project Management and I now have both editions. Scott answers lots of the Now What questions in a comfortable, conversational tone with a deep focus on getting things done in software development. He helps with planning and execution.

Your Turn

This is a woefully short list. Perhaps I'm missing something, but other than actually doing it (and failing and trying again) there aren't a lot of books that describe how to build large, complete applications that support the entire software development lifecycle.

What books have you read that answer these So What, Now What, and What For questions?

Related Links

• Six Essential Language Agnostic Programming Books

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By

I was at the ALT.NET Open Spaces Conference in Seattle last week. I LOVE Open Spaces conferences, my first was Foo Camp, and I've been to two ALT.NET Conferences. When Open Spaces is done right, it's a brilliant self-organizing thing of beauty. If you want to hire an Open Spaces Facilitator for your own conference, you might consider Steven "Doc" List. He did a fantastic job keeping us on track and keeping the energy level both high and positive.

These are the four Open Spaces principles:

• Whoever comes are the right people
• Whatever happens is the only thing that could have
• Whenever it starts is the right time
• When it's over, it's over

if it sounds a little meta-physical, it is, but it works. Anyone can convene a space/talk and anyone can participate. It's less about personality and power and more about people and discussion and it matches the ALT.NET way of thinking perfectly.

Dave Laribee coined the term ALT.NET last year. I did a podcast on ALT.NET with Dave last month. A few weeks before this year's conference even began, Jeremy Miller blogged about the need for ALT.NET. In a nutshell he said:

Specifically, I’d like to see ALT.NET start to fill what I see as a void for information and leadership on:

• OOP fundamentals. I think we as a community would achieve far more benefits on our projects from a stronger base of OOP design knowledge than we do for learning new API’s like WCF or LINQ. Learning the fundamentals will make us far better able to choose and use these new API’s.
• Development processes and practices. Microsoft builds and sells tools, and they naturally see tools as the solution to most problems. We the community should be completely in charge of determining our best practices because we are the ones out there building software. Go to a .Net centric development conference. Outside of some token Agile track, you will not see very much about configuration management, project management, requirements, testing, or the all important people issues. A lot of Linq to XYZ talks, but not much about running successful software projects.
• Alternative tools. Some of the tools Microsoft makes are great, but some are inferior to non-Microsoft options. Some of these alternative tools open up entirely new possibilities in how we can work.  I’d like to see the entire .Net community to simply broaden its horizons to be more accepting of tools originating from outside of Redmond.

I think he’s right on. Not everything MSFT does is the “last word,” Microsoft can do better on prescriptive guidance and process, and many .NET programmers could do to refresh their knowledge of Computer Science 101 Fundamentals, myself included.

Also, ALT.NET isn't about Microsoft vs. The World, as Ayende pointed out last year:

Saying things like "An ALT.NET developer would be using Castle Windsor before Enterprise Libraries ObjectBuilder.", or "An ALT.NET developer was using NHibernate before the Entity Framework." is giving the wrong impression. It gives the impression of you are either with us (good) and against us (bad). And you must follow Our (notice the royality speak) way and no other.

The other objection is that it is focusing on tools and not on a mind set. The way I see it, this is much more about keeping your head open to new approach and ideas, regardless of where they come from. In short, I really like the ideas and concepts that Dave presents, I don't want the idea to turn into "A .NET developers that seeks to use non Microsoft technologies." I would much rather it be "A developer that seeks to find the best tools and practices, and judge them on merit."

I think some folks at Microsoft perceive the ALT.NET crowd as being loud, small, divisive, or all of the above. Deep passion and belief can sometimes be perceived as loud or argumentative. I think a better way to put it would be "pragmatic." More and more MSFTies, ScottGu included, get this. I enjoy being a part of this group.

ALT.NET is about picking processes and tools that work for you and make you happy, picking the best one or a number of them, and using them all together, so I took a moment this afternoon and whipped up an idea that a bunch of us at the conference had late one night: The ALT.NET Geek Code.

Here's mine (note the DIV tag and CLASS in the source. Any designers want to whip up a nice CSS box, with a small ALT.NET Logo? Thanks to Simone for the nice CSS and layout and Adam Kinney for the Silverlight Badge!)

What's it mean? Well, it's a code that describes the processes and tools that I like to use to develop (as of this moment...these things tend to evolve.) You can get your own ALT.NET Geek Code and post it on your blog if you like. Perhaps we'll be able to use Google to scrape the Interwebs and do a distributed ALT.NET Tools Survey someday. ;)

Should you care about ALT.NET? Even if you aren't into the cult of personality, you can definitely learn something from getting involved in the conversation.

UPDATE: I also want to point out that I'm intending to use the term "Guys" in this post's title in the gender-neutral usage. I am acutely aware of the need for more women in this field, so much so that I convened a talk at this year's ALT.NET Open Spaces Conference called "What's with all these White Guys" which was attended by the both the women who came to the conference this year.

Related Links

• ALT.NET on Flickr : http://flickr.com/photos/tags/altdotnet
• ALT.NET on Twitter: http://hashtags.org/tag/altdotnet/

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By