Scott Hanselman

NuGet for the Enterprise: NuGet in a Continuous Integration Automated Build System

May 25, '11 Comments [26] Posted in NuGet | Open Source
Sponsored By

NuGet: Microsoft .NET Package Management for the EnterpriseI had the pleasure of speaking at TechEd 2011 North America last week in Atlanta. You can see ALL the videos of all the sessions on Channel 9. As an aside, you might notice that they are in the process of organizing video archives of ALL Microsoft developer events at http://channel9.msdn.com/Events. You can even see PDC 1999 if you like or see sessions by Speaker at http://channel9.msdn.com/Events/Speakers. Here are all my talks with a horrible headshot that I plan on asking Duncan to swap out ASAP.

My favorite talk was NuGet: Microsoft .NET Package Management for the Enterprise. I talked about NuGet, like I did in The Netherlands a few weeks ago, except the TechEd talk was focused much more on how NuGet fits into the software development lifecycle in a diverse Enterprise (or big boring company, if you prefer) environment.

Here's the video downloads, or you can click the slide at the right.

At my last company, we used Subversion for source control and CruiseControl for Continuous Integration (CI). I thought it'd be nice to setup a similar system using the latest free (and mostly free) tools. Note, you can do all this with TFS as well for both Source and Build. I'll do a post on that later. For now, I give you:

Setting up NuGet to build using Mercurial for Source Control and JetBrains TeamCity for Continuous Integration while pushing to a local Orchard NuGet Gallery Server

Oh yes, that's a long H3 right there but it's descriptive, right? Here's the general idea.

Progession Diagram: Source, Build, NuGet, Gallery

This of course, is not unique to NuGet, as NuGet is just a build artifact. At my last company we had several things that popped out of the build. Not just the DLLs, but also a ZIP file, MSI installer and even a complete configured and prepped Virtual Machine for sales people to pick up and give demos with our latest bits. You can setup your Continuous Integration system to be as awesome or as simple as you like. You should be thinking about which libraries and parts will create NuGet packages.

Another thing to think about is daily (or every build) packages vs. stable or release packages. There are some discussions on the NuGet site around a "-beta" switch and baked-in support for understanding stable vs. volatile builds. For now, consider two locations for your builds, one for each build and one for "blessed." For some, this might mean a folder for dailies and only blessed go to a server.

Here's the demo I did. You can change what you like and swap out for your favorite tools. I'll point out some gotchas and issues that hit me and might hit you. It's not perfect, but we're moving in the right direction.

Step 0 - Prep Stuff

Some of these steps are "make sure x is setup" type steps and can happen whenever, so don't take the ordering of the steps as totally crucial. Here's what I used and installed.

Step 1 - Make sure your project builds and you can make a NuGet package (nupkg)

I'm doing all this on one laptop, but you might have things spread out at your company. Do what makes you happy.

I made a trivial little class library called TechEdLibrary and confirmed it builds with MSBuild like this:

C:\dev\techedlibrary\TechEdLibrary>msbuild TechEdLibrary.csproj
Microsoft (R) Build Engine Version 4.0.30319.1
[Microsoft .NET Framework, Version 4.0.30319.225]
Copyright (C) Microsoft Corporation 2007. All rights reserved.

Build started 5/24/2011 4:52:54 PM.
..snip...
Done Building Project "C:\dev\techedlibrary\TechEdLibrary\TechEdLibrary.csproj"

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:00.13

The most important part is to make sure that your AssemblyInfo.cs is actually filed out and not just the defaults. This is because we'll want to update the NuGet's specification file using the values from the DLL and project. Basically we want the metadata of the project to drive the NuGet package (or we have to update it all manually.)

I need a spec file. I can do this a few ways. I can do it manually with just "nuget spec," I can build it off the resulting DLL with "nuget spec -assemblyPath bin\debug\techedlibrary.dll" or off the csproject with "nuget spec techedlibrary.csproj."

If I create it off the csproj, the NuGet spec file will be created with some $tokens$ that will be filled out at packaging time:

C:\dev\techedlibrary\TechEdLibrary>nuget spec TechEdLibrary.csproj
Created 'TechEdLibrary.nuspec' successfully.

In the talk at TechEd I mistakenly build it off the DLL and ended up hard-coding the versions. In a continuous integration system you'll want to update the version in your spec as the build versions. You can either do it with tokens like I will in this post, or you can pass in the version (often from an environment variable) into the command line like "NuGet Pack -version 1.0.1.0."

You can also  use the UpdateVersion.exe that Matt Griffith and I updated SO many years ago to change your AssemblyInfo.cs, then the NuGet package can pick it up. Again, there is no right answer, the point is to have a strategy. What drives the version and how does NuGet find out about it?

My TechEdLibrary.nuspec I just made looks like this:

<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
<metadata>
<id>$id$</id>
<version>$version$</version>
<authors>$author$</authors>
<owners>$author$</owners>
<iconUrl>http://www.hanselman.com/images/nugeticon.png
</iconUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance>
<description>$description$</description>
</metadata>
</package>

And my AssemblyInfo.cs (abridged) is like this. The attributes from the DLL will get copied into the $tokens$ and packaged.

[assembly: AssemblyTitle("TechEdLibrary")]
[assembly: AssemblyDescription("This is my cool library")]
[assembly: AssemblyCompany("Scott Hanselman")]
[assembly: AssemblyProduct("TechEdLibrary")]
[assembly: AssemblyVersion("0.9.*")] //The * means I'll get an automatic version bump
//and more...

Now I can pack it up with NuGet Pack TechEdLibrary.csproj. Note the command line output find the spec and does the token/metadata replacement?

C:\dev\techedlibrary\TechEdLibrary>nuget pack TechEdLibrary.csproj
Attempting to build package from 'TechEdLibrary.csproj'.
Building project for target framework '.NETFramework,Version=v4.0'.
Packing files from 'C:\dev\techedlibrary\TechEdLibrary\bin\Release'.
Using 'TechEdLibrary.nuspec' for metadata.
Successfully created package 'C:\dev\techedlibrary\TechEdLibrary\TechEdLibrary.0.9.4161.28882.nupkg'.

Now I can just open the .nupkg into the NuGet Package Explorer and see the version, author, ID (from Title) description and version are all there, brought in from the attributes and combined with my NuSpec. I can edit the NuSpec to taste as long as I maintain the $tokens$ I want.

My package in the NuGet Package Explorer

Now if I build the library again I'll get a new version from the .NET build system and that will cause a new NuPkg to be built with a new version. I make a change to my application's source, build again, then pack again. Note the results in my directory after I make a small change, build and pack.

C:\dev\techedlibrary\TechEdLibrary>msbuild TechEdLibrary.csproj 
Microsoft (R) Build Engine Version 4.0.30319.1
Build started 5/24/2011 5:07:58 PM.
blah build blah
Done Building Project "C:\dev\techedlibrary\TechEdLibrary\TechEdLibrary.csproj" (rebuild target(s))

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:00.32

C:\dev\techedlibrary\TechEdLibrary>nuget pack TechEdLibrary.csproj
Attempting to build package from 'TechEdLibrary.csproj'.
Building project for target framework '.NETFramework,Version=v4.0'.
Packing files from 'C:\dev\techedlibrary\TechEdLibrary\bin\Release'.
Using 'TechEdLibrary.nuspec' for metadata.
Successfully created package 'C:\dev\techedlibrary\TechEdLibrary\TechEdLibrary.0.9.4161.29041.nupkg'
.

C:\dev\techedlibrary\TechEdLibrary>dir *.nupkg

05/24/2011 05:07 PM 4,770 TechEdLibrary.0.9.4161.28882.nupkg
05/24/2011 05:08 PM 4,763 TechEdLibrary.0.9.4161.29041.nupkg
2 File(s) 9,533 bytes

Without settings really anything up but the versioning plan, spec file and checking if packing works, I've got a little system here and hopefully you can see how it'll work.

WEIRDNESS NOTE: We do have a double-build going on. NuGet.exe, for some weird reason, is running MSBuild again for us. That's lame, and a known bug. Surprisingly in a large number of CI systems in both the .NET and Java worlds "double builds" are common. Weaksauce, but common. Still, no excuse. That'll be fixed in NuGet.exe soon.

Step 2 - Initial Source Check-in

I like using BitBucket for small private projects and CodePlex for public Open Source projects. Both support Hg (the elemental symbol for "Mercury" as in Mercurial) and CodePlex supports TFS for both Source and Work Items. Since my demo is private and BitBucket supports unlimited private projects it was a good fit.

I cloned my project URL from BitBucket into a folder, then added, commited and pushed my bits to the BitBucket Site:

C:\dev\something>hg clone https://shanselman@bitbucket.org/shanselman/techedlibrary
...move my files in...
C:\dev\something>hg add
...yada yada yada...
C:\dev\something>hg commit
...yada yada yada...
C:\dev\something>hg push
...yada yada yada...

It's useful then to make sure I can get my source code into another folder and still build it. It's common to miss a file or two. Since the CI Server will be getting the code into a temporary folder you really need to make sure the source will build as it is, retrieved fresh from your SCM.

Step 3 - Setup your Build Server

I'm running TeamCity locally on http://localhost:8111/ with a "build agent" (there can be many) on the same machine. Visit Administration and make a new project, then a new Build Configuration (like Debug or Release).

You'll need to make a VCS root (Version Control System - there are like 60 different acronyms for version control systems. If you see a TLA (Three Letter Acronym) out there that you don't recognize, it's probably something that means "Source Control.") in TeamCity. Note that I selected Mercurial and set the HG Command Path with the full path to HG.EXE as the CI will call that to check out the code from BitBucket. Note also that I put the path without my name and password for the "pull changes from" as I put the name and password lower down.

Editing a VCS Root in TeamCity

Back in Administration, select Edit under your project. Note the nice lists of steps on the right.

The 7 steps in TeamCity

We want two steps, one for the MSBuild and one "custom command line" for the NuGet package step. The first step is easy, we call MSBuild on our TechEdLibrary.csproj.

The second step is a temporary hack. It's temporary because JetBrains TeamCity is building NuGet support in directly (screenshots of their internal build below!)

TeamCity Custom Scripts

The custom script is basically a BATCH file that looks like this:

del *.nupkg 

NuGet pack techedlibrary.csproj
forfiles /m *.nupkg /c "cmd /c NuGet.exe push @FILE yourapikey -source http://localhost:81/

There's a few interesting things going on here.

First, I delete all the nupkg files in the CI folder as it's all temporary and we don't want to accidentally push old stuff.

Second, I pack up the NuGet package like we saw before.

Finally, I don't know the name of the newly created *.nupkg file, so I cheat by making a DOS BATCH "for loop" that has only one item in it, the newly created .nupkg file! Then I call cmd.exe execute NuGet.exe pack with that new file as a parameter. Make sure you have the trailing backslash.

NOTE: You can also save the API Key in local storage with NuGet SetApiKey yourapikey -source http://localhost:81 and it'll be saved on a per-source basis.

Because I have an Orchard NuGet Gallery running locally, I have an API key for that server. I'm running my gallery server (it holds the packages and serves the OData Feed) locally on port 81 and the Orchard Site itself is on port 80.

My personal NuGet Gallery

I set the build to check Source Control every 60 seconds, if a change is detected, the source is retrieved, build, then NuGet packed, and published to my local server (or the public one).

TeamCity is something I like to have running on extra hardware I've got lying around. You can have up to 20 projects with their free version, so when combined with BitBucket and CodePlex where I keep my projects I can setup my own Continuous Integration System in just a few hours and have better confidence in my code. You can even use Amazon EC2 images to run your builds or just be a build agent.

Sneak Peak - TeamCity with NuGet Integration

I'm continuing to talk to companies and software vendors who are jazzed about NuGet. If you are one, watch my talk for ideas on what you can do as a commercial software entity and get into the discussion on CodePlex.

Here's some mockups of what JetBrains is planning for TeamCity. These are just ideas, and they aren't available, so don't stress or pressure them. ;)

This is a mockup of NuGet as a possible Build Step within TeamCity.

This is a mockup of NuGet as a possible Build Step within TeamCity.

Here's a mockup of a Build Feature where TeamCity downloads specific packages that will be needed by the build.

Here's a mockup of a Build Feature where TeamCity downloads specific packages that will be needed by the build.

Here's a screenshot of some early work that Martin Woodward from the TFS team is doing to make sure NuGet and Team Foundation Server work well together in a Continuous Integration environment. Feel free to contact Martin via his blog and continue the discussion on http://nuget.codeplex.com.

TFS and NuGet

Hope this helps you integrate NuGet into your company, however you like it.

Related Links

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. I am a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Wednesday, May 25, 2011 2:00:14 AM UTC
The sauce is strong with this one.
BrianHartung
Wednesday, May 25, 2011 2:05:13 AM UTC
Sweet!
Wednesday, May 25, 2011 2:54:21 AM UTC
"Here's some mockups of what TeamCity is planning for JetBrains."

Other way around...and thanks for working with JetBrains...they're awesome as their TeamCity
a
Wednesday, May 25, 2011 3:11:12 AM UTC
This was a good session Scott. I spent the whole day Monday back at work trying to find the right person in my enterprise to talk to about getting me some hosting space at the main data center. I've been modifying Orchard now to work with our corporate Single Sign-On so anyone in the company could use it. I'm trying to setup a NuGet and Orchard instance following the rules right now but next week may be a different story.

If I have no luck in a week or two then I'll resort to setting it up on our own web servers until I can get some corporate web hosting space.
Wednesday, May 25, 2011 4:04:09 AM UTC
How would you go about handling multiple .Net versions. Within my company (and I'd assume mange enterprises) we need to support both .Net 4 and .Net 3.5 within a single package.

We have a solution involving running the build for each .Net version and a lot of msbuild foo!

MSbuild community extensions also has some very handy tools for doing regular expression and XPath based replace operations that we employed to set the version etc in our .nuspec files.
Wednesday, May 25, 2011 5:51:59 AM UTC
Tim Savage - Take a look at the NuGet Docs. You can make a single NuPkg file with multiple (and I mean LOTS) versions in structure lib folders. The best example is the Newtonsoft.JSON package. Download it and unzip it. You would make your CI server prep a folder and get it all laid our correctly, then build a single package.
Wednesday, May 25, 2011 6:25:56 AM UTC
Teamcity and Nuget are awesome and together they will make for a good pair.

Do the teched WMV videos also contain the speaker? It's not as nice to only see the slides and hear the voice.

Their isn't enough photoshop to make your headshot any better ;-).
Wednesday, May 25, 2011 6:50:37 AM UTC
These are the kind of posts that are really useful in the "real" world. Thanks, Scott. I've been watching your talk from the video and found it quite interesting.

We've been using CI at work for several years now (TeamCity with SVN) and it's a completely must-have scenario for the enterprise (change SVN with your preferred SCM). Nowadays we're also changing some of our local libraries to be served from a local Nuget Server (a read-only one for now...), so we have more or less the same architecture you introduced in your talk/post.

But I'm not really sure if creating a new package for every single commit to SVN is a good practice for us. We can have several commits from several developers every day, so we may end up with tens of new packages every day as well!! Maybe creating a specific build configuration to package only when the developer decides the version is stable enough could do the trick, but we already have a batch file in the project to do so...

The only advantage I find in doing the package from TeamCity is that you can set up the specific version for it, according to the rev number, so that the version "makes sense", not like the autogenerated one :-( and then matches de .dll version, which we change with the rev number (e.g. 3.0.267, and not 3.0.4156.16736)

Anyway, great job. This a good point to start using Nuget at work if you still don't, and of course, use CI!!!!

Wednesday, May 25, 2011 8:29:20 AM UTC
I have set up the simple package server on an internal server, it was very easy to do and it looks like a great way to make internal tools available.

We are looking to use NuGet for reusable code. Using Subversion externals doesn't work too well. The difficult part is developing the code in such a way that it can be tested during development, and then packaged using NuGet. This is difficult if you want to reuse code from a package in Web Site vs. Web Application projects.

Should it go into App_Code or not?
How do I make a .pp file from the original .cs files, and ho do I introduce $tokens$ when packaging?

If you need to package code with $tokens$, then the original code is not valid, it can't be run or tested. You would need to package, deploy into a different project, run, test. Repeat. Or am I missing something?

More general: I'd love a post on how to distribute source in a package for reuse, especially for web projects and with the use of $tokens$.

Anyway, thanks for this post, lots of good information!
Mike
Wednesday, May 25, 2011 12:17:07 PM UTC
Great post... I can't wait to see the TFS example, in the meantime I'll be trying get the TFS way working myself :-)
Stif
Wednesday, May 25, 2011 3:54:45 PM UTC
Nice demo, but I was a bit disappointed to see that your package doesn't have any dependencies. Where can I find a demo that explains how NuGet dependencies work in a CI environment?
Wednesday, May 25, 2011 4:22:34 PM UTC
Sergi - Then create a NuGet package only for source that has been labeled with a certain manual label...that's a way of controlling an explosion of build artifacts.

Robert - If it had NuGet dependencies, I could have used NuGet.exe to update from its packages.config and download any packages it needed. Same with any references, no different from any existing CI system.

Mike - Before I write anything new up, have you looked at http://docs.nuget.org?
Wednesday, May 25, 2011 4:28:22 PM UTC
NuGet still has one huge bug that, while irrelevant to most hobbyists, it's fatal in most enterprises. That bug is that it's possible to package an assembly that doesn't have a strong name. Combine that with the requirement that an assembly can't be signed unless all of it's dependencies are signed (duh! of course), and NuGet becomes 100% incompatible with anything that needs to be signed. NuGet could have been a great dev tool, but sadly one small feature has it crippled to the point where it's useless for any serious development. That being said, I will still keep it installed for playing around with stuff that will never go to production.
Wednesday, May 25, 2011 6:48:00 PM UTC
John - That's actually a CLR issue, not a NuGet issue. NuGet is just a packaging system. Regardless of NuGet, if you sign an assembly, it can't refer to an unsigned one. I'm not sure how NuGet is crippled as it's orthoganal to the CLR issue. You're saying that it's possible to package a non-signed assembly. Sure. How would you fix that?
Thursday, May 26, 2011 12:09:33 AM UTC
Great talk Scott - you're such a command line freak (I mean that lovingly of course haha). I shared your talk with a couple clients of mine to show them how they can use NuGet in their enterprise. So far it's been very well accepted, which is great to see.
Thursday, May 26, 2011 7:04:28 AM UTC
@Scott

I haven't read those completely. But from what I can tell by skimming the docs, they are docs that explain what you can do. They don't have a section on "how to deal with the difference between Web Site and Web Application projects when choosing a location for your source files" or "how to develop and test code for a package that will contain Visual Studio project properties delimited by dollar signs which will normally result in [Identifier expected] and [Unexpected character '$'] compiler errors"


Mike
Thursday, May 26, 2011 7:35:24 AM UTC
Mike - Huh? How can I help you?
Scott Hanselman
Thursday, May 26, 2011 1:12:17 PM UTC
Scott, i have this issue:
previously i had a nuspec file with everything specified (name, author and regex for files) and i was packing everything i wanted to with "nuget pack myproject.nuspec".
Right now i just tried to follow your example but i realized that packaging was including the files twice... how can i deal with that ? because this ends ups with the following error "Cannot add part for the specified URI because it is already in the package".
And i definetely don´t want to include everything in the project, i need to have some control over files but i would like to use $token$.

Does this make sense to you ?

Great blog anyway !
Friday, May 27, 2011 5:27:10 PM UTC
German - I've had the same issue. So far, as I understand it, you need to use the csproj if you want to use the tokens. Let's move this conversation over to http://nuget.codeplex.com and see if we can find a way to have the best of both worlds.
Friday, May 27, 2011 5:45:03 PM UTC
Scott,

I'm not sure how I would fix that, now that NuGet is out in the wild. It's a general purpose tool whose primary use is distribution of .NET assemblies. Adding a rule to the package specs to say that any NuGet package that includes .NET assemblies that aren't strong-names is an invalid package would be ideal. Getting most popular open source communities to strong-name their assemblies before packaing them would similarly overcome the issue.

The point I was trying to make, and maybe I went a little rant-y and clouded it up, is that if I'm pulling dependencies into a signed assembly today, I can't use NuGet to help me. How many open source projects offer a download with strong-named binaries? Not many, because if you want strong names you can pull down the source, add your key to the proj file and recompile. The whole idea behind NuGet is to offer a simpler alternative. A single command pulls in a library along with any dependent libraries, and another single command updates all those external dependencies in an entire solution. Well that would be super-efficient if those NuGet packages came down with signed assemblies, but when they don't you're back to the pre-NuGet way of doing things.

By being a general-purpose tool instead of a .NET specific tool, it's handicapped itself in a way that severely limits it's usefulness as a .NET tool.
Friday, May 27, 2011 6:08:32 PM UTC
John - NuGet being "in the wild" doesn't mean it's not changing. It's auto-updating and releasing every month. Just type NuGet -u and it's updated. There's no reason we can't all talk over at NuGet.Codeplex.com and come up with solutions that work. Could be a command line switch, could be a global setting, etc.

I still take respectful issue with the idea that NuGet is "handicapped." Can this be solved by a switch like "-strongnamedonly"? That way it would only bring down a complete chain, or not at all.

Also, you're saying that most open source projects are not Strong-named. This is true. In fact, we find that a tiny tiny (>1%) percentage of assemblies are strong-named at all. (Also, make sure you're clear on the difference between strong naming and code signing, that seems be interchanged, but they are very different.
Saturday, May 28, 2011 8:14:23 PM UTC
Scott – Yes, but if you’re doing branching then the update process becomes more complicated. For example, when rebuilding a project from a release branch, I may not want the latest version of my dependency. In this case I would use Version Range Specification to have more control over my dependencies, right?
Sunday, June 19, 2011 6:35:14 AM UTC
Scott, how I can tranform css-file? Or How I can transform cshtml-file?
Monday, June 20, 2011 8:09:02 PM UTC
Scott,
Great write up...the session is definitely on my backlog. I cringed a little when you compared Mercurial to SVN but I realized you were talking about it from the perspective of using tortoiseHg/tortoiseSvn.

Also, check out what Paul Stovell is cooking up with Octopus
Wednesday, June 29, 2011 1:34:42 PM UTC
I wrote a small MSBUILD script to embed the package and publish portion of the above into your csproj file. It's generic so it should cover a lot of cases.

Hope this helps others.

Richard Adleta
Wednesday, January 04, 2012 6:55:34 PM UTC
Scott-
Nice post. Somehow missed this article last year.

Our team is trying to implement TFS 2010 right now. Is it possible to use it for source control and "continuous integration" to build from check-ins, run unit the project's unit tests, and deploy on pass? I have a few TFS-noob books to read yet, but I was wondering if you recommed any particular resources.

As a web developer, I'm also trying to lean how I might tell TFS on build to combine certain items (smash all the JS or CSS files into one)... if that's even possible.

In all honesty, I'd much rather use Mercurial + Team City but I don't have a choice.
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.