Scott Hanselman

Managing Multiple Configuration File Environments with Pre-Build Events

September 21, '07 Comments [35] Posted in ASP.NET | Musings | Programming
Sponsored By

ScottGu mentioned an idea to me last week that he'd had for managing configuration files like web.config depending on what the current build config is. Bil Simser mentioned one part of this in January and Rob Chartier offered batch file help on a mailing list in June. Since ScottGu is busy Managing Generally (he IS the General Manager) so I said I'd prove the concept for him.

Here's the general idea. It's not too hard. I'll use an ASP.NET Web Site and web.config as an example, but this will work with most any kind of project, exe's or .dll's.

1. From Visual Studio, go File | New Project, and select ASP.NET Web Application.
Note: Do NOT "New Web Site" as we want a .csproj and we're going to use a Pre-Build Event, not supported by Web Sites. I've named mine FooFoo.

New Project (2)

2. Right click in the Toolbars and ensure that the "Standard" toolbar is showing. You'll know if you see a dropdown that says "Debug" next to one that says "Any CPU."

Click the dropdown and select "Configuration Manager."

image

You'll probably have Debug and Release configurations, but you can also make custom ones and base them on existing configuration. In this dialog I've made a new "Deploy" and I'll base it on the "Release" configuration.

WindowClipping (8)

Make sure to create a Solution Configuration AND a Project Configuration, as they are different. Here I've made one called Deploy for the Project also. If you get an error message, be aware of the "Create new project configurations" checkbox. You might get a warning if you are making a new configuration and the dialog tries to make another configuration with the same name; uncheck the checkbox if that happens.

deploy

Of course, you can have as many Configurations as you'd like.

3. Add some custom configuration stuff in web.config, like connectionStrings:

<connectionStrings>
    <add name="Foo"
         connectionString="Data Source=localhost;Initial Catalog=DatabaseName;
                           User Id=sa;Password=debug;"
         providerName="System.Data.SqlClient" />
</connectionStrings>

See now I've made the password in my nonsense connectionString = "debug"? Now, create three new web.config's by CTRL-dragging the web.config on top of the project. Name them web.config.debug, web.config.deploy, and web.config.release. Make the password equal to "deploy" and "release" respectively.

WindowClipping (6)

4. Ok,  now we've got different configuration and different configuration files. Let's create a batch file called "copyifnewer.bat" and here's the contents:

@echo off
echo Comparing two files: %1 with %2

if not exist %1 goto File1NotFound
if not exist %2 goto File2NotFound

fc %1 %2 
if %ERRORLEVEL%==0 GOTO NoCopy

echo Files are not the same.  Copying %1 over %2
copy %1 %2 /y & goto END

:NoCopy
echo Files are the same.  Did nothing
goto END

:File1NotFound
echo %1 not found.
goto END

:File2NotFound
copy %1 %2 /y
goto END

:END
echo Done.

Basically this batch file will copy a file over another if the files don't match. It's not strictly "copyifnewer" (like, not at all) but it does the job.

Why bother with a batch file to check for changes and not just copy over the file every time? Well, each time you copy over a web.config it restarts all the designers and running AppDomains that are watching that file. No need to copy over a file if it hasn't changed...everything will churn less.

Put this copyifnewer.bat file in the root of your project.

WindowClipping (10)

Why not use PowerShell? One word - speed. Batch files are fast. Full stop. This is a build, so it needs to be fast.

5. Create a Pre-build Event. Right-click on your Project and select Properties. Click Build Events and in the "Pre-build event command line" and enter this value:

"$(ProjectDir)copyifnewer.bat" "$(ProjectDir)web.config.$(ConfigurationName)" "$(ProjectDir)web.config"

 Notice the magic dust, the $(ConfigurationName) project variable, that contains "Debug" or "Release" or "Deploy."

WindowClipping (9)

6. Build. Now if you build, you'll see in the Build Output the batch file being run and the files being copied. Because it's a Pre-Build Event it'll be seen in both the Build Output in Visual Studio .NET. 

When you build within Visual Studio the currently selected item in the drop-down list is the current configuration.

image

The configuration value can also be passed in on the command line when building with MSBUILD. 

msbuild FooFoo.sln /p:Configuration=Deploy

 Administrator Visual Studio 2008 Beta 2 Command Prompt

And there you go. The connection string in the web.config now contains deployment-specific configuration data.

<connectionStrings configSource="separateConnStrings.config"/>

Bad things are that you've got to keep web.config's in sync if there's lots of settings, but you could totally break it apart via "include files."

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Friday, September 21, 2007 7:24:00 AM UTC
We do did something like this at Koders, except we use NAnt to copy over the correct file when running a build on the build server. For local machines, we'd do this manually. I like this approach. Nice and simple.

Personally, I think trying to have as many settings in the database as possible is a good thing. That way, the differences between your various environments is minimal. Perhaps even just a connection string difference.

It'd be nice if you could deploy the very same web.config file to all your servers and each server would read the connection string that applies to it.

For example:

<ConnectionStrings>
<add name="Dev" connectionString="blah" />
<add name="Staging" connectionString="blah" />
<add name="Prod" connectionString="blah" />
</ConnectionString>

Perhaps the server would have an environment variable you set. Then again, you could just use configSource="localserver.config". I'm really just rambling here thinking out loud (or is it "out type"). I just want to avoid all that repetition.
Friday, September 21, 2007 9:04:37 AM UTC
I've got to say, for once I really actively dislike ScottGu's tip. The problem is that it rather breaks a sensible workflow. You want the build on your initial deployment environment to be the same as the build on your final, live environment. Verifyably, inarguably, exactly the same. Rebuilding the project rather breaks this, even if you assume that you've got an source control setup locked down enough to be guaranteed your labels are consistent.

The way we approach it here is to have "delta" files which specify how the config changes depending on the deployment environment. We originally started by using the Enterprise Library solution to that, but it's somewhat unstable (it has trouble with network drives, for some reason) and rather underspecified. As a consequence, we're now using our own powershell cmdlets to do the work.

The question of whether or not to do "whole file" or "delta" is a hard one. Your dev environment is likely to support and/or use many settings that are of no relevance in production. At least it will if, like me, you're in the habit of putting developer shortcuts into your applications. The real question comes in when you've got a new setting. Do you want that setting to automatically propagate through, potentially with the wrong value, or be ignored, potentially rendering your deployment dead?

I'd rather like Phil's idea of the "one true config file" to be generalized to the whole file, but then of course, you need a more complex syntax for expressing what gets overridden and when, which makes it harder to debug when things go wrong...
Julian Birch
Friday, September 21, 2007 9:09:46 AM UTC
We do this, except that our method also allows for machine specific config files (so that, for example, our build server has a config that doesn't automatically send real e-mails when integration tests are run). We use the name of the computer as an extension instead of debug/release (though we also have those options) and match it against the standard COMPUTERNAME environment variable. Environment variables can be referenced directly in an MSBuild file.

We have a build step that copies the relevant config file to match the build configuration or the machine. If a machine-specific config is present, it takes precedence over a build-configuration-specific one. Machine-specific-configs are rare.
Friday, September 21, 2007 10:16:27 AM UTC
Just implemented this and it works a treat!
Friday, September 21, 2007 12:22:13 PM UTC
What I started doing a long while back, was to let the web.config also know about the target machines. One example is to give web.config these three variables:

LiveServerMachineName, LiveConnectionString, DevConnectionString

And have a "config" class with a shared (static) method who's call looks like:

config.CurrentConnectionString

So, the CurrentConnectionString method checks the current machine name, and if it's the same as the LiveServerMachineName, it returns the LiveConnectionString, else it returns the DevConnectionString.

The nice thing about this is, every developer gets the DevConnectionString while running on localhost, and when you deploy it you don't have to remember to do anything.

Of course, this only works in a simple "we have one live server" enviornment, but it is easily changed to work the other way (DevServerMachineName and reverse the IF) or to have multiple targets and options etc.

The point for me was to have ONE TRUE config file and not have to rely on someone remembering to switch files on deployment.
Travis Laborde
Friday, September 21, 2007 12:23:32 PM UTC
Is there a solution for handling web sites?
Friday, September 21, 2007 12:24:09 PM UTC
Researching deployment strategies I've come to some of the same conclusions as Phil and Julian:
I'd like to have one master .config file, not 4 (one for each stage). Ideally I would like to create _override or xslt translation files to customize the master files for each stage. I'm just not sure how exactly...
Creating a specific build for each stage poses a risk, a small (and possibly manageable) one, but still... From a QA perspective I want to deploy only a tested and accepted build through to the next stage.

I love the way Telligent use _override files to modify custom .config files in CommunityServer (http://scottwater.com/blog/archive/cs-3-0-update/). This obviously doesn't work for Web.config's, nevertheless it's cool to be able to deploy a new version of CS without having to compare all tweaked configs.
Jasper de Lange
Friday, September 21, 2007 12:41:24 PM UTC
I've always taken a different approach.

In the machine.config for every computer (local dev workstations, stage servers, build servers, production servers), I add the following:
<appSettings><add key="Environment" value="Staging"/></appSettings>

Then, any configuration element that is environment-specific gets the environment appended, like so:
<connectionStrings><add name="Customers.Staging" provider="..." connectionString="..."/></connectionStrings>
<appSettings><add key="NTDomain.Staging" value="test.mydomain.com"/></appSettings>

Now, I can keep *all* settings in a single file. My apps take care of requesting the appropriate setting by looking appending the "environment" to the setting key.

Been doing this at clients since 2002 and it has worked well. To me, it's simpler than the post-build concept outlined above.

Caveats:
* If you don't want all settings in one file, this is bad. For example, if you use SQL auth on production, you will be sharing that password with developers. If you use Windows auth, it's not a problem.
* You'll want a small utility class to handle this. Mine is called EnvironmentalConfigurationManager and I'll post it online one day when I eventually get my garage sale up.

Friday, September 21, 2007 12:43:30 PM UTC
This is exactly what we do, however since posting this I've become a little wiser and older in my ways and use a NAnt merge task using properties to accomplish this. I'll post a follow-up on my blog shortly about this as it's cleaner and doesn't require me to be in my IDE to get the right config file.
Friday, September 21, 2007 12:48:47 PM UTC
At Latitude, we have been using this idea for a while, but similar to Gavin, we add on to it and use build events and Nant to manage multiple environment configuration files. Rather than maintaining individual versions of the web.config for each environment, or even using include files, we have one web.config-like file, the web.format.config, that is really a copy of the web.config except with all of the appSetting values, environment-specific values, and connection strings set to NAnt property strings. All of these values and connection strings are stored in property tags in a NAnt script, and we use a NAnt build--we call it ConfigMerge--to replace all of the NAnt property strings with their property values from the appropriate file. By using one 'format' file, we can ensure that all appSettings are included and spelled correctly, and that all modifications are reflected in every environment through only one file.
Friday, September 21, 2007 12:53:31 PM UTC
As someone else mentions in the comments, you should definitely just use msbuild for this.
Just create one config file, and use specific tags for items you wish to vary. Then let your build server change these, depending on each build type.
For your developers, you will need to adjust their project file to do the same.

it's easy and very maintainable!
Friday, September 21, 2007 1:15:25 PM UTC
I've used the following for application level settings

<configSections>
<sectiongroup name="Environments">
<section name="Local" ...></section>
<section name="Dev" ...></section>
<section name="Test" ...></section>
<section name="Prod" ...></section>
</sectiongroup>
</configSections>

<Local>
<add key="Description" value="My Computer" />
<add key="ServerName" value="MyComputerName" />
</Local>

<Dev>
<add key="Description" value="Development Server" />
<add key="ServerName" value="DevServer" />
</Dev>

Then in code:
Public Enum Options
Invalid
Local
Dev
Test
Stage
Prod
End Enum

Private Shared Sub SetEnvironment()
Dim env As String() = System.Enum.GetNames(GetType(Options))
For i As Integer = 1 To env.Length - 1 ' skip invalid environment
If LCase(HttpContext.Current.Server.MachineName) = LCase(ConfigurationManager.GetSection("Environments/" & env(i))("ServerName")) Then
_Environment = CType(i, Options)
_Description = ConfigurationManager.GetSection("Environments/" & env(i))("Description")
_Domain = ConfigurationManager.GetSection("Environments/" & env(i))("Domain")
End If
Next
End Sub
Fred Price
Friday, September 21, 2007 1:22:05 PM UTC
This would marry well with the environment overrides in the EntLib 3.0+: Environmental Overrides
Friday, September 21, 2007 1:53:15 PM UTC
I definitely prefer storing deltas over multiple copies of the (mostly) same file.
Check out the XmlMassUpdate task in the MSBuild Community Tasks project. It allows you to specify deltas within an XML file to apply to a different part of the XML file (or a different file).
So you could have something like:
<system.web>
<trace enabled="true" />
</system.web>
<substitutions>
<Debug>
<trace enabled="true" />
</Debug>
<Release>
<trace enabled="false" />
</Release>
</substitutions>

You could then call it in your BeforeCompile target, passing the $(Configuration) as part of the SubstitutionsRoot.

Another option is to use a dependency injeciton framework like StructureMap which has built in support for environment-based configuration.
Friday, September 21, 2007 1:55:18 PM UTC
I had issues getting the pre-build event to consistently recognize changes with the copyifnewer batch process in an app.config (Winform app). After changing to a post build event and copying over the .exe.config in the output directly (instead of simply modifying the app.config) it worked great! Thanks for the article Scott. Below is what I configured as a post build event:

"$(ProjectDir)copyifnewer.bat" "$(ProjectDir)app.config.$(ConfigurationName)" "$(ProjectDir)$(OutDir)$(TargetName).exe.config"
Friday, September 21, 2007 2:13:17 PM UTC
Something about this methodology does not sit well with me. I don't care for maintaining 3 files that should have some slight variations.
Friday, September 21, 2007 2:36:36 PM UTC
I'm not a fan of this method. If you start using customer providers or have customer config sections having multiple files to edit is too error prone and hard to maintain.

What I do is use nant to modify the web.config with deployment specific settings using the xmlpoke task.
Here is an example I found on Google.

Web Deployment projects will also do this for you, but unfortunately its looking like we will have to wait a bit before we can use those in VS 2008 though.

Friday, September 21, 2007 2:48:53 PM UTC
One tip:
This works great until you let TFS control your projects. So, if you are using TFS, make sure to remove the web.config files from source control, but keep them as part of the project.
Troy Gould
Friday, September 21, 2007 3:23:28 PM UTC
I like this approach, but have one major issue with this specific implementation. The web.config file typically contains a lot of hacker food. Of course, the web server knows better than to serve any file with a .config extension. If you name a file "web.config.debug", aren't you giving up that security feature? I realize that you can change the build action to "None" to prevent these files from being deployed. I ask, why risk it?

I use a similar technique, but I use a naming convention of "dev.web.config". That keeps the true file extension which ensures the proper security rules will be applied on the web server and helps with Intelisense when I need to edit these files. I store them in a solution folder and let the Team Build overwrite the primary web.config depending on the location the website is being copied to.

One more thing, most config sections allow the use of the "configSource" attribute. I keep most of the guts of the configs in separate files. That way, most of the environment-specific config files are very lightweight; just a bunch of references to mostly redundant config sections.
Paul Varese
Friday, September 21, 2007 5:22:58 PM UTC
One thing maybe we can add is "Do not copy" to the .config.<config> files, in order to reduce the output footprint
Rajiv
Friday, September 21, 2007 6:29:43 PM UTC
You can also use the ReplaceConfigSections task which comes with Web Deployment Projects. Then you do not have to recompile a project that is already built. You just run a deployment script that adjusts the config and pushes the files. I have a tutorial on it here...

http://brennan.offwhite.net/blog/2007/04/26/post-build-deployments-with-msbuild/
Friday, September 21, 2007 8:15:31 PM UTC
This in an interesting solution, thou I usually prefer to have #endifs in a single file (say a web.config in this sample) and run the C pre-processor as a pre-build step to generate all the required variations of the file (or a single one based on the configuration, it that's the case).

The downside to this approach, the pre-processor directives usually end-up confusing the XML editor in VS; on the plus side, you have only a single file to keep up to date.
Claudio A. Heckler
Friday, September 21, 2007 8:28:14 PM UTC
One more followup on this and over the weekend I'll post an updated blog entry on my site. The NAnt merge is the way to go (can't figure out how to do this with MSBuild so I'll leave it to those to try). I don't like having the batch file as it's another resource and doesn't buy me anything. I can let NAnt do this for me. One more enhancement I've done is based on the machine you're deploying to or running from it'll pick up the correct settings. For example if you're running on the actual dev build box (not sure why but who knows) then it'll just grab the right config file. Otherwise you can specify what environment to deploy to as there are separate entries for each environment (but not necessarily server). Finally, I make it a point to have ONE app.config/web.config and use the appSettings and connectionStrings setting (which allows you to reference an external file) along with a new file generated by NAnt. This way, I don't have to mess around with the original config file as it's never actually used. Hard to explain in a comment here so like I said, I'll blog about this in depth with some screenshots and examples. In any case, managing deployments and settings for multiple environments is a big chore, no matter how great your solution is.
Saturday, September 22, 2007 10:26:48 AM UTC
Why not use PowerShell? One word - speed. Batch files are fast. Full stop.


That's one for the ages.
J.S.
Saturday, September 22, 2007 1:57:41 PM UTC
We are doing something similar using a substitution technique. We have a post build step that rips through the config file and replaces the substitution tokens with the environment specific values. It isn't a perfect solution, but it has worked great for our needs. If you are interested, you can read about it here.

Matt.
Monday, September 24, 2007 2:10:16 AM UTC
As Jeff mentioned you should just marry the work that has already been done in EntLib 3.0 with environmental overrides and use that as apart of the build process. The problem with this approach is keeping the files in sync. I may have something that is just as simple as a connection string that changes from test to beta to prod and then other various settings that are different. Really the config file for each environment is made up of various pieces and EntLib 3.0 does a good job of "building" one file for each environment if it needs to be over written.
Monday, September 24, 2007 2:01:18 PM UTC
Hi Scott - why not edit the .csproj msbuild file to do the copying?

On a related but slightly different problem, I used msbuild to handle a different web.config file per developer. Documented here.

msbuild to the rescue.

Cheers
Matt
Wednesday, October 03, 2007 2:18:35 PM UTC
I like ScottGu's approach to solving using existing solutions namely Visual Studio build configuration and MSBuild. I would like to add flexibility to this approach by using Web Deployment Project and/or configSource attribute for a config section in web.config.

1. Have Build configurations for different environments. (DEV, QA, STAGE, PROD)
2. Create a custom section in web.config to hold environment specific settings and add/update code to access settings from this section.
3. Use an external config file to be configSource for this section by creating a env.config file. Additionally also create qa.env.config, stage.env.config and prod.env.config
4. Replacing the env.config with the correct external config source can be accomplished during build time or JIT compilation time. either way the effect is the same.
4a. If you just specify the configSource for custom section in web.config to point to an external file, these settings will picked up during JIT compile time.
4b. If you use Web Deployment Projects, "Enable Web.config file section replacement", the merge of external config will happen during build and packaging time, resulting in a web.config physical file with correct values.
5. Since most of use CC.NET and NAnt, have a step before the build to copy the correct env.config into the application directory, before proceeding with the build. Very similar to Project Pre-Build event.

Above will ensure that whatever build you are making will accommodate for the correct environment specific config sections to be available. Additionally we can have the different env specific external files to hosted in a Source control system and delegate the control over to suitable personal to manage their environments.

I'm in the process implementing this solution and would like to hear your opinion.

Thanks,
Prabhu
Prabhakar Harita
Thursday, October 04, 2007 2:22:35 AM UTC
Dude, no VB love for me. How do you do this with a VB.Net project? I'm assuming the gory stuff is hidden from us VB'ers, unlike in the C# stuff.
Thursday, October 04, 2007 11:55:44 PM UTC
If you use VB.Net apps, look at bottom right of compile tab for the 'Build Events...' button. I missed it several times for some reason. Hope that helps someone.
Monday, October 15, 2007 8:54:46 PM UTC
I was tesing this stuff in my local machine but I had a problem while creating a new project. If I select a new Project, there is not any option to select web application. This option only be appeared If I select a new web site. There might be somewhere I am doing wrong. FYI I am using Visual Studio 2005 Team Suite. Could you pls guide me ?
Monday, October 15, 2007 8:57:05 PM UTC
I have only one question about build process of Team Suite. Could you please post any blog about new build features of Team Suite. How can we configure the build for our different environment like Dev, Staging, Producation so no need to change configuration setting everytime if we push new build on staging or producation. That would be very helpful. Thxs
Thursday, November 01, 2007 9:54:47 PM UTC
great article! I noticed when i try to do a build via command line for a newly added configuration I receive this error.

"The specified solution configuration "P3" is invalid. Please
specify a valid solution configuration using the Configuration and Platform
properties (e.g. MSBuild.exe Solution.sln /p:Configuration=Debug
/p:Platform="Any CPU") or leave those properties blank to use the default
solution configuration."

I can build it in VS without any problems..just command line that is causing me grief. Any ideas?
todd
Thursday, November 01, 2007 10:03:32 PM UTC
disregard .. typo in solution caused the problem.
todd
Wednesday, November 14, 2007 6:03:10 PM UTC
I've worked on a project where we have CI with deployments working on TFS. Basically we've used a combination of the Web Deployment projects for websites and a WCF project. Which is interesting because the WCF project deployment does not like precompiled sites. So if you do not mark the deploymnet as "Allow Updates", the .svc files seem to get compibled with the pointers for the .compile files. We got around this by adding a command to the Build Type to remove all .compile files and marked it "Allow Updates".

We control all of the app.config, etc files by doing something similar to Scott's examples, except we left out the "latest and greatest" check.

Good post Scott.
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.