Microsoft Developer Support or ("CSS" - Customer Support Services) is where you're sent within Microsoft when you've got problems. They see the most interesting bugs, thousands of issues and edge cases and collect piles of data. They report this data back to the ASP.NET team (and other teams) for product planning. Dwaine Gilmer, Principal Escalation Engineer, and I thought it would be interesting to get some of that good internal information out to you, Dear Reader. With all those cases and all the projects, there's basically two top things that cause trouble in production ASP.NET web sites. Long story short, Debug Mode and Anti-Virus software.

Thanks to Dwaine Gilmer, Doug Stewart and Finbar Ryan for their help on this post! It's all them!

#1 Issue - Configuration

Seems the #1 issue in support for problems with ASP.NET 2.x and 3.x is configuration.

People continue to deploy debug versions of their sites to production. I talked about how to automatically transform your web.config and change it to a release version in my Mix talk on Web Deployment Made Awesome. If you want to save yourself a headache, release with debug=false.

Additionally, if you leave debug=true on individual pages, note that this will override the application level setting.

Here's why debug="true" is bad. Seriously, we're not kidding.

• Overrides request execution timeout making it effectively infinite
• Disables both page and JIT compiler optimizations
• In 1.1, leads to excessive memory usage by the CLR for debug information tracking
• In 1.1, turns off batch compilation of dynamic pages, leading to 1 assembly per page.
• For VB.NET code, leads to excessive usage of WeakReferences (used for edit and continue support).

An important note: Contrary to what is sometimes believed, setting retail="true" in a <deployment/> element is not a direct antidote to having debug="true"!

#2 Issue - Problems with an External (non-ASP.NET) Root Cause

Sometimes when you're having trouble with an ASP.NET site, the problem turns out to not be ASP.NET itself. Here's the top three issues and their causes. This category are for cases that were concluded because of external reasons and are outside of the control of support to directly affect. The sub categories are 3rd party software, Anti-virus software, Hardware, Virus attacks, DOS attacks, etc.

If you've ever run a production website you know there's always that argument about whether to run anti-virus software in production. It's not like anyone's emailing viruses and saving them to production web servers, but you want to be careful. Sometimes IT or security insists on it. However, this means you'll have software that is not your website software trying to access files at the same time your site is trying to access them.

Here's the essence as a bulleted list

• Concurrency while under pressure: This causes problems in big software. Make sure your anti-virus software is configure appropriately and that you're aware of which processes are accessing which files, as well as how, why and when
• Profile your applications: .NET and the Web are not black boxes. You can see what's happening if you look. Know what bytes are going out the wire. Know who is accessing the disk. Measure twice, cut once, they say? I say measure a dozen times. You'd be surprised how often folks put an app in production and they've never once profiled it.
• Anti-Virus Software: It can't be emphasized enough that site owners should ensure they are running the latest AV engine and definitions from their chosen anti-malware vendor. They've see folks hitting hangs due to flakey AV drivers that are over two years out of date.  Another point about AV software is that it is not just about old-school AV scanning of file access. Many products now do low level monitoring of port activity, script activity within processes and memory allocation activity and do not always do these things 100% correctly. Stay up to date!
• Know where you're calling out to: Also, connection to remote endpoints: calling web services, accessing file systems etc. All of this can slow you down if you're not paying attention. Is your DNS correct? Did you add your external hosts to a hosts file to remove DNS latency? 
• processModel autoconfig=true: This is in machine.config and folks always mess with it. Don't assume that you know better than the defaults. Everyone wants to change the defaults, add threads, remove threads, change the way the pool works because they think their textboxes-over-data application is special. Chances are it's not, and you'd be surprised how often people will spend days on the phone with support and discover that the defaults were fine and they had changed them long ago and forgotten. Know what you've changed away from the defaults, and know why. Don't program by coincidence.

...and here's the table of details:

Spread the word! What kinds of common issues do YOU run into when running production sites, Dear Reader?