Scott Hanselman

Fixed: "Windows Process Activation Service (WAS) is stopping because it encountered an error."

November 12, '08 Comments [21] Posted in IIS | Tools
Sponsored By

I'm not yet clear what I did, but I'm blogging it so it can be found if someone else has this issue.

For whatever reason, last week both of my Vista 64-bit machines suddenly stopped being able to start IIS (Internet Information Server). The service just wouldn't start. I started getting this error instead "Cannot start service W3SVC on Computer '.'" which wasn't too helpful.

image

A visit to the System Event Log via the Event Viewer in Computer Management told me these four errors:

"The World Wide Web Publishing Service service depends on the Windows Process Activation Service service which failed to start because of the following error: The system cannot find the file specified."

and

"The Windows Process Activation Service service terminated with the following error:
The system cannot find the file specified."

and

"Windows Process Activation Service (WAS) is stopping because it encountered an error. The data field contains the error number."

and

"The directory specified for the temporary application pool config files is either missing or is not accessible by the Windows Process Activation Service. Please specify an existing directory and/or ensure that it has proper access flags. The data field contains the error number."

imageUnfortunately there's little information to go on in any of these error messages. However, it's clear (as mud) from the last error that there's a directory missing or not accessible. I'll add "anymore" to that because it worked before. That means that something changed.

If IIS won't start because Windows Process Activation Service won't start, then I need to get WAS started up first. However, I don't know what directory it doesn't have access to.

I can see from the Services application that WAS isn't its own executable, but rather lives inside of an instance of svchost.exe, where a lot of services live.

So I'll fire up Process Monitor and set the filters (filters are VERY important if you want to avoid being overwhelmed quickly in procmon) to show only svchost.exe processes.

Even still, there's a lot of svchost.exe processes out there and they will quickly fill the monitor up. I'll need to setup some strategic (read: guessed) highlighting as well.

The hotkey to stop capturing in procmon.exe is Ctrl-E. Basically I'll clear the screen, hit Ctrl-E to capture, try to start WAS (pronounced WAAZ), watch it fail, the stop capture with Ctrl-E.

Based on the vague message about application pools temporary files and a directory I'll make a guess and configure highlighting to find paths that contain "temp," "log," "config" or "app" in Process Monitor as seen in the screenshot below.

image

After I run the capture, I scroll around looking for suspicious stuff. One of the nice things about Process Monitor is that you can EXCLUDE things in a given capture after that fact. For example, I saw a pile of Audio and Media related stuff that was visually confusing and cluttering the point, so I excluded it.

The result is here:

image

It looks like there SHOULD be a folder call c:\inetpub\temp\apppools and on my Vista 64 machines, in the last two weeks to a month, it just disappeared. No idea why. I just noticed recently when I tried to move from a local web development service to IIS itself.

I created the folder, started WAS, then IIS and I was back up and running.

I'll pass the feedback on to the WAS team about the obscure error messages, but I thought I'd share this little ten minute debugging session to point out a few things that I think are important and possibly helpful, Dear Reader:

  • Know What Your Processes Are Doing (or at least, know how to find out)
    • Knowing how to look INSIDE the Windows "Black Box" using tools like ProcMon makes you realize that no OS is a Black Box at all. It's very empowering to know that you CAN see inside.
    • TASK: Learn Process Monitor and Process Explorer.
  • Enable Your Intuition
    • Debugging is 95% tools and 5% intuition. Know what tools can get you that next bit of information you need to take the next step in your analysis.
    • If you feel you've hit a wall in your analysis, knock that wall down. Your process is doing IO to a file/registry/device/network/etc. Watch it. Look for failures.

My next mission is to find out WHY and HOW this directory disappeared on both my machines. What did I install or run recently? Enjoy!c

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. I am a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Wednesday, November 12, 2008 9:21:09 AM UTC
I've actually had a similar problem after moving the index location and deleting the old one before rebooting the service. Same mostly useless error message "could not find file".

Ended up reinstalling, as copying the configuration from another machine left... somewhat spotty results. WSearch has a slightly more complicated configuration structure AFAICT.

Windows components that just fail if their configuration is missing should really have better error messages, or even better just log an event and revert to defaults. At least a (in-box) tool to reset the config would be nice.
fowl
Wednesday, November 12, 2008 9:24:55 AM UTC
This isn't a whole lot of help now, I know, but there is actually a KB article on this very error message. However, it doesn't tell you why it might have happened.

(It is almost always worth checking the KB for the error code on any system/app log errors.)

Thanks,

M
Wednesday, November 12, 2008 11:15:30 AM UTC
I'd hazard a guess that the disk cleanup procedure you blogged about recently may have done this.

I hit a similar error downloading from MSDN where the File Transfer Manager couldn't be found - I think the cleanup blew it away from a temporary download area. It freed up 200GB of disk space though -- so I'm not complaining!

/Damian
Wednesday, November 12, 2008 12:02:26 PM UTC
Or, you could google that last error message and find http://technet.microsoft.com/en-us/library/cc734935.aspx, which has, as the first option for solving this problem:

To resolve this issue, make sure that the appPools directory meets the following requirements:
- The directory must exist.
- The directory cannot be on a UNC path.
- The directory must be available to WAS and should have the following permissions:
* SYSTEM: Full Access
* Administrators: Full Access
* IIS_IUSRS: Read

If google doesn't find it, then you'd have to use procmon, etc...
configurator
Wednesday, November 12, 2008 12:45:32 PM UTC
Can't you look through the 'Previous Versions' tab of the parent folder's properties in Vista to find out when it disappeared? (giving you the benefit of the doubt that IIS has in fact been working on your machine(s) until fairly recently, and assuming that the WPAS requirement wasn't added in a recent patch)
tb
Wednesday, November 12, 2008 1:00:13 PM UTC
Does anyone el
Wednesday, November 12, 2008 1:02:21 PM UTC
Oops. Sorry about that accidentally premature submission.

Does anyone else find it troubling that a folder that MUST exist, even if it's just empty, is located inside a folder called "temp"? One of my hobbies is deleting folders called "temp". Oh well, at least I now know about this one.

Wednesday, November 12, 2008 2:33:24 PM UTC
Scott,
Thanks for taking us with you as you seek out the answer. I am not experienced in using procmon.exe as a tool to decipher cryptic event messages. I am excited to see how this turns out.

Ian B
Ian B
Wednesday, November 12, 2008 2:56:44 PM UTC
It was recently reported on Slashdot that AVG Virus Scanner, after a recent software update, started marking user32.dll as a virus. Do you use AVG, and what are your auto-clean settings? If not, perhaps another similar program did the same thing to this directory?
Wednesday, November 12, 2008 3:28:00 PM UTC
+1 on the poor choice for requiring a subdirectory under "temp" to exist. Anything inside "temp" should always be considered transient and removable.

Two more points which can be taken away from this object lesson:

3. Know your tools.
Google is your friend :) The probability is high that someone, somewhere has encountered your problem before, and there is information about it somewhere on the internet. You just have to find it.

4. Information is power.
When logging errors in your own application, be very specific about the problem and give as much context as you can. This also goes for exception messages.
Wednesday, November 12, 2008 4:26:08 PM UTC
Scott,

Nice troubleshooting. A little bit more on that temp folder in case of interest. In IIS6, security was set using ACLs within the metabase. This works similar to NTFS permissions on disk. There were a number of problems with that. One is general awareness and weaker tools. For example, it would be great to have a ProcessMonitor for the IIS metabase. The bigger issue is that SIDs and encrypted ACL information doesn't copy well between servers, so sharing configurations between servers was difficult in IIS6.

So, in IIS7, the IIS team changed this considerably. There is no such thing as ACLs within the metabase (now applicationHost.config) anymore. Instead, WAS, which runs as Local System, will copy just the relevant information for a particular app pool to the temp folder and ACLs it down with a new type of ACL that only allows access for a particular worker process. This way the app pool worker process (which runs under Network Service or a custom user that you define), will access the temp .config folder that WAS provided. This ensures that each app pool worker process can only manage its own settings and it cannot see settings from other app pools (or example usernames/passwords).

Thus the reason for the new temp folder. That also explains why it's an absolute dependency and not an optional folder as could be assumed from the name "temp".

Wednesday, November 12, 2008 5:25:16 PM UTC
Maybe the explanation is in this post :)
Wednesday, November 12, 2008 5:52:21 PM UTC
Wow, I Googled all over (and I KNOW I'm good at Googling) and I couldn't find a darn thing. Funny that it IS out there, I just picked the wrong words. Still, it's a lame error message.

ScottF - Great explanation! Thanks!

MikeP - I agree. I suspect I "cleaned" more aggressively than I should have, but seriously, you should be able to delete ANYTHING called Temp.

DavidN - Totally. If they'd just told me the path, problem solved.

Wednesday, November 12, 2008 7:33:58 PM UTC
Scott F.,

The reason for needing such a folder makes sense. My objection is that it is named "temp". Like Mike and Scott H, I regularly delete anything named "temp" that I find on my hard drive, because I know that many applications are terrible about cleaning up after themselves. For IIS to require a "temp" folder to always exist is a poor design decision; if it is a required folder, it should be named in a way that indicates to the user (who is supposed to be in control of the machine after all) that it is required).
Thursday, November 13, 2008 2:48:00 AM UTC
Yet another example of why the Systernals guys are gods!
I have used the same trick for other problems with good success as well.
John
Thursday, November 13, 2008 3:14:04 AM UTC
Why cant the application just re-create it's folder if it lives in temp?
Thursday, November 13, 2008 6:35:32 AM UTC
I really like your explanation on how you 'debugged' this particular problem using tools available to diagnose the OS and the processes running on your machine. I used the sysinternal tools, amongst others, to diagnose and remove a significant virus on my system about a year ago. Understanding your system and the tools available to see what's happening under the hood is immensely important for a huge range of problems.

Jeff Atwood described his battle with malware and how he successfully recovered using nothing more than diagnostic tools and a bit of intuition at:
http://www.codinghorror.com/blog/archives/000888.html

Important lesson to the 'advanced' users out there - know your underlying foundations and how to reach inside.
Thursday, November 13, 2008 8:10:00 AM UTC
Second to amasses.

Some of my apps require a temp folder to exist too.
What it does is store temporary data there (hence the name), I don't think that's a bad decision.
What IS a bad decision is throwing errors all around when you can't find the folder. If a temp folder isn't there.. create it. If you don't have the permission to create it THEN you can start bitching to the event log.
Thursday, November 13, 2008 9:53:35 PM UTC
If you get really bored turn on a ProcMon filter to exclude SUCCESS (also pronounced Suckless) and see what a real mess a running windows instance really is!
Saturday, November 15, 2008 2:43:21 AM UTC
As a DSI guy, I find this post really helpful. Not only the result is nice and good, but the whole troubleshooting process is impressive.
Sunday, November 30, 2008 7:36:37 AM UTC
I think I found the culprit which sometimes causes the problem. Did you install Sharepoint recently?
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.