Fixed: "Windows Process Activation Service (WAS) is stopping because it encountered an error."
I'm not yet clear what I did, but I'm blogging it so it can be found if someone else has this issue.
For whatever reason, last week both of my Vista 64-bit machines suddenly stopped being able to start IIS (Internet Information Server). The service just wouldn't start. I started getting this error instead "Cannot start service W3SVC on Computer '.'" which wasn't too helpful.
A visit to the System Event Log via the Event Viewer in Computer Management told me these four errors:
"The World Wide Web Publishing Service service depends on the Windows Process Activation Service service which failed to start because of the following error: The system cannot find the file specified."
and
"The Windows Process Activation Service service terminated with the following error:
The system cannot find the file specified."
and
"Windows Process Activation Service (WAS) is stopping because it encountered an error. The data field contains the error number."
and
"The directory specified for the temporary application pool config files is either missing or is not accessible by the Windows Process Activation Service. Please specify an existing directory and/or ensure that it has proper access flags. The data field contains the error number."
Unfortunately there's little information to go on in any of these error messages. However, it's clear (as mud) from the last error that there's a directory missing or not accessible. I'll add "anymore" to that because it worked before. That means that something changed.
If IIS won't start because Windows Process Activation Service won't start, then I need to get WAS started up first. However, I don't know what directory it doesn't have access to.
I can see from the Services application that WAS isn't its own executable, but rather lives inside of an instance of svchost.exe, where a lot of services live.
So I'll fire up Process Monitor and set the filters (filters are VERY important if you want to avoid being overwhelmed quickly in procmon) to show only svchost.exe processes.
Even still, there's a lot of svchost.exe processes out there and they will quickly fill the monitor up. I'll need to setup some strategic (read: guessed) highlighting as well.
The hotkey to stop capturing in procmon.exe is Ctrl-E. Basically I'll clear the screen, hit Ctrl-E to capture, try to start WAS (pronounced WAAZ), watch it fail, the stop capture with Ctrl-E.
Based on the vague message about application pools temporary files and a directory I'll make a guess and configure highlighting to find paths that contain "temp," "log," "config" or "app" in Process Monitor as seen in the screenshot below.
After I run the capture, I scroll around looking for suspicious stuff. One of the nice things about Process Monitor is that you can EXCLUDE things in a given capture after that fact. For example, I saw a pile of Audio and Media related stuff that was visually confusing and cluttering the point, so I excluded it.
The result is here:
It looks like there SHOULD be a folder call c:\inetpub\temp\apppools and on my Vista 64 machines, in the last two weeks to a month, it just disappeared. No idea why. I just noticed recently when I tried to move from a local web development service to IIS itself.
I created the folder, started WAS, then IIS and I was back up and running.
I'll pass the feedback on to the WAS team about the obscure error messages, but I thought I'd share this little ten minute debugging session to point out a few things that I think are important and possibly helpful, Dear Reader:
- Know What Your Processes Are Doing (or at least, know how to find out)
- Knowing how to look INSIDE the Windows "Black Box" using tools like ProcMon makes you realize that no OS is a Black Box at all. It's very empowering to know that you CAN see inside.
- TASK: Learn Process Monitor and Process Explorer.
- Enable Your Intuition
- Debugging is 95% tools and 5% intuition. Know what tools can get you that next bit of information you need to take the next step in your analysis.
- If you feel you've hit a wall in your analysis, knock that wall down. Your process is doing IO to a file/registry/device/network/etc. Watch it. Look for failures.
My next mission is to find out WHY and HOW this directory disappeared on both my machines. What did I install or run recently? Enjoy!c
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
(It is almost always worth checking the KB for the error code on any system/app log errors.)
Thanks,
M
I hit a similar error downloading from MSDN where the File Transfer Manager couldn't be found - I think the cleanup blew it away from a temporary download area. It freed up 200GB of disk space though -- so I'm not complaining!
/Damian
To resolve this issue, make sure that the appPools directory meets the following requirements:
- The directory must exist.
- The directory cannot be on a UNC path.
- The directory must be available to WAS and should have the following permissions:
* SYSTEM: Full Access
* Administrators: Full Access
* IIS_IUSRS: Read
If google doesn't find it, then you'd have to use procmon, etc...
Does anyone else find it troubling that a folder that MUST exist, even if it's just empty, is located inside a folder called "temp"? One of my hobbies is deleting folders called "temp". Oh well, at least I now know about this one.
Thanks for taking us with you as you seek out the answer. I am not experienced in using procmon.exe as a tool to decipher cryptic event messages. I am excited to see how this turns out.
Ian B
Two more points which can be taken away from this object lesson:
3. Know your tools.
Google is your friend :) The probability is high that someone, somewhere has encountered your problem before, and there is information about it somewhere on the internet. You just have to find it.
4. Information is power.
When logging errors in your own application, be very specific about the problem and give as much context as you can. This also goes for exception messages.
Nice troubleshooting. A little bit more on that temp folder in case of interest. In IIS6, security was set using ACLs within the metabase. This works similar to NTFS permissions on disk. There were a number of problems with that. One is general awareness and weaker tools. For example, it would be great to have a ProcessMonitor for the IIS metabase. The bigger issue is that SIDs and encrypted ACL information doesn't copy well between servers, so sharing configurations between servers was difficult in IIS6.
So, in IIS7, the IIS team changed this considerably. There is no such thing as ACLs within the metabase (now applicationHost.config) anymore. Instead, WAS, which runs as Local System, will copy just the relevant information for a particular app pool to the temp folder and ACLs it down with a new type of ACL that only allows access for a particular worker process. This way the app pool worker process (which runs under Network Service or a custom user that you define), will access the temp .config folder that WAS provided. This ensures that each app pool worker process can only manage its own settings and it cannot see settings from other app pools (or example usernames/passwords).
Thus the reason for the new temp folder. That also explains why it's an absolute dependency and not an optional folder as could be assumed from the name "temp".
ScottF - Great explanation! Thanks!
MikeP - I agree. I suspect I "cleaned" more aggressively than I should have, but seriously, you should be able to delete ANYTHING called Temp.
DavidN - Totally. If they'd just told me the path, problem solved.
The reason for needing such a folder makes sense. My objection is that it is named "temp". Like Mike and Scott H, I regularly delete anything named "temp" that I find on my hard drive, because I know that many applications are terrible about cleaning up after themselves. For IIS to require a "temp" folder to always exist is a poor design decision; if it is a required folder, it should be named in a way that indicates to the user (who is supposed to be in control of the machine after all) that it is required).
I have used the same trick for other problems with good success as well.
Jeff Atwood described his battle with malware and how he successfully recovered using nothing more than diagnostic tools and a bit of intuition at:
http://www.codinghorror.com/blog/archives/000888.html
Important lesson to the 'advanced' users out there - know your underlying foundations and how to reach inside.
Some of my apps require a temp folder to exist too.
What it does is store temporary data there (hence the name), I don't think that's a bad decision.
What IS a bad decision is throwing errors all around when you can't find the folder. If a temp folder isn't there.. create it. If you don't have the permission to create it THEN you can start bitching to the event log.
Comments are closed.
Ended up reinstalling, as copying the configuration from another machine left... somewhat spotty results. WSearch has a slightly more complicated configuration structure AFAICT.
Windows components that just fail if their configuration is missing should really have better error messages, or even better just log an event and revert to defaults. At least a (in-box) tool to reset the config would be nice.