RTFLF - Read the Expletive Log File
A buddy of mine and I had a nice slap in the face yesterday. I was helping him deploy an ADO.NET Data Service to a large company's staging server and we were seeing REALLY odd behavior.
We'd request something like /myservice.svc and get a 404. But we could request /myservice.svc/Stuff or /myservice.svc/?metadata.
We settled in to debug this. We thought we were "getting down to basics." You know, you've done this. The conversation goes something like:
"Ok, people, what's the definition of insanity? Trying the same thing and expecting a different result."
"Right...let's challenge all our assumptions. Let's start from scratch. Can get Hello World working?"
"What's the ACLs on that file? Is the .svc extension registered? Are we sure we have the right version of .NET?"
We were both tired and we wasted a couple of hours basically dicking around, hitting Refresh and hoping for another solution. I wanted to plugin procexp and filemon and get down to some serious CSI: IIS-type debugging, but here's the rub: We didn't have access to the machine. Only "large company's" guys were allow to touch anything. We could make suggestions, watch a SharedView session, but the human latency of the whole process was slowing us by a factor of at least five.
But I can't blame it all on the process. In retrospect, it was my fault. I'm a good debugger. I know this and I'm happy to say it. However, I can recognize a ninja when I see one. Well, if you can see the ninja, maybe they aren't a ninja, but still. I reached out to a real debugging ninja. What did he do that I was missing?
I ignored a basic tenet of debugging. It wasn't that I didn't RTFM. I didn't RTFLF.
My debugger-ninja-friend started out by simply asking us to Start|Run and type "LogFiles."
At this point I realized that this process was going to make me look and feel like an idiot. My internal lights went on and I realized my buddy and I hadn't bothered to check any log files. We'd been treating IIS like it was a black box. It's not. It logs the hell out of everything that goes in and out if you want it to.
We were trying to debug a 404 on this .svc. We opened the log in Notepad, went to the bottom, searched up for ".svc" and there it was:
This Page was blocked by Microsofts URL Scan 3.0 Reason=Dot-in-path-detected
You could have knocked me over with a feather. I've said myself, UrlScan is step 0. If you're debugging a weird 404, UrlScan is the first and most obvious place to look and it was all there, in the log files. You remember, the log files I never looked at. ;)
Did my ninja friend know or care? No, because he RTFLF. A painful reminder to me as I wasted a bit of a ninja's time. Everyone knows, don't piss off a ninja. He was cool about it though.
Today's Lesson: Whatever it was, it was probably logged. Try there first.
* Pic from Dr. McNinja.