Scott Hanselman's Computer Zen - Hammering in Screws

« How to run a Disk Scan/chkdsk/fsck and T... | Main | Matisyahu is coming to Portland »

Hammering in Screws

Posted 2005-06-21 05:21 PM in XML | Bugs | Tools.

I sent this email out to some Engineers at the company yesterday. It was intended to hurt no one's feelings and no one was singled out. I don't know the details of who or this or that, and that's not the point.

Hello folks,

Just a friendly reminder from the Hanselman. Today, roughly 10 people spent roughly half a day debugging a single problem and assigning culpability on various versions of various products. Various email threads were started and much ReplyToR’ing occurred.

The problem was badly formed XML. Specifically a space in a closing XML Tag.

These folks worked very hard and it’s commendable that they found the problem. That said, I would like to take this opportunity to gently remind everyone of these few principles so that we all, myself included, might learn from their suffering:

TextPad isn’t an XML editor.

It’s not even a good syntax highlighting editor. Visual Studio and XMLSpy are and would have caught an error like this on load. (Editor: as would have dropping it into IE)

“Entia non sunt multiplicanda praeter necessitatem” which is Latin for “Keep it simple, stupid.”

Return to the first principles when debugging anything and assert your assumptions.

The plural of “anecdote" is not “truth”

If you heard that such and such was broken or doesn’t work, ask around to those involved directly and get the truth (as it is known) directly from them. We’re not such a large company that we need to ReplyToAll as much as we do.

That said, thanks to the group who did put in the hard work and find this problem, allowing the [product] to continue forward.

Thanks for your patience and attention!

Reaction in the hallway was generally positive, while no doubt others cursed.

Of course, no one is beyond reproach and I've got MORE than my share of Doh! moments, but this email was meant to remind people to use the tools they've got in the toolbox, to question the basics when debugging and to deal in concrete facts rather than conjecture.

My questions to you, Dear Reader, are these:

Do you ever send emails to the team to remind folks of First Principles?
Do you have a growing ReplyToAll culture that should be stamped out and replaced with walking to someone's office?
How long should an email thread go before it is stopped and replaced with a meeting?
Is it the pressure to ship that causes all of us to miss misplaced semicolons and the like, or just human nature?
Is "flaming potato" a part of your software engineering culture? Do you struggle with it?
Too harsh? Not harsh enough?

Tracked by:
"Rules to Better" (Kent J. Chen's Weblog) [Trackback]
"Interesting Finds" (Jason Haley) [Trackback]
"My Year In Blogging - 2005" (ComputerZen.com - Scott Hanselman) [Trackback]

Tuesday, June 21, 2005 4:28:37 PM (Pacific Standard Time, UTC-08:00)

Scotty

As you know me well.... I concur completely

Similarly my #1 rule on http://www.ssw.com.au/SSW/Standards/Rules/RulesToBetterEmail.aspx
is not to use email for communication

Cheers Adam

Adam Cogan

Tuesday, June 21, 2005 4:30:50 PM (Pacific Standard Time, UTC-08:00)

Answers to your questions:
Do you ever send emails to the team to remind folks of First Principles?
--Seems like I have to send reminders out about quarterly.
Do you have a growing ReplyToAll culture that should be stamped out and replaced with walking to someone's office?
--I used to be guilty of this one and finally realized what a waste of time it is. Reply all is hoping someone else will pick up the ball and run with it.
How long should an email thread go before it is stopped and replaced with a meeting?
--I don't know if there is a hard and fast rule but after about 4-5 replies I usually escalate it to a meeting. With certain folks whose calendars are always full it may go a little longer.
Is it the pressure to ship that causes all of us to miss misplaced semicolons and the like, or just human nature?
--Got me. I see folks outside the dev process affected by this who theoretically don't have a pressure to ship.
Too harsh? Not harsh enough?
--About right.

Chris Kinsman

Tuesday, June 21, 2005 4:31:06 PM (Pacific Standard Time, UTC-08:00)

Considering the amount of resources taken to find the issue, I would say the email you sent was fair. We all need reminders from time to time...

J. Michael Palermo IV

Tuesday, June 21, 2005 4:34:06 PM (Pacific Standard Time, UTC-08:00)

Doesn't seem to harsh, like you said, we've all had our 'doh' moments. My favorite was point to a dev database environment, from the proofing server environment, and not figuring out how or why my data was not on the proofing server, but showing up in my application ( the dev server database!)

Yeah, DOH!

Thankfully this was not 1/2 a day or anything, but.... it took a little time to realize.

The biggest problem was I broke my own rule, I had changed a connection setting in a sheer moment of laziness to check something, and then was invariably interrupted, forgetting to change the connection back.... you see where I'm going with this.

I've been burned by bad XML too, I've been kind of liking the XML Spy that you suggested some time back, that would have caught this issue, no?

JB

John Batdorf

Tuesday, June 21, 2005 5:30:49 PM (Pacific Standard Time, UTC-08:00)

Where I am working now the team and the dev area is small enough that we just shout out our questions and generally someone will either a) has the answer, b) some related helpful comment, or c) very insightful hint. So emails and meetings are not used that much as far as debugging is concerned.

But, if you remember me, I used to work in the same office area as you do and I can definitely see the email threads in my mind as I was reading your post :)

Harsh? I think it's about right for what it is and who it is for. Personally I think it is not harsh enough :-)

Alex Hung

Tuesday, June 21, 2005 6:28:49 PM (Pacific Standard Time, UTC-08:00)

Do you have a growing ReplyToAll culture that should be stamped out and replaced with walking to someone's office?
How long should an email thread go before it is stopped and replaced with a meeting?
============

Another tool that developers seem to have forgotten is the telephone. If you find your self in a back-and-forth email exchange pick up the phone and call the other dev. If you are typing emails or on IM you are not writing code (at least not anything you'll want to ship) and neither is your team mate. Voice communication (phone or office door) is often faster than the keyboard alternative.

I think the "Reply to All" is often used to try and drum up support for your side of the issue. "What a stupid idea Joe has. Think I'll CC the rest of the group, they'll support me."

Walt Ritscher

Tuesday, June 21, 2005 6:45:31 PM (Pacific Standard Time, UTC-08:00)

Scott,

It is always tough to be in a lead position. I don't know your leadership style and office culture, so it is hard to judge if it was harsh message or not. My current group is sensitive and it would have come across as harsh, as the problem would have already made the offenders feel bad enough and coming out with the reminders email out have been like finger point at the guilty party(s).

As for your questions, I rarely do the principles stuff via email. Meetings are a good way to do them as you can get a good read of your team as you go over them. It helps me to feel is if everyone is "onboard" and helps them to see how I feel about them firsthand. You never really know how an email like that is received (everyone is different). If you look them in the eye, you can tell if one on one follow up is needed.

This comment will get too long if I go through all the questions, so I'll just say questions like these are great to think through. Thanks for giving me some things to chew on. :)

Tuesday, June 21, 2005 7:08:46 PM (Pacific Standard Time, UTC-08:00)

We all need these reminders from time to time. When the pressure is on, it's easy to fall into the "tunnel vision" mode when debugging and continue down the wrong fork in the trail, hoping that in the end, the solution will come by going the wrong way faster.

Sometimes all that is needed is for the devs to feel they have permission to take a step back, breath a bit, and approach it afresh.

You ended it on a positive note, I think it was well written.

haacked@gmail.com (Haacked)

Tuesday, June 21, 2005 7:31:27 PM (Pacific Standard Time, UTC-08:00)

I think it's important to do the reminders everyonce in a while, and an ocassion as this seems to drive the point better than if you just sent the info out of the blue (imho). That said, regarding your questions:

1. Yes, I do, at least on some teams I worked on. However, the case I've done it with is when I see some repeated mistakes or things that can be improved...
2. Rarely. People never get private offices around here, so talking to someone usually just involve slightly raising your voice or looking up from the keyboard. I'll let you decide for yourself if this is good or bad :)
3. I have no opinion whatsoever on this :)
4. I believe it can usually be attributed to two things that are not directly pressure-to-ship related: a) people tend to always assume their assumptions hold, even when they obviously leak, and b) distraction, pure and simple. Well, I guess both are human nature, after all :)
5. oh yeah. been there, done that. I'm officially known around the company as the hot potato handler, since they tend to dump all that shit on me when a team can't seem to fix a problem after working long on it... it's good sometimes, but it also sucks big time others.
6. I thought it was ok. Probably much inocent that what I might have said :)

Tomas Restrepo

Tuesday, June 21, 2005 8:40:46 PM (Pacific Standard Time, UTC-08:00)

First principals are like parents advise: you will ignore it until *you* get hurt. You learn them from experience but can't just "inject" in to people, surely not just by emailing them ;).

As far as email culture goes, here's your medicine: switch exclusively to IM. We often have prefered IM over meetings, emails and every other medium of communication and use it even if my collegue is sitting just 10 ft away. Most people I've met at work who insisted walking in to each others offices or do email (some even insisted always replieng emails by a phone call!) were one of those rusted obsolete fat "very senior" 39+ dudes.

Shital Shah

Wednesday, June 22, 2005 12:33:43 AM (Pacific Standard Time, UTC-08:00)

>>The plural of “anecdote" is not “truth”
ROTFL

Daniel Moth

Wednesday, June 22, 2005 1:02:42 AM (Pacific Standard Time, UTC-08:00)

Do you ever send emails to the team to remind folks of First Principles?
I have a few times. Of course it seems that I come off as arrogant when I do.

Do you have a growing ReplyToAll culture that should be stamped out and replaced with walking to someone's office?
Yes, happens every hour of every day.

How long should an email thread go before it is stopped and replaced with a meeting?
I let them go twice through my inbox then I either go visit the person or if it needs more than a few people I'll call a meeting with the goal to get the issue resolved.

Is it the pressure to ship that causes all of us to miss misplaced semicolons and the like, or just human nature?
Both. Crunch mode does not work and overtaxed developers are like dogs that have run too many races in a day. They just don't have the focus or energy and no matter how sweet that rabbit is, they would rather be out drinking beers. There's also the human factor but then architects aren't human so we don't make these mistakes ;)

Is "flaming potato" a part of your software engineering culture? Do you struggle with it?
We seem to always be fighting fires which just doesn't make sense to me. Why not stop making buildings constructed with kindling in the first place?

Too harsh? Not harsh enough?
Was a good example. Positive feedback but you highlighted what needed to be done so hopefully it got the point across to those that mattered.

Bil Simser

Wednesday, June 22, 2005 2:29:34 AM (Pacific Standard Time, UTC-08:00)

My blog post about the proper tools for a job (see http://drazen.dotlic.name/weblog/archive/2004/09/15/174.aspx) was inspired by exactly the same event - a colleague was editing important XML file with Notepad and forgot to close a tag (or was missing just the slash, doesn't matter).

I guess some people never learn ;)

Drazen Dotlic

Wednesday, June 22, 2005 5:50:45 AM (Pacific Standard Time, UTC-08:00)

I'm missing the human problem. Somebody accidently put a space in an end tag. Humans do that sorta thing. Where was the check-in suite that validated the XML before it became part of the official build?

Chris Sells

Wednesday, June 22, 2005 6:19:57 AM (Pacific Standard Time, UTC-08:00)

a bit condescending in tone. Better to phrase everything in the positive. Especially in mass emails.

Peter Kellner

Wednesday, June 22, 2005 6:44:51 AM (Pacific Standard Time, UTC-08:00)

Why do I keep thinking about this? and commenting again? It's because your email is the thing we all want to send but doesn't pass muster. Best to put in drafts folder and delete later. Here is a specific example why.

"TextPad isn’t an XML editor."

Hoped thoughtful response from engineer: "Gee, Scott is right. I am a clumbsy and lazy programmer because I do use textpad rather than an XML editor sometimes. I will now change my ways and always use a real XML editor no matter how much of a hurry I am in and only have one small change to make. Thank you scott for pointing out my errant ways and I will be better next time"

... I don't think so.

Peter Kellner

Wednesday, June 22, 2005 6:54:20 AM (Pacific Standard Time, UTC-08:00)

>> TextPad isn’t an XML editor

but what about Notepad2? ;-)

Luke Hutteman

Wednesday, June 22, 2005 7:03:14 AM (Pacific Standard Time, UTC-08:00)

I'll try to comment on the non-technical part of the situation. People often misinterpret the content and tone expressed in email messages. I'm always extra-careful (like you were) when sending email messages to a group of co-workers talking about any problem that one (or more) of them were involved. But, it doesn't matter. There's always someone that gets defensive and feels like he/she should start explaining what happened and dodging from finger-pointing.
Emails are a blessing and a curse in the office... When you start using emails to avoid talking in person (I know a bunch of people like this), then you start losing the ability to understand what the other people "meant by that"... You just don't get to know your co-workers well enough to be able to exchange short email when you need to.. Every email starts to need a long disclaimer and advance excuse-me's so you don't hurt nodoby's feelings.
That only contributes to the unfair stereotype of us techies being socially-advert, self-centered outcasts. Speak up, people.

Sergio Pereira

Wednesday, June 22, 2005 7:55:17 AM (Pacific Standard Time, UTC-08:00)

Peter...I was kind of hoping for "Gee, I've been stuck in a rut and using TextPad to edit files since before XML came around. Maybe I could be more effective with another tool?"

Chrils...This was a configuration file on a built system that was failing early and silently. There wasn't a build suite that could catch it. It was a 3rd party component that failed, but it was looking to OutputDebugString. The problem was found when using DbgView.exe.

Scott Hanselman

Wednesday, June 22, 2005 8:13:00 AM (Pacific Standard Time, UTC-08:00)

Personally, I think we rely too much on email for communication. Such emails (and I have sent my share) does nothing for the psychology of the team. It's been my experience to try to humanize software engineering as much as possible, since, unfortunately, it is still a very human process.

The messages are usually harmless and correct - the medium isn't always.

Joel Semeniuk

Wednesday, June 22, 2005 8:55:51 AM (Pacific Standard Time, UTC-08:00)

The problem with emails is that they lack the part of communication that is non-verbal and is equally as important as the actual words themselves. I dont know how you are at work, I don't know how you are with your team or how they feel about you. With that in mind, I would have BRIEFLY made those points in a team meeting. As it stands (again with the caveats above) the email you sent sounds very annoying. Almost like an elementary school teacher talking to pupils. There is an old saying "Unsolicited advise is criticism". Compound that with the fact that its such an annoying bug, if I were working on it I would feel pretty upset with myself. I wouldnt need the parental sounding advise from "The Hanselman". Sort of like "now now kids, this is your daddy speaking here.." But thats just me..... Lets talk about you. Why do you feel it necessary to post this on a public blog? Are you looking for affirmation that you are doing your job? Are you looking for feedback on your management style? Scott, your a brilliant developer and I admire your work, but posting a running commentary of company communications (for lack of a better term) was not one of your best ideas. If you need help with interpersonal communication and management there maybe more qualified resources available to you than the developers and geeks (like myself) who enjoy your blog. It sounds almost as though you are looking for affirmation and validation. If 7 out of 10 readers agree with your points, does that make these points any less annoying to the recipients of your email? Then again, maybe your team loves you and fully understand the intention of the email. And maybe I'm just blabbering along here...... Good luck nonetheless.

Thomas Wagner

Wednesday, June 22, 2005 9:16:02 AM (Pacific Standard Time, UTC-08:00)

Do you ever send emails to the team to remind folks of First Principles? - No its pedantic and demoralising, and can be seen as trying to belittle or collegues.

Do you have a growing ReplyToAll culture that should be stamped out and replaced with walking to someone's office? - No email as needed, if a meetings required we have one.

How long should an email thread go before it is stopped and replaced with a meeting? - Depends on the thread... how long is a piece of string?

Is it the pressure to ship that causes all of us to miss misplaced semicolons and the like, or just human nature? - Human Nature

Is "flaming potato" a part of your software engineering culture? Do you struggle with it? No and N/A

Too harsh? Not harsh enough? Depends on the situation

Atul Thakor

Wednesday, June 22, 2005 9:18:06 AM (Pacific Standard Time, UTC-08:00)

Scott, was the file generted by the third party app or was it included in your build. Either way why was this not caught by low or high level tests at this area. Those tests should have failed the build. It could be that this was the case and the build was never put before a user, I would hope so.

Andrew Stopford

Wednesday, June 22, 2005 9:39:31 AM (Pacific Standard Time, UTC-08:00)

Thinking more about this... I dont think it is a fundamental programming knowing your principles problem. Maybe this is is a different lesson to be learned.

'Validate your XML with a DTD or XSD'

It should be the job of the Team Leader/ Development Manager to update your code practices and ensure that the developers are aware of XML validation when dealing with XML
and you've cured this problem at the source.

The team will feel happier as they've improved their programming practices and the team can rest assure this type of problem should not accure again.

Rather then "Keep it simple and hopefully it wont happen again" ;-)

Atul Thakor

Wednesday, June 22, 2005 9:54:34 AM (Pacific Standard Time, UTC-08:00)

A team is a group of people working towards a common goal, until something goes wrong and the finger pointing starts.

Wallym

Thursday, June 23, 2005 4:24:41 AM (Pacific Standard Time, UTC-08:00)

You've hit on exactly my point here. I believe you are 100% right in your thinking and intentions with the email. (and I am over 39 and still use email for the record). I think the way you would get the response below is with an email something like this which is less condescending and more respectful. (Unless you actually have stupid engineers doing this and therefore you should be condescending).

"Folks, I'm sure you are all aware of the problems that can be caused by miss typed and checking in code without matching xml tags. Today, several of us wasted many hours chasing down a problem that was caused by this. Years ago I learned the hard way never to edit XML in any editor that doesn't verify XML files before saving them. Please please please, let's not have any more hard lessons. Everyone, ALWAYS verify XML correctness with a tool before checking it in, and I would strongly suggest using a good syntax checking XML editor such as ???. I appreciated everyone's hard work in tracking this down and lets make it our last problem like this."

OK, it's 5:20 in the morning and my writing is not at it's best, but I do think this style is more affective at achieving the desired result.

From Scott:
Peter...I was kind of hoping for "Gee, I've been stuck in a rut and using TextPad to edit files since before XML came around. Maybe I could be more effective with another tool?"

Peter Kellner

Friday, June 24, 2005 12:28:34 AM (Pacific Standard Time, UTC-08:00)

I like this blog for its technical content. However, I feel this entire blog -- and, perhaps, your entire leadership style -- would be a lot better if you stuck to that excellent technical content and spent less time being amused with yourself.

Proving you know the latin for "keep it simple" ... claiming you know more about Africa than people from Africa ... using your too-frequent faux white boy hip-hop phrases all detract from your message in some very meaninful ways. Instead of seeing you as a guide, people are more likely to see you as full of yourself.

If I got the mail above from you I'd harbor some resentment for a pretty long time.

You're a smart guy. We already know that. You don't need to keep overdoing it in trying to prove it to us.

Friday, June 24, 2005 9:00:09 AM (Pacific Standard Time, UTC-08:00)

PH...ouch. Why do people who leave comments like these never leave their name, email or website? I wish you would so we could chat more about your feedback. Thanks for your opinion.

The Latin was a joke. It's not Latin for KISS, it's Latin for Occam's Razor...I guess the joke didn't work.

Faux White Boy phrases? I'm a fake white guy? Not sure what you mean here, but I don't tend to use too much urban language in my posts. Unless "dig" suddently came back in style and "ReplyToR'ing" is Hip-Hop. ("ReplyToR" means to Reply to Everyone or Reply to All. It's geek speak.)

As far as the Africa thing, the Hotel Rwanda stuff was just a post about an opinion. I've stayed in Africa, half my family is there, and I've studied in college. It was just a short post with a thought. Rwandans have since replied and confirmed it. It wasn't meant to show off, just to provide commentary.

I'm sorry it offends. I will give your feedback some thought. Thanks!

Scott Hanselman

Friday, June 24, 2005 4:32:34 PM (Pacific Standard Time, UTC-08:00)

I think maybe you should have taken your own advice: deliver this message in person rather than email.

Other than that criticism, it's totally reasonable.

Jeff Atwood

Comments are closed.