First Rule of Software Development

July 20, 2007 11:37 PM Comments [19] Posted in Programming

fight-club-dvd We had a great email go out today from Brian Windheim, one of the Architects here. He said I could share it. It was sent internally to his team. I like Brian's style.

Frank broke the First Rule of Fight Club Software Development yesterday:

Never commit code just before you leave for the day.

The CCNET (Continuous Integration) build subsequently broke – despite the fact that he ran a local build first – and team members who were still in the office had difficulty progressing with their work for several hours.

First, the remedy.  If this ever happens again, simply revert the source repository to the prior known-good state using SVN’s revert changes.  Don’t try to apply band-aids, and don’t waste time solving a problem that the original developer could probably solve in about two minutes.  Using the tools correctly saves oodles of time. 

Second, the reaction.  Folks, don’t panic.  Ever.  If a quick peek doesn’t find the answer, don’t be afraid to pick up the phone and call the dude who broke the build.  In 15 years of software development I have many times been the guy that broke the build, fixed the build, got mad when I was working at midnight to fix something broken by the guy that just went to Mexico on vacation for two weeks, reprimanded the guy the broke the build, been reprimanded for breaking the build, and so on.  Get used to it.

Third, the lesson.  Don’t commit things without waiting for CCNET to tell you that you can go home.  All the more reason to have a faster build, no?

Broken builds break hearts, I say.

Comments [19]
« Hanselminutes Podcast 73 - The Interns | Home | Task: Remove a Programmatic Crutch »
Friday, July 20, 2007 11:48:36 PM UTC
Fourth, the consequences. Keep a dunce cap around (or some other anti-trophy) so that he-that-broketh-the-build. shall be shamed most publicly. ;)
Kevin Dente
Saturday, July 21, 2007 3:18:08 AM UTC
We used to use a version of the dunce cap, but we stopped using it. There should be no consequences for breaking the build on a team that keeps the build clean as a simple matter of respect for the rest of the team. The CI server is there to be the defensive line of the codebase. It's ok if it gets knocked down every once and a while. It happens. As long as it tells you that it found a problem.

There's possibly a more fundamental problem in play here... It might be a Whole Team failure that allows for the circumstances where one guy is doing some solo coding on a Friday evening. The times that I've had to be that guy were times when our team practices got out of kilter, and I've broken the build in those circumstances. We recognize in these times that a broken build is the least of our problems and that it's merely symptomatic of deeper team problems.
Scott Bellware
Saturday, July 21, 2007 5:56:24 AM UTC
Blah, this is just evidence of a broken source code management system. Using a decent SCM, the other developers can just back up to the last good revision and commit their work as a child of that, leaving the broken build hanging around as a "micro branch" for the original developer to fix up when they return. No harm, no foul.
Good Tools Are Your Friend
Saturday, July 21, 2007 5:57:59 AM UTC
I wholeheartedly agree with the don't go home until you confirm that your build is clean. I am also a believer in live together or die alone. The team succeeds or fails together.
Phillip Forteza
Saturday, July 21, 2007 6:10:32 AM UTC
Regarding this comment, the last thing a healthy team needs is macho marine-style hazing whenever simple, honest mistakes are made.
No Thugs
Saturday, July 21, 2007 6:10:42 AM UTC
You are looking at this all wrong. You should have tools to protect you against this instead of requiring the developer to do this work (automate remedial tasks). If a checkin breaks your build or causes test failures, your system should automatically revert that checkin for you (and alert the developer).

Inevitably people will break the build. Your system should be robust enough to handle it when it happens.
Peter
Saturday, July 21, 2007 6:47:37 AM UTC
+1 better tools.

I use mercurial at work while everyone else uses perforce. We'll occasionally get the ``nobody update, build broken!'' emails, but I just work from a working revision and let everyone else deal with that fire.

I set up the CI system, and it's useful to know at what points the tree is OK, but I'd rather have my developers checking in code that *almost* worked and then going away for (hopefully just) a weekend than being afraid that all the other team members won't be able to function if the code isn't perfect.
Dustin
Saturday, July 21, 2007 7:23:59 AM UTC
Using a decent SCM, the other developers can just back up to the last good revision and commit their work as a child of that, leaving the broken build hanging around as a "micro branch" for the original developer to fix up when they return. No harm, no foul.


What's a decent SCM? (out of curiousity) I'm not saying that this is rocket science, nor am I advocating for any dunce caps. I think that it's reasonable to avoid breaking the build if possible. It's easy to revert changes locally, but on teams of dozens or more, it's more reasonable to revert those changes on the server, so the dozens who are doing Updates don't have to think about the broken build.

@Peter - That's an interesting point. Do you have a system like this? I haven't used any, personally, that self-heal, but perhaps I'm missing a big one that is out there that does this for you.

Should this problem be solved with local revisions/sandboxes or should it be solved by auto-reverts at the server?
Scott Hanselman
Saturday, July 21, 2007 11:11:44 AM UTC
I don't know about commercial SCMs, but in the free software space, Monotone is designed around exactly that sort of work flow, and thus is what I consider a decent SCM. It will take a little time to warp your head around the model, but you won't regret it.
Good Tools Are Your Friend
Saturday, July 21, 2007 1:58:16 PM UTC
We use "Break the Build - Buy a round of beers"
Raj
Sunday, July 22, 2007 3:28:57 AM UTC
Close on the heels of this: never release software on a Friday - unless you like working on the weekends.
Tom Opgenorth
Sunday, July 22, 2007 1:16:20 PM UTC
We have a system where all source submissions must build and pass the quality bar, e.g. unit tests, code coverage, fxcop, etc, *before* it's committed. There's no way for anyone to break any build but on their own local machine. It's wonderful.
Chris Sells
Monday, July 23, 2007 9:05:04 AM UTC
In italy, we use the term "chocolate maker" for someone who does something without care. I don't know where this term came from...

...so, in my team, we have an original "Lindt - Maitre Chocolatier" (master of chocolate makers, I think) badge for the guy who breaks the build. He will wear the Maitre Chocolatier badge instead of our company badge :-)

Filini
Monday, July 23, 2007 2:35:32 PM UTC
We call that "doing a Pepper" after one of the guys who did this most evenings.

We also used to have the Pingu of shame that the dev had to wear upon breaking the build =)
Robbie
Tuesday, July 24, 2007 3:07:04 AM UTC
While investigating some continuous integration build systems, we found out that JetBrain's TeamCity (same folks behind the excellent Resharper tool) has a nice feature which allows a developer to run a personal build (compile, run code base under static code analysis and code coverage, execute unit tests, etc.) and once it's green, the build system will actually commit all changes to the source repository. Hope it helps! :)
Brian Di Croce
Tuesday, July 24, 2007 5:49:48 PM UTC
Wow broke the build, as a far as i know just fix it and move on as a group we are pretty anal its just a cost of doing business, i have worked with some top notch people all of whom eventually break the build...

Its like making a big deal about a stock broker who picks a dog, a carpenter who makes a bad cut etc...

Strectch
Wednesday, July 25, 2007 4:05:28 AM UTC
@Stretch: Agreed! Breaking the build is an occupational hazard - deal with it and move on.

And in a related vein, I have to agree with @Scott (Bellware) - breaking the build should not be cause for ridicule. That type of behavior is a sure-fire sign of larger problems with the team... even if the team doesn't think so at the time.

We used to have the equivalent of a dunce hat at an old employer and as much as I hated to see red in my CC tray icon, I liked knowing some poor dev was being publicly called out. It was only after leaving that job and moving to a much more supportive and collaborative that I realized that practices like the dunce hat only caused friction within the team.
Steven Harman
Wednesday, July 25, 2007 5:42:16 AM UTC
Just to be clear @Steven and @Stretch, I agree that Dunce Caps and ridicule are totally inappropriate, and I also agree that a self-healing build is ideal. So, yes, the tools need to be improved. The point of this post was "Never commit code just before you leave for the day." which is very different from "don't break the build" which I recognize is specious.
Scott Hanselman
Wednesday, July 25, 2007 4:08:54 PM UTC
A coincidence that this follows closely after my own post on Keeping It Green. As a vendor of a CI tool (Pulse) that supports testing your changes before you commit them, I have to agree that tools can really help here :). Still, the most important element is culture, where people understand the value of CI and keeping the build happy.
Jason Sankey
Comments are closed.
Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer’s view in any way.