First Rule of Software Development
We had a great email go out today from Brian Windheim, one of the Architects here. He said I could share it. It was sent internally to his team. I like Brian's style.
Frank broke the First Rule of Fight Club Software Development yesterday:
Never commit code just before you leave for the day.
The CCNET (Continuous Integration) build subsequently broke – despite the fact that he ran a local build first – and team members who were still in the office had difficulty progressing with their work for several hours.
First, the remedy. If this ever happens again, simply revert the source repository to the prior known-good state using SVN’s revert changes. Don’t try to apply band-aids, and don’t waste time solving a problem that the original developer could probably solve in about two minutes. Using the tools correctly saves oodles of time.
Second, the reaction. Folks, don’t panic. Ever. If a quick peek doesn’t find the answer, don’t be afraid to pick up the phone and call the dude who broke the build. In 15 years of software development I have many times been the guy that broke the build, fixed the build, got mad when I was working at midnight to fix something broken by the guy that just went to Mexico on vacation for two weeks, reprimanded the guy the broke the build, been reprimanded for breaking the build, and so on. Get used to it.
Third, the lesson. Don’t commit things without waiting for CCNET to tell you that you can go home. All the more reason to have a faster build, no?
Broken builds break hearts, I say.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
There's possibly a more fundamental problem in play here... It might be a Whole Team failure that allows for the circumstances where one guy is doing some solo coding on a Friday evening. The times that I've had to be that guy were times when our team practices got out of kilter, and I've broken the build in those circumstances. We recognize in these times that a broken build is the least of our problems and that it's merely symptomatic of deeper team problems.
Inevitably people will break the build. Your system should be robust enough to handle it when it happens.
I use mercurial at work while everyone else uses perforce. We'll occasionally get the ``nobody update, build broken!'' emails, but I just work from a working revision and let everyone else deal with that fire.
I set up the CI system, and it's useful to know at what points the tree is OK, but I'd rather have my developers checking in code that *almost* worked and then going away for (hopefully just) a weekend than being afraid that all the other team members won't be able to function if the code isn't perfect.
Using a decent SCM, the other developers can just back up to the last good revision and commit their work as a child of that, leaving the broken build hanging around as a "micro branch" for the original developer to fix up when they return. No harm, no foul.
What's a decent SCM? (out of curiousity) I'm not saying that this is rocket science, nor am I advocating for any dunce caps. I think that it's reasonable to avoid breaking the build if possible. It's easy to revert changes locally, but on teams of dozens or more, it's more reasonable to revert those changes on the server, so the dozens who are doing Updates don't have to think about the broken build.
@Peter - That's an interesting point. Do you have a system like this? I haven't used any, personally, that self-heal, but perhaps I'm missing a big one that is out there that does this for you.
Should this problem be solved with local revisions/sandboxes or should it be solved by auto-reverts at the server?
...so, in my team, we have an original "Lindt - Maitre Chocolatier" (master of chocolate makers, I think) badge for the guy who breaks the build. He will wear the Maitre Chocolatier badge instead of our company badge :-)
We also used to have the Pingu of shame that the dev had to wear upon breaking the build =)
Its like making a big deal about a stock broker who picks a dog, a carpenter who makes a bad cut etc...
And in a related vein, I have to agree with @Scott (Bellware) - breaking the build should not be cause for ridicule. That type of behavior is a sure-fire sign of larger problems with the team... even if the team doesn't think so at the time.
We used to have the equivalent of a dunce hat at an old employer and as much as I hated to see red in my CC tray icon, I liked knowing some poor dev was being publicly called out. It was only after leaving that job and moving to a much more supportive and collaborative that I realized that practices like the dunce hat only caused friction within the team.
Comments are closed.