Scott Hanselman

browse by category or date

This Developer's Life 2.0.2 - Pressure

May 17, 2011 Comment on this post [9] Posted in Podcast
Sponsored By

clampIn this second episode of the second season of This Developer's Life we talk to a group of Top Gun developers: the StackOverflow team, about the day their database melted - and how the team responded.

      Also, please visit the new site at http://thisdeveloperslife.com. We've got sixteen episodes so far, and we are pretty proud of them. Don't listen to podcasts? Don't commute? Surely you have a long plane flight coming up? Maybe a cross country drive? Load up. It's all free.

      You can download the MP3 here (53 minutes) and visit our site at http://thisdeveloperslife.com.

      Please consider subscribing with iTunes, or Zune. If you enjoy it, take a moment and please do REVIEW our show on iTunes.

      Or if you have a BitTorrent client and would like to help save us bandwidth money, as well as the bragging rights of downloading legal torrents via RSS, get our Torrent Feed at ClearBits.

      The bandwidth and other costs for this week's show were picked up DevExpress and CodeRush! Visit them and thank them on Twitter.

      This Developer's Life is brought to you by CodeRush for Visual Studio.

      This Developer's Life is brought to you by CodeRush for Visual Studio.

      Announcing our listener contest...This Developer's Life - Crowdsourced

      Oh yes. We want to hear your stories. Record your best developer stories and send them to us and if we think they rock, we'll include them in the next episode of This Developer's Life.

      What we need from you:

      • Your story. We don't want interviews, we want stories. Tell us about your passion, or something crazy that happened at work while solving some technical problem.
      • Keep your audio clean. Use a decent microphone or at least make sure you don't "overdrive" your microphone by talking to close or two loudly. Don't record while mowing the lawn and don't record in a giant echo chamber.
      • Be passionate. Talk to us like you're talking to a friend.
      • Don't worry about editing or music. Just share. We'll handle the Lady Gaga mashups.
      • Note we may move your audio around or change the order of stuff to make it more listenable or interesting or both.
      • Change the names of companies and people to protect the innocent (or guilty)
      • Know that by giving us your audio you're releasing it the Creative Commons and that we may or may not use it for a future show.

      Send us a link to your audio file and what you're talking about and we'll do the rest. See you next time!

      About Scott

      Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

      facebook twitter subscribe
      About   Newsletter
      Hosting By
      Hosted in an Azure App Service
      Comment on this post [9]
      Share on Twitter or Facebook or use the
      May 18, 2011 0:09
      Having spent a couple decades working on enterprise manufacturing and financial systems my first reaction was that this was a lot of sturm und drang over the potential loss of some question and answer data. I get that high availability and data integrity credibility is important for a site for programmers, but it would have been nice to get some sense of the financial impact the outage had (as well as the impact of the loss of questions and answers). You probably can't get folks to talk about the time that a Dow 30 company's manufacturing line or order processing systems went down and they lost millions of dollars due to a few hours of systems downtime. And by the same token I suspect Jeff isn't going to lay out how much ad revenue StackOverflow lost due to the outage.

      My overall feedback is it would have been nice to at least mention the fact that there are financial consequences to downtime and data loss, and they need to be weighed vs. the time and effort to recover. I suspect that a decision making process not colored by the pride or ego factors of showing other programmers that you can recover from even the most heinous database corruption, but rather on strict financial cost/benefits, would have led to a decision to restore the backup and get the ad revenue flowing again ASAP rather than take the downtime to recover the lost Q&A data.
      patja
      May 18, 2011 1:50
      So, how was the corrupt database actually fixed/recovered?
      Robert G
      May 19, 2011 18:06
      What was the soundtrack to this podcast because it was rockin!
      Jamie Coates
      May 20, 2011 3:51
      Way too light on and only stated the obvious. Needed more detail as @patja said.
      DotNet Dude
      May 20, 2011 6:20
      Hey there, I just wanted to be OCD and point out that in the second dot point on tips for recording stories, the too's are written as to and two.
      Luke Fitz
      May 20, 2011 16:11
      @petja

      We tried to recover our corrupt db file and were unable to get it to a steady state, within an hour we decided to go with the backup, I then used the corrupt db as a source and sucked in all the information into the good / live db. It was tricky, I had to deal with remapping cascades of identity at 3am my time. Still objectively we are talking a few hundred posts on a free website, its not like lives would be lost if I messed up, or planes would crash. Still on a personal note there was heavy pressure.

      It's tricky, the podcast deals with the emotions and story, its less about being a lesson for other dba out there.
      SamSaffron
      May 24, 2011 3:13
      Jamie - All the music is listed at http://www.thisdeveloperslife.com/post/2-0-2-pressure
      Scott Hanselman
      May 31, 2011 11:58
      This show was rockin!
      Abrian Stemmet
      June 06, 2011 22:51
      I'm trying to "help save you bandwidth money, as well as gain the bragging rights of downloading legal torrents via RSS by getting your Torrent Feed at ClearBits" but episode 2.0.2 still isn't available from the torrent feed (http://www.clearbits.net/creators/178-this-developers-life).

      This would not be such an issue, but all of the locations I can get the MP3 from are blocked by my employer, so I would like to torrent the MP3 to my home computer, then FTP it from there.

      Thanks.
      Name

      Comments are closed.

      Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.