A Bug Report is a Gift
Sponsored By
There were lots of reactions to my blog post Everything's broken and nobody's upset. Some folks immediately got the Louis CK "Everything's Amazing and Nobody's Happy" reference. Some folks thought it was a poorly worded rant. Some folks (from various companies) thought I was throwing developers under the bus, accusing them of not caring. Others saw a meta-goal and started a larger discussion about software quality.
The questions I asked were these...but the most significant one was added a few hours later, suggested by a reader.
- Is this a speed problem? Are we feeling we have to develop too fast and loose?
- Is it a quality issue? Have we forgotten the art and science of Software QA?
- Is it a people problem? Are folks just not passionate about their software enough to fix it?
- It is a communication problem? Is it easy for users to report errors and annoyances?
After the post I went back and tried to file bug reports for all the issues I bumped into. For products where I couldn't find an easy bug reporting site I used Twitter. Google and Microsoft were universally pleasant when I reported the bugs and seemed sincerely interested in helping.
Is it easy for your users to report a bug? Does you app automatically report crashes? Is your site friendly? Are you support people friendly and engaged? Are your bug support forums filled with reports but no company representatives? Do you use GetSatisfaction or UserVoice?
Anatomy of a Good Bug Report
Ideally the "best" bugs are those that can be reproduced given enough context. You can save developer time and trouble by giving them as many details as you can. Developers can save everyone time with bug reporting tools that collect that information for users.
For example, on Windows if you run "msinfo32.exe" you can get a complete snapshot of your system.
Send a Frown and Send a Smile
I am a big fan of the "Send a Frown" way of getting bugs.
The Office 2013 Previews even have hotkeys for this!
Sending feedback is even better with screenshots.
The help menu for Skype includes a Give Feedback menu as well as "Skype Status."
As a user, if you can include information like:
- What were you doing? What were you trying to do?
- Advanced folks: What weird add-ins/extensions or hacks are you using that you haven't mentioned but you really should? *cough* Adblock *cough*
- Can you make it happen every time?
- Can you include a screenshot? Bonus points for a screencast!
This is a lot to ask of a user!
Some examples of where to file bugs
Here's some sites you can use to report bugs in certain applications. Note that some are fancy, some spartan, some just forums, some actual bug tracking software, made for and used by developers.
- Google Chrome/Chromium - http://new.crbug.com
- Firefox and all Mozilla products - https://bugzilla.mozilla.org/
- Spotify - http://www.spotify.com/se/help/support-forum/
- Internet Explorer - http://connect.microsoft.com/IE
- Visual Studio & ASP.NET http://connect.microsoft.com/visualstudio and
- Suggestions http://aspnet.uservoice.com and http://visualstudio.uservoice.com
Or, the ultimate place to file bugs for your favorite software, as my friend Anil points out:
@shanselman on Twitter, duh!
— Anil Dash (@anildash) September 24, 2012
How hard does the user have to work to file a bug? It's OUR bug but the user not only hit the bug but also has to work to report it!
Every click or manual step that our users have to invest in our bug is a click too many. A bug is the pain that hurts the user and keeps hurting as they report it! A good bug report is a gift that keeps on giving and we should treat it as such.
I'd love a world where all crashes are automatically reported and there's a "Send a Frown/Smile" button on everyone's computer that offers to record a short screencast with just a single "record/stop" button.
What product do you think has the best bug filing experience? Sound off in the comments!
This week's sponsor: Be part of GENERATION APP. Your Idea. Your App. 30 Days. Begin your 30-day journey to create a Windows Store style app and talk 1-on-1 with a Windows 8 app development pro. Get started today.
Related Posts in this Three Part series on Software Quality
- Everything's broken and nobody's upset
- A Bug Report is a Gift
- Help your users record and report bugs with the Problem Steps Recorder
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
Hosting By
Where can I file bug reports for Windows 8 and the included apps?
The support links inside the apps refer to "Microsoft Answers".
I posted some bugs there, but was not too happy with the reponses I got there. Most answers were some standard answers copy and pasted. Some did not even read my reports. Some made wild suggestions, such as reinstalling Windows.
Is there a better way?
The support links inside the apps refer to "Microsoft Answers".
I posted some bugs there, but was not too happy with the reponses I got there. Most answers were some standard answers copy and pasted. Some did not even read my reports. Some made wild suggestions, such as reinstalling Windows.
Is there a better way?
I totally agree with the sentiment of this post Scott. From a personal perspective I think it's critical to identify areas of software that users aren't getting on with. Further than this I think that the reporting/receiving of a bug report is only the beginning, what I believe matters equally (or possibly more) is how bug reports are responded to.
Does it indicate an area of the system which is too complex? does the report indicate a training issue? Is it a technical issue with client configuration? Is it a known issue? what's the ETA for a fix? is there a workaround?
Whilst initially it's a little downheartening to have bugs reported on something you've worked hard on I always try to take the positives, a) it proves people are using the system! b) they are engaged enough to provide feedback and c) the report and prospective fix will improve the quality for all the users.
I think point b above is really crucial, there's nothing worse than having a problem but not having the feedback mechanisms in place to become aware of it. Making bugs easy to report is key to improving quality. Even the best designed workflows and processes may fail/be improved after rigorous usage and feedback.
I'm also a supporter of getting products out to beta groups/power users early to pickup the issues which may tarnish user experience but can if identified early, can ensure the release is as polished as possible or identify issues in the system design which are more easily incorporated at the early stages of development.
Does it indicate an area of the system which is too complex? does the report indicate a training issue? Is it a technical issue with client configuration? Is it a known issue? what's the ETA for a fix? is there a workaround?
Whilst initially it's a little downheartening to have bugs reported on something you've worked hard on I always try to take the positives, a) it proves people are using the system! b) they are engaged enough to provide feedback and c) the report and prospective fix will improve the quality for all the users.
I think point b above is really crucial, there's nothing worse than having a problem but not having the feedback mechanisms in place to become aware of it. Making bugs easy to report is key to improving quality. Even the best designed workflows and processes may fail/be improved after rigorous usage and feedback.
I'm also a supporter of getting products out to beta groups/power users early to pickup the issues which may tarnish user experience but can if identified early, can ensure the release is as polished as possible or identify issues in the system design which are more easily incorporated at the early stages of development.
On this very issue, I *so* wish the Windows group would get on Connect.
The Visual Studio/.NET Framework group has it right there. I've gotten a brick for the last 4 releases of Visual Studio for all the bug reports/fixes I filed- I am more than happy to report bugs, especially when they're dealt with. It makes my life easier, and presumably others' as well.
The only way I know to file bugs against Windows is an awful process- open a support issue, fork over your credit card so they can ding you however many hundreds of dollars, then get ready to fight with the first-line flunky that gets assigned the issue until they understand it (or they get tired of it, ding you the hundreds of dollars anyway, then you repeat the process until you get someone that *does* get it and refunds the other one). Horrible- the whole process from the beginning screams, "Go away!"
The Visual Studio/.NET Framework group has it right there. I've gotten a brick for the last 4 releases of Visual Studio for all the bug reports/fixes I filed- I am more than happy to report bugs, especially when they're dealt with. It makes my life easier, and presumably others' as well.
The only way I know to file bugs against Windows is an awful process- open a support issue, fork over your credit card so they can ding you however many hundreds of dollars, then get ready to fight with the first-line flunky that gets assigned the issue until they understand it (or they get tired of it, ding you the hundreds of dollars anyway, then you repeat the process until you get someone that *does* get it and refunds the other one). Horrible- the whole process from the beginning screams, "Go away!"
@Martin:
@Doug:
Hit the nail on the head, both of you. It doesn't matter if a company makes it easy to file bugs if they're simply going to dismiss them all with a BS "Have you tried turning it off and on again" form letter, without so much as scanning over the report.
Or, worst case, the response is "Too bad, we're no longer working on this product" or "Too bad, we're not going to change it." That's the bug-response equivalent of "Go away."
K
@Doug:
Hit the nail on the head, both of you. It doesn't matter if a company makes it easy to file bugs if they're simply going to dismiss them all with a BS "Have you tried turning it off and on again" form letter, without so much as scanning over the report.
Or, worst case, the response is "Too bad, we're no longer working on this product" or "Too bad, we're not going to change it." That's the bug-response equivalent of "Go away."
K
For every reported bug there are probably ten other users who didn't bother. In one project, I enabled a global error handler which would automatically email me errors as they occurred along with all relevant information.
The result was that I would know exactly whenever an exception occurred and I could contact the user to get more info. Oftentimes, they would be pleasantly surprised (and perhaps a little creeped out) that I knew what was going with my system. I even had foresight to snapshot form submissions etc, so I could re-submit on their behalf, saving them the hassle.
It seems trivial in this age of web services to hook up a reporting mechanism into your apps. As long as there is someone there at the other end to deal with it.
The result was that I would know exactly whenever an exception occurred and I could contact the user to get more info. Oftentimes, they would be pleasantly surprised (and perhaps a little creeped out) that I knew what was going with my system. I even had foresight to snapshot form submissions etc, so I could re-submit on their behalf, saving them the hassle.
It seems trivial in this age of web services to hook up a reporting mechanism into your apps. As long as there is someone there at the other end to deal with it.
@Rob:
That's dangerously shaky legal ground. Wouldn't recommend it unless you're talking internal projects. Monitoring, for the US at least, has to be explicitly declared in most locales (No hiding it in an EULA).
That's dangerously shaky legal ground. Wouldn't recommend it unless you're talking internal projects. Monitoring, for the US at least, has to be explicitly declared in most locales (No hiding it in an EULA).
Totally agree about the quality of a bug report. I've written a few times myself on the anayomy of a bug report http://timheuer.com/blog/archive/2011/10/12/anatomy-of-a-good-bug-report.aspx
App store comments are also a good source of feedback. You could also provide a comment field in an uninstaller.
The easier it is to create a bug report the more you are likely to get. Overall I think this is a good thing, but it will take resources to handle it all. The quality is also likely to be very mixed.
I have to wonder how much this will all cost, and if there are any systems to group similar bug reports or make the process more efficient for people trying to resolve the bugs.
I have to wonder how much this will all cost, and if there are any systems to group similar bug reports or make the process more efficient for people trying to resolve the bugs.
Apple Maps - 'Report a problem' link available in the behind page area.
However, because I live in London and am 38 years old I don't have enough years left in me to be able to report all the problems.
However, because I live in London and am 38 years old I don't have enough years left in me to be able to report all the problems.
My favourite tool here is BugHerd, and they've been very successful because they capture exactly the right information and it's crowd-sourced by the people who use your app.
I also think it can't be stressed enough to make it easy for the user to create good, full bug reports.
I'm sure many of you have gotten those bug reports that say, "When I try to do so-and-such I get an error message." What's the error message? "Don't remember. Didn't write it down."
I'm sure many of you have gotten those bug reports that say, "When I try to do so-and-such I get an error message." What's the error message? "Don't remember. Didn't write it down."
We developed both a user feedback and automated error reporting (AER) feature for our Excel add-in, Analyse-it.
Both features allow the user to include a screenshot, if it helps shed light on the problem, and also include the document (an Excel workbook in this case) they were working with. For feedback the user can explain their question, and for AERs they can tell us what they were doing when the problem occurred. AERs also automatically include a stack trace and information about the environment - version numbers, threads, other add-ins, and so on.
On the server side for AERs we calculate a checksum based on the application version and stack trace so we can merge identical bug reports into a single bug report. If a user experiences a crash for a fixed issue we notify them a fix is available, or if the fix isn't publically released yet which release the fix is scheduled to appear in.
It was a lot of work initially, especially considering we're a small company, but the effort is definitely worth it. We've been able to zero-in on awkward bugs and fix them very quickly. And we've gathered great feedback from users with good ideas for improvements and new features. This only improves the quality of the application and shows customers we care about the product.
Both features allow the user to include a screenshot, if it helps shed light on the problem, and also include the document (an Excel workbook in this case) they were working with. For feedback the user can explain their question, and for AERs they can tell us what they were doing when the problem occurred. AERs also automatically include a stack trace and information about the environment - version numbers, threads, other add-ins, and so on.
On the server side for AERs we calculate a checksum based on the application version and stack trace so we can merge identical bug reports into a single bug report. If a user experiences a crash for a fixed issue we notify them a fix is available, or if the fix isn't publically released yet which release the fix is scheduled to appear in.
It was a lot of work initially, especially considering we're a small company, but the effort is definitely worth it. We've been able to zero-in on awkward bugs and fix them very quickly. And we've gathered great feedback from users with good ideas for improvements and new features. This only improves the quality of the application and shows customers we care about the product.
I for one think twitter is a horrible way to file bugs. As someone who has gotten to many "bug reports" or "feedback" tweets that say, "This doesn't work why not?" Or "blah blah sucks" I find 140 characters is simply not useful for describing issues and solutions. If you can't find a real bug reporting solution and question on StackOverflow followed by a tweet with a link to that has been much more effective for me being able to help my customers.
Ditto Doug McDonald.
Agree with the sentiment of this post.
Dream that we could formulate software pesticides.
Agree with the sentiment of this post.
Dream that we could formulate software pesticides.
Please don't encourage people to use Twitter for filing bugs. I contribute to an open source project and we have tons of problems with people tweeting bug reports and mentioning them on irc which means that tracking the problem, especially when you have thousands of them, is impossible.
People use Twitter to file bugs because, typically, no other way is made apparent. Are users REALLY going to go sign up for yet another account just to make your life easier?
There's really no shortage of bug-reporting frameworks in which to wrap an application, and no excuse (on new apps, at least) to not provide a built-in reporting mechanism.
There's really no shortage of bug-reporting frameworks in which to wrap an application, and no excuse (on new apps, at least) to not provide a built-in reporting mechanism.
What do you think of Support tickets being answered in a "woops, sorry about that" Kind of fashion- as opposed to the all to common "CREATED TICKET NUMBER UT-4528872. ANSWER: we found that...." -esque rigid support.
Our commercial web application uses a continuously running queue of logging statements to track what a user is doing. If an exception is encountered, we dump this queue with a copy of the current stack trace and other exception information into an email sent to our support department. We are generally able to find the source of the exception and release a hotfix that evening to correct the issue.
Keeping this queue may be considered "heavy", but each of our customers are siloed with their own copy of the application and generally have only around 100 users. It works for us, and we get rich bug reports without the user having to do anything. Our customers love the fact we can correct these occasional issues without them having to do anything special.
Keeping this queue may be considered "heavy", but each of our customers are siloed with their own copy of the application and generally have only around 100 users. It works for us, and we get rich bug reports without the user having to do anything. Our customers love the fact we can correct these occasional issues without them having to do anything special.
I'd like to suggest using the following keybindings instead:
That is, of course, if they aren't already taken. I don't have a way to test Office 2013 Previews atm.
The irony of requesting this here instead of at the Office site is not lost on me. :) <- CTRL+ALT+)
CTRL + ALT + ( Frowny-Face
CTRL + ALT + ) Smiley-Face
That is, of course, if they aren't already taken. I don't have a way to test Office 2013 Previews atm.
The irony of requesting this here instead of at the Office site is not lost on me. :) <- CTRL+ALT+)
Making it easy for the user to report a bug is crucial, of course. But almost as important: make it easy for the developer to include bug reporting support, too! For the .NET world, Red Gate's Smart Assembly does just that with the automated error reporting feature. Disclosure: I do not work for Red Gate but I write articles for their Simple-Talk.com site. Regardless of that association, though, I find their tools very good and very useful.
Reporting any type of bugs to Microsoft results in the auto generated message "It does not happen on our machine".
Bug closed!
Bug closed!
I thoroughly agree about good metadata being submitted with bug reports. For instance, a stack trace, preferably with line numbers (yes, you can get line nums in an app even in production code), the error message, etc. can be incredibly effective at reducing the time to resolve the bug.
But you have to be careful, esp in regulated industries (Legal or Medical). You can't be logging or including ANY data that might directly or indirectly ID a person.
And screenshots are notorious for this. The user might not even notice that SOC SEC number shown in the status bar of the app when they grab that screenshot for support, but a lawyer might, and it's the perfect vector for a lawsuit.
But you have to be careful, esp in regulated industries (Legal or Medical). You can't be logging or including ANY data that might directly or indirectly ID a person.
And screenshots are notorious for this. The user might not even notice that SOC SEC number shown in the status bar of the app when they grab that screenshot for support, but a lawyer might, and it's the perfect vector for a lawsuit.
I have a few Windows apps out there with from 45-1200 users each. The in-app manual error reporting with optional machine/appstate collection has done wonders for me and the occasional development partner. I still get a few user emails, of course - but I just ask the user to initiate the report from the app (and why I want it that way).
This has worked well for at least fifteen years, and has allowed me to be responsive, in contrast to other applications that the user encounters daily.
This has worked well for at least fifteen years, and has allowed me to be responsive, in contrast to other applications that the user encounters daily.
Don't forget about the wonderful "Steps Recorder" in Windows 8. I think it was called "Steps to Reproduce" or something like that in Windows 7. In Windows 8 just start typing "steps" and it should find the built in "Steps Recorder".
The user can run the utility and it will automatically take screen shots and note all the mouse clicks, etc. Try it NOW!
The user can run the utility and it will automatically take screen shots and note all the mouse clicks, etc. Try it NOW!
Thats not it...
http://www.infoq.com/presentations/Predictable-Failures
http://www.infoq.com/presentations/Predictable-Failures
I happened to come across both this post and "Everything's broken and nobody's upset" in the last few days.
I rarely comment on blogs, but in the spirit of "Sending a Smile" I shall break that habit to say thank you for a couple of very interesting articles.
I definitely agree with the sentiments of both.
I rarely comment on blogs, but in the spirit of "Sending a Smile" I shall break that habit to say thank you for a couple of very interesting articles.
I definitely agree with the sentiments of both.
Does anyone know what happened to Connect? A couple of months back, huge numbers of bug reports disappeared without explanation.
Some had been fixed, so I could sort-of* understand deleting them in a clean-up. But others had been closed as "won't fix", often due to time constraints on an upcoming release and with a promise to re-visit them at a later date. Not only have we lost the bug reports, we've lost the workarounds as well!
* Sort-of because some of the "fixes" were "upgrade to the next version", and it would have been nice to keep the workarounds for anyone stuck on the affected version.
Some had been fixed, so I could sort-of* understand deleting them in a clean-up. But others had been closed as "won't fix", often due to time constraints on an upcoming release and with a promise to re-visit them at a later date. Not only have we lost the bug reports, we've lost the workarounds as well!
* Sort-of because some of the "fixes" were "upgrade to the next version", and it would have been nice to keep the workarounds for anyone stuck on the affected version.
@Richard:
In the case of connect, a "Won't Fix" with a promise to revisit is less something that might be fixed in the future, and more something that will never be brought up again.
Unfortunately, they also tend to be removed if they're re-posted as still occurring in the new version of the software.
In the case of connect, a "Won't Fix" with a promise to revisit is less something that might be fixed in the future, and more something that will never be brought up again.
Unfortunately, they also tend to be removed if they're re-posted as still occurring in the new version of the software.
@K: But these weren't re-posts. Some were bug reports which had been on the system since .NET 1.1, which have now disappeared without a trace. My dashboard used to show about twenty feedback items that I'd submitted; it now has three.
@Scott:
I have enjoyed these blog posts. The main thing is that we talk about the problems and not just ignore them. I have already learned one valuable thing from the discussions in this post, which brings me to...
@Jeff LeBert:
Thank you so much for mentioning the Problem Steps Recorder. I had no idea that this existed in Windows 7. Working in support, I have dealt with some Issue Reporting Challenged Individuals and this will come in very handy!
I have enjoyed these blog posts. The main thing is that we talk about the problems and not just ignore them. I have already learned one valuable thing from the discussions in this post, which brings me to...
@Jeff LeBert:
Thank you so much for mentioning the Problem Steps Recorder. I had no idea that this existed in Windows 7. Working in support, I have dealt with some Issue Reporting Challenged Individuals and this will come in very handy!
If you have the time, please check this gift we sent to Microsoft Community Forum.
This is a couple years old problem, and we are having a really hard time to make them understand our exact problem. -.-
http://answers.microsoft.com/en-us/windowslive/forum/mail-email/wrong-time-being-shown-for-bangladeshi-users/c0dea752-abde-4a4e-9bdb-e15d63338db1
If our gifts are treated in this way - we might not be very eager to gift in future :/
This is a couple years old problem, and we are having a really hard time to make them understand our exact problem. -.-
http://answers.microsoft.com/en-us/windowslive/forum/mail-email/wrong-time-being-shown-for-bangladeshi-users/c0dea752-abde-4a4e-9bdb-e15d63338db1
If our gifts are treated in this way - we might not be very eager to gift in future :/
Hi Scott,
Thanks for the reply. Looks like everything (well, almost everything in Outlook) is broken (for a nation), everyone was upset, they kept reporting the same bug for the last 5 years at least - and nobody (at MS) took care of it.
I know, Microsoft produces really fantastic products and services. I am a MSP, I am using MS products for about a decade, they have an active community, have a cool bug reporting system and according to you - "google and Microsoft were universally pleasant when I reported the bugs and seemed sincerely interested in helping. " .
Except this one, maybe insignificant to MS, but unpleasant particular bug. A whole nation is suffering from it. And truly, nobody cares. The previous ones who reported gave up, the present people who were reporting were ignored. The ones who are supposed to response, didn't bother responding.
What should we do in such case, Scott?
Thank you for your time to read this awfully long comment coming from a frustrated sufferer.
Thanks for the reply. Looks like everything (well, almost everything in Outlook) is broken (for a nation), everyone was upset, they kept reporting the same bug for the last 5 years at least - and nobody (at MS) took care of it.
I know, Microsoft produces really fantastic products and services. I am a MSP, I am using MS products for about a decade, they have an active community, have a cool bug reporting system and according to you - "google and Microsoft were universally pleasant when I reported the bugs and seemed sincerely interested in helping. " .
Except this one, maybe insignificant to MS, but unpleasant particular bug. A whole nation is suffering from it. And truly, nobody cares. The previous ones who reported gave up, the present people who were reporting were ignored. The ones who are supposed to response, didn't bother responding.
What should we do in such case, Scott?
Thank you for your time to read this awfully long comment coming from a frustrated sufferer.
Do you mind if I quote a couple of your posts as long as I provide credit and sources back to
your webpage? My blog site is in the very same niche as yours and my users would really benefit from a lot of
the information you present here. Please let me know if this okay with you.
Thank you!
your webpage? My blog site is in the very same niche as yours and my users would really benefit from a lot of
the information you present here. Please let me know if this okay with you.
Thank you!
Comments are closed.
I concur with all you outlined, and find it coincidental that you posted this the same day as Darren Murph's article on Engadget about companies' ongoing relationships with customers being important. Bug reporting is another form of customer service.