A Bug Report is a Gift
There were lots of reactions to my blog post Everything's broken and nobody's upset. Some folks immediately got the Louis CK "Everything's Amazing and Nobody's Happy" reference. Some folks thought it was a poorly worded rant. Some folks (from various companies) thought I was throwing developers under the bus, accusing them of not caring. Others saw a meta-goal and started a larger discussion about software quality.
The questions I asked were these...but the most significant one was added a few hours later, suggested by a reader.
- Is this a speed problem? Are we feeling we have to develop too fast and loose?
- Is it a quality issue? Have we forgotten the art and science of Software QA?
- Is it a people problem? Are folks just not passionate about their software enough to fix it?
- It is a communication problem? Is it easy for users to report errors and annoyances?
After the post I went back and tried to file bug reports for all the issues I bumped into. For products where I couldn't find an easy bug reporting site I used Twitter. Google and Microsoft were universally pleasant when I reported the bugs and seemed sincerely interested in helping.
Is it easy for your users to report a bug? Does you app automatically report crashes? Is your site friendly? Are you support people friendly and engaged? Are your bug support forums filled with reports but no company representatives? Do you use GetSatisfaction or UserVoice?
Anatomy of a Good Bug Report
Ideally the "best" bugs are those that can be reproduced given enough context. You can save developer time and trouble by giving them as many details as you can. Developers can save everyone time with bug reporting tools that collect that information for users.
For example, on Windows if you run "msinfo32.exe" you can get a complete snapshot of your system.
Send a Frown and Send a Smile
I am a big fan of the "Send a Frown" way of getting bugs.
The Office 2013 Previews even have hotkeys for this!
Sending feedback is even better with screenshots.
The help menu for Skype includes a Give Feedback menu as well as "Skype Status."
As a user, if you can include information like:
- What were you doing? What were you trying to do?
- Advanced folks: What weird add-ins/extensions or hacks are you using that you haven't mentioned but you really should? *cough* Adblock *cough*
- Can you make it happen every time?
- Can you include a screenshot? Bonus points for a screencast!
This is a lot to ask of a user!
Some examples of where to file bugs
Here's some sites you can use to report bugs in certain applications. Note that some are fancy, some spartan, some just forums, some actual bug tracking software, made for and used by developers.
- Google Chrome/Chromium - http://new.crbug.com
- Firefox and all Mozilla products - https://bugzilla.mozilla.org/
- Spotify - http://www.spotify.com/se/help/support-forum/
- Internet Explorer - http://connect.microsoft.com/IE
- Visual Studio & ASP.NET http://connect.microsoft.com/visualstudio and
- Suggestions http://aspnet.uservoice.com and http://visualstudio.uservoice.com
Or, the ultimate place to file bugs for your favorite software, as my friend Anil points out:
@shanselman on Twitter, duh!— Anil Dash (@anildash) September 24, 2012
How hard does the user have to work to file a bug? It's OUR bug but the user not only hit the bug but also has to work to report it!
Every click or manual step that our users have to invest in our bug is a click too many. A bug is the pain that hurts the user and keeps hurting as they report it! A good bug report is a gift that keeps on giving and we should treat it as such.
I'd love a world where all crashes are automatically reported and there's a "Send a Frown/Smile" button on everyone's computer that offers to record a short screencast with just a single "record/stop" button.
What product do you think has the best bug filing experience? Sound off in the comments!
This week's sponsor: Be part of GENERATION APP. Your Idea. Your App. 30 Days. Begin your 30-day journey to create a Windows Store style app and talk 1-on-1 with a Windows 8 app development pro. Get started today.
Related Posts in this Three Part series on Software Quality
- Everything's broken and nobody's upset
- A Bug Report is a Gift
- Help your users record and report bugs with the Problem Steps Recorder
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
The support links inside the apps refer to "Microsoft Answers".
I posted some bugs there, but was not too happy with the reponses I got there. Most answers were some standard answers copy and pasted. Some did not even read my reports. Some made wild suggestions, such as reinstalling Windows.
Is there a better way?
Does it indicate an area of the system which is too complex? does the report indicate a training issue? Is it a technical issue with client configuration? Is it a known issue? what's the ETA for a fix? is there a workaround?
Whilst initially it's a little downheartening to have bugs reported on something you've worked hard on I always try to take the positives, a) it proves people are using the system! b) they are engaged enough to provide feedback and c) the report and prospective fix will improve the quality for all the users.
I think point b above is really crucial, there's nothing worse than having a problem but not having the feedback mechanisms in place to become aware of it. Making bugs easy to report is key to improving quality. Even the best designed workflows and processes may fail/be improved after rigorous usage and feedback.
I'm also a supporter of getting products out to beta groups/power users early to pickup the issues which may tarnish user experience but can if identified early, can ensure the release is as polished as possible or identify issues in the system design which are more easily incorporated at the early stages of development.
The Visual Studio/.NET Framework group has it right there. I've gotten a brick for the last 4 releases of Visual Studio for all the bug reports/fixes I filed- I am more than happy to report bugs, especially when they're dealt with. It makes my life easier, and presumably others' as well.
The only way I know to file bugs against Windows is an awful process- open a support issue, fork over your credit card so they can ding you however many hundreds of dollars, then get ready to fight with the first-line flunky that gets assigned the issue until they understand it (or they get tired of it, ding you the hundreds of dollars anyway, then you repeat the process until you get someone that *does* get it and refunds the other one). Horrible- the whole process from the beginning screams, "Go away!"
Hit the nail on the head, both of you. It doesn't matter if a company makes it easy to file bugs if they're simply going to dismiss them all with a BS "Have you tried turning it off and on again" form letter, without so much as scanning over the report.
Or, worst case, the response is "Too bad, we're no longer working on this product" or "Too bad, we're not going to change it." That's the bug-response equivalent of "Go away."
The result was that I would know exactly whenever an exception occurred and I could contact the user to get more info. Oftentimes, they would be pleasantly surprised (and perhaps a little creeped out) that I knew what was going with my system. I even had foresight to snapshot form submissions etc, so I could re-submit on their behalf, saving them the hassle.
It seems trivial in this age of web services to hook up a reporting mechanism into your apps. As long as there is someone there at the other end to deal with it.
That's dangerously shaky legal ground. Wouldn't recommend it unless you're talking internal projects. Monitoring, for the US at least, has to be explicitly declared in most locales (No hiding it in an EULA).
I have to wonder how much this will all cost, and if there are any systems to group similar bug reports or make the process more efficient for people trying to resolve the bugs.
However, because I live in London and am 38 years old I don't have enough years left in me to be able to report all the problems.
I'm sure many of you have gotten those bug reports that say, "When I try to do so-and-such I get an error message." What's the error message? "Don't remember. Didn't write it down."
Both features allow the user to include a screenshot, if it helps shed light on the problem, and also include the document (an Excel workbook in this case) they were working with. For feedback the user can explain their question, and for AERs they can tell us what they were doing when the problem occurred. AERs also automatically include a stack trace and information about the environment - version numbers, threads, other add-ins, and so on.
On the server side for AERs we calculate a checksum based on the application version and stack trace so we can merge identical bug reports into a single bug report. If a user experiences a crash for a fixed issue we notify them a fix is available, or if the fix isn't publically released yet which release the fix is scheduled to appear in.
It was a lot of work initially, especially considering we're a small company, but the effort is definitely worth it. We've been able to zero-in on awkward bugs and fix them very quickly. And we've gathered great feedback from users with good ideas for improvements and new features. This only improves the quality of the application and shows customers we care about the product.
Agree with the sentiment of this post.
Dream that we could formulate software pesticides.
There's really no shortage of bug-reporting frameworks in which to wrap an application, and no excuse (on new apps, at least) to not provide a built-in reporting mechanism.
Keeping this queue may be considered "heavy", but each of our customers are siloed with their own copy of the application and generally have only around 100 users. It works for us, and we get rich bug reports without the user having to do anything. Our customers love the fact we can correct these occasional issues without them having to do anything special.
CTRL + ALT + ( Frowny-Face
CTRL + ALT + ) Smiley-Face
That is, of course, if they aren't already taken. I don't have a way to test Office 2013 Previews atm.
The irony of requesting this here instead of at the Office site is not lost on me. :) <- CTRL+ALT+)
But you have to be careful, esp in regulated industries (Legal or Medical). You can't be logging or including ANY data that might directly or indirectly ID a person.
And screenshots are notorious for this. The user might not even notice that SOC SEC number shown in the status bar of the app when they grab that screenshot for support, but a lawyer might, and it's the perfect vector for a lawsuit.
This has worked well for at least fifteen years, and has allowed me to be responsive, in contrast to other applications that the user encounters daily.
The user can run the utility and it will automatically take screen shots and note all the mouse clicks, etc. Try it NOW!
I rarely comment on blogs, but in the spirit of "Sending a Smile" I shall break that habit to say thank you for a couple of very interesting articles.
I definitely agree with the sentiments of both.
Some had been fixed, so I could sort-of* understand deleting them in a clean-up. But others had been closed as "won't fix", often due to time constraints on an upcoming release and with a promise to re-visit them at a later date. Not only have we lost the bug reports, we've lost the workarounds as well!
* Sort-of because some of the "fixes" were "upgrade to the next version", and it would have been nice to keep the workarounds for anyone stuck on the affected version.
In the case of connect, a "Won't Fix" with a promise to revisit is less something that might be fixed in the future, and more something that will never be brought up again.
Unfortunately, they also tend to be removed if they're re-posted as still occurring in the new version of the software.
I have enjoyed these blog posts. The main thing is that we talk about the problems and not just ignore them. I have already learned one valuable thing from the discussions in this post, which brings me to...
Thank you so much for mentioning the Problem Steps Recorder. I had no idea that this existed in Windows 7. Working in support, I have dealt with some Issue Reporting Challenged Individuals and this will come in very handy!
This is a couple years old problem, and we are having a really hard time to make them understand our exact problem. -.-
If our gifts are treated in this way - we might not be very eager to gift in future :/
Thanks for the reply. Looks like everything (well, almost everything in Outlook) is broken (for a nation), everyone was upset, they kept reporting the same bug for the last 5 years at least - and nobody (at MS) took care of it.
I know, Microsoft produces really fantastic products and services. I am a MSP, I am using MS products for about a decade, they have an active community, have a cool bug reporting system and according to you - "google and Microsoft were universally pleasant when I reported the bugs and seemed sincerely interested in helping. " .
Except this one, maybe insignificant to MS, but unpleasant particular bug. A whole nation is suffering from it. And truly, nobody cares. The previous ones who reported gave up, the present people who were reporting were ignored. The ones who are supposed to response, didn't bother responding.
What should we do in such case, Scott?
Thank you for your time to read this awfully long comment coming from a frustrated sufferer.
your webpage? My blog site is in the very same niche as yours and my users would really benefit from a lot of
the information you present here. Please let me know if this okay with you.
Comments are closed.
I concur with all you outlined, and find it coincidental that you posted this the same day as Darren Murph's article on Engadget about companies' ongoing relationships with customers being important. Bug reporting is another form of customer service.