Top 10 Success Factors for a Bug Bash

Now that the bug bash I announced in the previous post is over, I wanted to share some experiences about organizing, conducting, and concluding a successful bug bash.  But first, let me tell you why a bug bash is so helpful for a software development project.  Theory aside, here's how my team has benefitted from our bug bash:

1. Effectiveness.  We discovered over 50 issues in 1.5 hours.  Compared to regular testing, it would take a few days to get to this number.
2. Usability Verification.  We received reassurance that the application is intuitive enough that people can use it without reading a user manual.
3. Stability Verification. We received reassurance that the application is stable; out of 50 issues, only 1 was a critical error, and people were able to complete all tasks we asked them to complete.
4. Install Verification.  We verified that the install process ran successfully on different configurations.

With these results in mind, let me now segway to the list of Best Practices we've leveraged to ensure the success of the bug bash.  

Before the Bug Bash

Best Practice 1 - Send an Advanced Invitation

People have busy schedules so make sure you give them enough notice.  In my case, I sent an invite 3 business days in advance.

Best Practice 2 - Set Specific Start and End Times

When scheduling a bug bash, we clearly identified the duration of the bug bash to remove any ambiguity around the time commitment we were asking for.

Best Practice 3 - Invite More People Than You Need

I wanted to have 8 people for the bug bash and ended up sending the invitation to 30 or 40 to get my 8.  Also, keep in mind that even you have enough people signed up, their schedules may change, and you'll be left out in the cold.  So try to book more people than you need.

Best Practice 4 - Invite People from Different Business Groups

Include multiple business groups into the Bug Bash - anybody who is willing to participate should be welcomed (unless your application is highly specialized and requires a deep understanding of a particular domain).  Different groups will bring fresh perspectives to the table, and you'll get bugs from behaviors developers or testers wouldn't think of.

Best Practice 5 - Provide Written Instructions With An Appropriate Level of Detail

Our Bug Bash was for a sales tool.  Sales people don't like reading manuals, and the tool needed to be intuitive enough for a first time user.  To keep the situation as real as possible, we provided our bug bashers with a description of a business situation in which the tool would be used and listed 8 high-level tasks our bug bashers had to accomplish.  We also included a list of known issues in this version of the tool so people wouldn't waste time logging them.

Best Practice 6 - Include A Feedback Form or Spreadsheet

To ensure that the bug bashers would submit feedback in a consistent manner and format, we provided them with a simple Excel spreadsheet with pre-defined columns.  The standard entry format made it extremely easy for our testers to consolidate the results.

During the Bug Bash

Best Practice 7 - Send Out a Reminder To Start and To Finish

Our bug bash ran from 1 pm to 2:30 pm.  At 12:58, I sent out an announcement to everybody who accepted the invitation and reminded them that the bug bash was starting.  Shortly after 2:30 pm, I sent out another announcement letting people know that the bug bash was completed.  Why should you do this?  Your bug bash is a high priority for you.  For your bug bashers, this is an add-on activity that they graciously agreed to do. Sending out the announcements helps them remember their commitment and releases them back to their regular work lives.  

Best Practice 8 - Be Available To Answer Questions

Our bug bash instructions included the team's Distribution List and a request to email us should any questions or issues arise.  We monitored the alias throughout the bug bash and were able to answer a few questions that came in. This ensured that people wouldn't get blocked from completing the tasks we asked them to complete.

After the Bug Bash

Best Practice 9 - Thank Your Bug Bashers

After the bug bash was over, we sent out a Thank You note to all the participants and  listed their names.  That's the least any team can do to express their appreciation of the effort put in by the bug bashers.

 

Best Practice 10 - Triage the Issues and Log Bugs

After our Tester consolidated the feedback of the participants into a single list, the team sat down and went through every single issue together.  We ended up dismissing some issues, raised a few questions to the client and a user experience architect as a result of others, and logged the rest as bugs into our bug tracking program.

Armed with this checklist of best practices, you too should be able to complete a successful bug bash and achieve the following goals:

1. Effectiveness (high ratio of bugs to time spent)
2. Usability Verification
3. Stability Verification
4. Install Verification