Good bug reports come in four parts

19 April 2013

Peter Hilton

by Peter Hilton

This article explains which information you need for a complete bug report, because good bug reports come in four parts and incomplete bug reports can waste a lot of time on communication overhead.

If you’re testing a software application and writing bug reports for a development team, then you should include four separate pieces of information for each bug:

  1. what went wrong - the observed behaviour

  2. what you did - reproduction steps

  3. what you expected to happen - the expected behaviour

  4. the work-around, if there is one.

If you have all of these, the bug is more likely to get fixed, and at lower cost. Unless you have developers who work for free (in which case we need to talk), this is a big deal.

Observed behaviour

Writing down what you saw is the first step: something bad happened, so you describe it. The trick here is to be specific: include the actual error message, attach a screenshot that shows a user-interface issue for copy the exact incorrect calculation result.

These example bug reports are too vague:

List of beers is broken.
Emelisse beer has wrong price.
User reputation doesn’t go up.
Bug in beer description.

These are better, because they include specific details:

List of beers is shown in random order.
The price for Emelisse Imperial Russian Stout is shown as `NaN'.
User reputation isn’t recalculated when marking a beer as `tried'.
Encoding errors in the Provenier Dubbellam description.

Including enough detail in the observation is important for two reasons:

  1. it’s all the developer has to go on if he cannot reproduce the issue

  2. the developer only knows he’s reproduced the same issue if he can reproduce exactly the same result.

One more thing: please don’t paste screen shots inside documents in a document in a word-processor - save an image file instead. If you must use Windows, then either get a good screen capture tool or paste into MS Paint (press Win-R, enter mspaint, Control-V) and save from there.

Expected behaviour

As well as describing what happened, it’s helpful if you can also specify what you expected to happen instead so that the developer knows what a fix is expected to do.

Observed: beers are listed in alphabetical order.

With just this observation, a likely response is `so what’s your point?'. Sometimes it takes an alternative to make it clear that the current behaviour is wrong.

The expected behaviour may be problematic:

Observed: beers are listed in alphabetical order.
Expected: beers are sorted by popularity.

This makes it clear what the fix is, but would be a problem if the application doesn’t capture popularity. Describing the expected behaviour helps identify these cases when the actual behaviour is not a bug, because the expected behaviour is not possible.

Reproduction steps

When investigating a bug report, the first thing a developer has to do is reproduce the issue, so they can gather additional details about what is going on, and be able to confirm that the resulting fix actually resolves the issue.

As with the description of the observed behaviour, too little detail is a problem. In fact, you can’t add too much detail here, because of the huge difference in how long it takes to follow idiot-proof instructions and how long it takes to work it out for yourself.

This level of detail is described in, er… detail, in How to write user-interface instructions.

For example, this is going to save time:

Reproduction steps:

  1. Log in as admin.

  2. In the navigation menu, Administration » Delete beer.

  3. On the Delete beer page, open the Beer pick-list.

Observed: list of beers is in random order.
Expected: beers are sorted alphabetically.

Note that we add the expected and observed behaviour immediately after the step where they apply.

There is an art to writing reproduction steps: making the list as short as possible. Perhaps it doesn’t matter which user you log in as or whether you go and get a coffee (or beer) after the second step. Additional unnecessary steps can mislead the developer who investigates. In practice, a developer will often try to simplify the steps after having reproduced the issue, to avoid this.

These reproduction steps are essentially the same as a test-scenario, so you can use them to test whether the bug is fixed, manually or in an automated test.

Work-around

If there’s a work-around then fixing the bug may be less urgent.

Observed: `discontinue beer' button doesn’t work.

Not being able to use the `discontinue beer' function to remove beer from the catalogue when it’s no longer in stock could result in a serious number of disappointed customers.

Work-around: use the web service interface to set beer status to `discontinued'

On the other hand, if it’s just as convenient for the catalogue maintainer to do the same thing another way, then a major issue just became a minor one, and you can spend your development budget on something more satisfying.

The work-around is of course optional: you can only add one if you know one. Even if you don’t, the development team might.

Bug title

Since you are capturing the bug report in a bug tracker (right?), you end up with at least one more thing: a title or summary, that is used for a list of bugs:

List of beers should be sorted alphabetically.
The price for Emelisse Imperial Russian Stout is shown as `NaN'.
The user’s reputation should go up if they drink beer.
Encoding errors in the Provenier Dubbellam description.

The inconsistency in this list is a result of randomly summarising either the expected or observed behaviour and using that as the title. It’s easier if you just use one or the other.

Consistency is easier if you use the observed behaviour as the bug title:

List of beers is shown in random order.
The price for Emelisse Imperial Russian Stout is shown as `NaN'.
User reputation isn’t recalculated when marking a beer as `tried'.
Encoding errors in the Provenier Dubbellam description.

Using the expected behaviour instead doesn’t work because the person reporting the bug might not know what the correct behaviour is. In the first example, it may be obvious that beers in random order is wrong (beer order matters!), but not what the correct order is.

Incomplete bug reports

An incomplete bug report is not necessarily a problem, because sometimes the expected behaviour is simply that the observed behaviour not happen.

Adding a beer to the shopping basket causes a `fatal server error'

However, for every bug report that didn’t really need all four parts, there are always more where they are missing because someone thought something was obvious when it wasn’t.

The following scenarios slow everyone down:

  1. The observed behaviour is vague or missing and the developer has to bounce the report and tell you to try again. If you’re lucky, they’ll choose their words carefully and avoid accidentally pissing you off in the process.

  2. The reproduction steps are missing, and the developer has to choose between spending time trying to discover how to reproduce the bug instead of something more useful, or bouncing the bug and having to wait for more information, causing a delay. Or both.

  3. The expected behaviour is not described and the developer has to avoid either implementing the wrong fix or spending extra time trying to discover what you want.

In the long run, it takes less time to include all of the information than to think about whether it’s all needed and then fix the problem in the case where missing information was. Not all laziness is constructive.

Conclusion

Better bug reports save time, and time is a lot of money, because software maintenance is expensive. Fortunately, it only takes a little discipline and practice to write better bug reports.