Resolving Bugs

5 Resolving Bugs

108 Chapter 5: Re solving Bugs it may be treated differently and require different lifecycles depending on the

submission process. Figure 5-1 shows the bug states and transitions for both MSF for Agile and CMMI.

MSF for Agile v5.0 MSF for CMMI v5.0

[New]

[New]

Build Failure

Active

Proposed

[Fixed] Deferred Duplicate as Designed

[Investigation] Cannot Reproduce

[Not Fixed]

[Approved] Investigate

Complete Obsolete

Test Failed

Resolved

[Regression]

Active

A Bug’s Life 109

110 Chapter 5: Re solving Bugs The “Bug Work Item Type Differences” section discusses differences

between these bugs. For now the major differences are that more data is gath- ered in the CMMI bug. Look at a couple of scenarios and some potential processes for handling them.

Customer Reported Bug When a customer reports a bug, it should first go to the testing team for val- idation to ensure it is reproducible and is actually a bug (as opposed to a change request). In an Agile project it may be put in the backlog for the next iteration. In a CMMI project it may go to a change board before going to the testers; it depends on your processes.

When the test team gets the bug, as part of the verification process, a Test Case should be created. In addition the Test Case should then be linked to the

Bug Difference s and Modifications 111

Reactivations An important point is in dealing with reactivations of both bugs and require- ments. Bugs can either be new bugs or “fixed” bugs that were not actually fixed. The following process applies in each situation: For the testers to begin testing a feature, the feature must be complete (or should be, in most cases). This is indicated by the developer setting the requirement to Resolved. (This is true in both process templates.) When testers find a new bug, they should not only create the bug but also set the requirement back to Active. This causes the story to appear in the Reactivations report (discussed in Chapter 9).

For an existing bug that has been resolved, the tester should set the bug back to Active if the bug is not actually fixed. It is up to the team to deter- mine whether the tester should also set the requirement that the bug is asso- ciated with back to Active. The act of setting the bug back to Active can cause

112 Chapter 5: Re solving Bugs

Table 5-1: Fields in MSF for CMMI Bug WIT That Are Not in MSF for Agile Bug WIT

Field

Need

How Found What was going on when this bug was found? This is a free form field.

Original Estimate How long is it going to take to fix this bug? Proposed Fix

What is the developer proposing to fix this bug? Root Cause

How did this bug get in the code? The default values are Unknown, Coding Error, Communication Error, Design Error, and Specification Error.

Symptom How was this bug manifesting itself? (The screen didn’t complete a redraw, the data was incorrect, and so on.) This is a free form field.

Bug Difference s and Modifications 113

filed does not mean that a) anyone is actively working on it, b) that it is a ver- ified bug, or c) that it is a bug. Because everything starts as active, this can tend to skew the metrics by reporting a higher than accurate number of bugs.

BUG WORKFLOW The discussion of workflow is based on the built-in MSF process templates and not by any other influences. For that reason you should also determine if the process flow makes sense for your team. For instance, when a bug has been verified (assume this is a more formal CMMI type of process), moving

a bug from Proposed to Active doesn't make a lot of sense if you want to manage work in progress. After a bug has been investigated (and it is back in the Proposed, Investigation Complete state), when it is approved for work

114 Chapter 5: Re solving Bugs members that an issue exists, and you can create an issue work item to further

detail why something is blocked. Adding this field is at your discretion and depends on whether you have a problem which you need to solve.

The Found in Environment field is not necessary in its default form. A free form field provides no benefit for quantification purposes (and doesn’t pro- vide benefit for fixing the bug). The main reason for this is that with the Sys- tem Info data gathered (see this in the “The Generated Bug” section), all the information you need is collected for you. Where this provides some benefit is when you can quantify where in the process the bug was found (Develop- ment, Testing, denoting internal team testing, User Acceptance Testing, or Production). This helps gauge the successfulness of the testing process. To do this, update the work item with an Allowed Values rule and put in these val- ues (refer to Chapter 9).

Bug Difference s and Modifications 115

The Proposed Fix field is valuable if there is a period of time between the bug being found and the developer starting on the fix, or if there needs to be

a design review process for a complex bug. There are no hard-and-fast rules for this one. It is valuable, but it may not be valuable to you in your current situation.

The Root Cause field is valuable no matter what methodology you use. Without pinpointing how a bug was introduced, you can never make changes to prevent it from occurring again. The root cause default values are fairly broad and encompass almost everything you will encounter. However, if you can add additional values that can help narrow the root cause, that will

be beneficial in devising a plan to fix it. This field should be retrofitted to the Agile Bug work item type. In addition, make the list of Allowed Values a Global List so that it can easily be updated for every Team Project.

116 Chapter 5: Re solving Bugs in the Agile Bug work item is again because of the proximity of users to the

development team; they can just call someone over and show them. This, however, does not take into account the maintenance period after the appli- cation is released; the development team is not going to be there forever, and there will be some communication done through the use of work items in the future.

Triaging bugs is always necessary; you need to know the priority of the bug. The big difference here is that the product owner sets the priority with the team during a planning meeting. For the CMMI Bug work item, it does- n’t make sense to have this field. The reason is that the Priority field is required. Theoretically, you can look at the Triage field first, and if it does not say “Triaged,” you know the priority is invalid but that doesn’t make much sense. So there are two options you can take: Make the priority field not

The Generated Bug 117

118 Chapter 5: Re solving Bugs This control provides the context of the bug for the developer. Here is the

set of steps the testers took before they encountered the bug. The context is provided by showing the order of steps that the testers took and which steps passed and which steps failed. You also see the time index into the video for every step. Clicking this starts the video at that location. With the video, the developer can immediately see the problem. Directly below this is the screen- shot and comments captured by the tester—again the information is imme- diately available and nothing is lost in translation.

Below this are the test configurations and the applications that were not recorded at the time of the test. (If the tester specifically identified an appli- cation, it will be called out here.) Finally, the list of data collectors that were running during the test is displayed and their output. The key file here is the IntelliTrace log because this enables the developer to “replay” what was hap-

The Generated Bug 119

120 Chapter 5: Re solving Bugs

How a Developer Uses IntelliTrace

By way of an introduction, IntelliTrace captures historical data related to the code executed by the test. In this way, developers can move back and forth through the test run examining variable data and conditions at a point in time, and they don’t need access to the actual code. On the other hand, the developer can also attach the IntelliTrace log to the code and replay the test- ing session after the fact.

When developers are assigned the bug, one of the first things they do is look at the context of the bug and try to reproduce it. For this, they can just walk through the list of steps in the Steps to Reproduce section of the filed bug. When they have successfully done this, the developer will most likely jump into the IntelliTrace log (Figure 5-7).

How a Developer Use s IntelliTrace 121

thrown during the test. The Test Data shows all information related to the actual tests that were run, including the Test Case and the test steps taken during the Test Case and whether they passed or failed.

Before you step into finding and fixing the bug, it is worth noting that you can find two types of bugs during a test: an exception bug (the application does something it cannot actually do, such as creating a file in a location where the application does not have permission to create that file) and a logic bug. Exception bugs will be listed in the Exception data, and debugging these problems is extremely simple; the developers just double-click the exception and are taken to the line of code that it occurred on. Logic bugs are a differ- ent problem because developers can’t point to a single spot in the code and say this is where everything went wrong; they need to track it down.

122 Chapter 5: Re solving Bugs

Fixing the Bug 123

124 Chapter 5: Re solving Bugs Three items on this report are of interest: Associated Changesets, Associ-

ated Work Items, and Impacted Tests. Associated Changesets

If you have used TFS but have never used automated builds, you are miss- ing out on a big feature. In many organizations, a manifest must be created showing which files and which versions of files are included in a release. This is painstaking (and painful) and time consuming to do correctly. Automated builds solve that problem with no work. All you need are automated builds! Changesets are associated with builds through the following mechanism: As part of a successful build (this is not true if the build fails), Team Build creates

a label. (This label is per build definition.) When performing a new build, TFS looks at all the changesets added after the previous build (which is identified

Setting the Build Quality 125

NOT ALL WORK ITEMS ARE ASSOCIATED WITH BUILDS

A word of caution: Not all the work items associated with a build are listed. Consider the following: A requirement has a task. A developer writes code and checks the code in and associates it with the task. The build runs, and the report show that only the task is associated with the build. Why? There is no changeset related to the requirement work item.

So what can you do about this? When you do a code check-in, make sure to associate it with both the task and the requirement. This can also be done as a post check-in step but is easier to do at the time of the check in. You need to be aware of this out-of-the-box limitation.

126 Chapter 5: Re solving Bugs screen. In addition, there were no integrated functional testing tools to take

advantage of this value. With 2010, this might change; if it does you should use this setting to inform and communicate to testers whether a build is suit- able for a variety of things. In addition, testers can update this field to further communicate with the release team. You must decide what you are going to use the build quality for. For example, the first quality available to you is Ini- tial Test Passed. Well, what initial test? The Unit Tests that ran as part of the build or when the testers got the build and ran their regression tests? It can mean different things to different people, so you must define these before using them so that everyone has a common understanding.

Table 5-2 describes the default options available to you. Table 5-2: Build Quality Usage and Description

A ssigning a New Build 127

To customize these build qualities, go to the Build Explorer page in Team Explorer, and select Manage Build Quality. After the build is complete, the testing team should redeploy the code to wherever it is the testers are testing from.

DEPLOYING TEST CODE Test code can be deployed automatically as part of the build. This is much easier to manage with a virtual testing infrastructure because you can have many more different versions of the build available to testers. So whose re- sponsibility is it? The testers are responsible for deploying and testing the code, but it’s nice when the developers help with an automated deploy script or an MSI package or a set of instructions.

128 Chapter 5: Re solving Bugs In MTM, you can assign a new build in two ways: from the Plan tab, Prop-

erties page or from the Track tab, Assign Build page (Figure 5-11). (They are the same screen, so it’s just a matter of preference.) When you assign a new build, MTM lists the latest build for the given build definition.

Verifying That the Bug Is Fixed 129

The impacted tests between two different builds are shown; the builds do not need to be consecutive, so you can see all the tests that have been impacted over a period of time. Each impacted test is shown once for each Test Suite that the test is a part of. (This is why the same test shows up twice in Figure 5-12.) When you select one or more tests, you have the option to reactivate the Test Cases. You cannot run these tests directly from this dialog. To return to this list, select Track tab, Recommended Tests page. On this page is the option to show the related work items. Selecting this shows the screen in Figure 5-13.

130 Chapter 5: Re solving Bugs

Figure 5-14: Verify Bugs screen

Dealing with Impacted Te sts 131

an automated Test Case. As soon as you create an action recording, the play- back is not available to you, so determine when this makes the most sense. However, when using the playback feature, the benefit is immediately appar- ent. Now you simply play back the existing recording until you get to the point in which the bug occurred and simply execute that part of the test man- ually. This makes bug verification simple and virtually painless. When you complete the test run of the bug and click Save and Close, a bug resolution dialog displays (as shown in Figure 5-15).

132 Chapter 5: Re solving Bugs defects. To do this, they have only the information provided by the develop-

ers and their experience in running these Test Cases before to determine which Test Cases are best to execute.

TEST IMPACT ANALYSIS WON'T FIND EVERYTHING TIA does not know about changes made to external components or to things such as data. For this reason it is still the responsibility of the testers to make sure that the right tests are executed. Use TIA as a guide but not as the only guide.

For a small number of tests, the easiest option is to reset all the impacted tests to Active. Use the Test Plan Summary page to determine how many are

Summar y 133

contained in the filed bug, and you know the value of the information col- lected for you automatically by Microsoft Test Manager and Test Runner. Knowing what information is gathered and how developers can use that information is critical to improving the communication between developers and testers. When the developers have completed their bug fixes, testers can

be notified through a number of different mechanisms. After the testers have the resolved bug back in their hands, they not only can verify that the bugs were fixed, but also rerun high-value tests to increase their confidence that no regression bugs were created by the bug fixes.

You have been dealing with manual Test Cases and a completely manual process for testing and verifying code, but starting in Chapter 6, “Automat- ing Test Cases,” you learn how to automate many aspects of the testing process, execute those tests, and verify the results.

This page intentionally left blank