Test Results by Build

Test Results by Build

Failed

In Progress

Passed

18 16 14 12 10

BlogEngine.NET Dev 20100524.1

BlogEngine.NET

BlogEngine.NET

Dev 20100525.1

Dev 20100526.1

Reporting with Microsoft Excel 267

Table 9-3: Fields and Placement to Create Excel Reports

Figure Report Filter

Legend Field Axis Field

Value

9-19 Test Plan Name

Date, Test Run Title Result Count Trend 9-20 N/A

Outcome

Point Count Trend 9-21 Work Item Linked. N/A

Outcome

Date

Point Count Trend Work Item Type,

Work Item Linked.

Title, Test Case.Title

Work Item Linked. Iteration Path

9-22 Work Item Linked. Outcome

Point Count Trend Work Item Type,

Work Item Linked.

Title, Test Case.Title

Work Item Linked. Iteration Path

268 Chapter 9: Reporting and Metrics know information about the number of Test Cases that have not been

executed (that is, they have no results) the Point Count Trend provides that information.

Figures 9-21 and 9-22 are actually closely related. The only thing that sep- arates them is the addition of the Outcome dimension on Figure 9-22. And Figure 9-22 is simply a refinement of already existing data but shows so much more information. Figure 9-23 is another way to present the same information shown in Figure 9-22 but in a far simpler manner. Figure 9-23 uses the Test Suite Hierarchy dimension to represent the information; you can see that hierarchy in the label (NWC\BlogEngine.NET\Test Iteration 1). It is easy to remove this label after the fact. This report required only three fields to create, whereas the reports shown in Figures 9-21 and 9-22 required six fields.

Figure 9-24 is a simple way to determine which tester executed how many

Metrics 269

over time.” If you take measurements and do not compare them to previous measurements, they are useless. If you take measurements of all processes, the value of the measurements is useless because you have no clearly defined purpose for gathering those measurements. There are many great quotes from people of all walks of life that apply. Some of those that apply are listed here.

“Where you cannot measure your knowledge is meagre and unsat- isfactory.”

—Lord Kelvin (Sir William Thomson) “I believe in evidence. I believe in observation, measurement, and

reasoning, confirmed by independent observers. I’ll believe any- thing, no matter how wild and ridiculous, if there is evidence for it. The wilder and more ridiculous something is, however, the firmer

270 Chapter 9: Reporting and Metrics When trying to improve a company’s process, the company must start

with a desire to make a change. If a company does not want to change, gath- ering metrics is not useful and a further waste of time on top of the time already wasted. For those organizations that do want to make a change, that is the first step. It is exactly like any 12-step program: You can’t get help until you admit you have a problem. But when you do, and are willing to make a change, the possibilities are almost endless. This is not a simple decision. Not because companies don’t want to make a change but because there is a cost associated with gathering metrics, reviewing them, and then improving the process—it is not free. But the reality is that if an organization spends the time and money to do it correctly, it can get a ROI by improving efficiency, increas- ing quality, and increasing customer satisfaction.

In general two areas exist in which people want to make improvements:

Metrics 271

gathering and work to apply it on a project-by-project basis. (Or augment an organizational plan if these ideas are relevant to you.)

What to Measure The first question to ask is, “What should I measure?” To answer that you need to ask another question, “What am I having problems with?” This may

be a bit more difficult to answer. What are you having problems with right now, or what areas do you want to improve efficiency and quality in? You’ve probably identified some items you want to improve—even if you don’t have hard data on what the specific problem is, you “know” a problem exists. After you identify what you want to change, cut the list down to just one or two items. Trying to measure everything at one time can throw up too many vari- ables for you to adequately determine root cause and also cause you to not

272 Chapter 9: Reporting and Metrics

a specific problem but are discussed toward the end of this section. Now look at each of these items to determine how to measure them, the cause, and the solution.

How to Capture a Metric When you first set out to gather a measurement, you need to do a little bit of upfront work. Worry about how to fix it after you have proven it. This work starts with defining the problem that you want to prove. (Assuming that you don't already have a baseline, you need to create one first.) Maybe the prob- lem is a high number of defects with a certain feature in the application.

After you define the problem, determine the measurement that you want to use. In this example, you may determine that comparing the bug count in

Metrics 273

You may need to determine the accuracy of this perception. Whatever you do, realize that the development team must be your ally, and

you cannot gather information that does not somehow benefit the team. At the end of the day, people do what makes things better for them. That may sound cynical, but if you do not approach the process like this, you can have an uphill battle.

First-Time Defect Rate The first-time defect rate is a popular metric that people like to look at, which tells how many defects are filed against requirements after the developer says, “I’m done.” Usually first-time defect rates are quite high. The downside

274 Chapter 9: Reporting and Metrics have a requirement in the Active state and a series of tasks also in the Active

state. As the developers finish each task, they set the state to Closed. (Even if you use the CMMI template, you can simply skip the Resolved state for Task work item types.) After all the tasks related to the requirement are closed, the developers set the requirement to Resolved. This is the indicator to the testers that they can test the requirement. During testing, if they find bugs, they will do a couple of things. First, they file a bug that by default is associated with the Test Case. The next step the testers perform is more difficult. They assign the bug to the developers. Why is this more difficult? Because, if they simply assign bugs to the developers as they file them, you can’t determine a first time defect versus a reoccurring defect. With a constant flow back and forth, it is difficult to determine.

This is one of the lessons you should take away about metrics: You must

Metrics 275

It won’t be perfect because customers change their mind all the time, but at least you can tell when it was the customer changing its mind versus the developers making a mistake.

WHAT TO TEST Chapter 3, "Planning Your Testing," started this discussion. When time is short you must ensure that you test 100% of the normal path activities. That is, you want to test the parts of the system that will be most used by the cus- tomer. Test as many of the alternative and exception paths as you can, (Obvi- ously, if you can, test all of them 100%, but that is rarely the case.) If you can't wait for testers to create every Test Case, make sure they create the impor- tant ones first. Tests such as boundary conditions and other nuanced tech-

276 Chapter 9: Reporting and Metrics Related Metrics

This is a list of metrics that can have some type of impact on the first-time defect rate metric. Although this metric appears be fairly easy to capture and reduce, other data is pertinent to this metric:

• Requirement complexity —The more complex the requirement, the greater likelihood of first-time defects. Try breaking the requirements down into smaller requirements.

• Number of external systems involved —Some things are beyond your control. At a certain point, you need to accept it, but you should account for it if possible.

• Defects versus change requests —Customers can and do change their mind. This leads to what looks like defects but in reality are not. Hav-

Metrics 277

the Active state. When a tester goes to verify a bug, if the bug is not resolved, the tester must set the bug back to Active. Typically, every time a bug is reac- tivated, a requirement must also be reactivated, but this largely depends on your teams’ strategy. (The built-in reports can give you information on either work item type, but the Bug work item type is more important.) If testing occurs on unresolved functionality, this does not apply. For this reason, what appears on the Reactivations report may give you different information depending on your process.

Lowering the Reactivation Rate Fortunately, the tools available in MTM and Visual Studio should reduce reactivations. Where the tools won’t help as much is when customers file bugs. This is where process is important and somewhat independent of any

278 Chapter 9: Reporting and Metrics