Test, Test, and Retest 783

Step 14: Test, Test, and Retest 783

without procedural assistance, browsing as is felt necessary. If, however, the eval- uators are not familiar with the system’s content and purpose, they may be pro- vided with scenarios listing the steps a user would take to perform a sample set of realistic user tasks.

During the session, the evaluators will compare their findings with the list of usability principles. Detected problems should be related to the specific heuristics they violate. Multiple heuristics can be linked to any one identified problem. Other relevant problems or issues can also be noted. Two or more passes should

be made through the system.

Evaluator comments can be recorded either by the evaluator or by an observer. Evaluator-written comments require additional effort by the evaluator during the review process and can break the continuity of the evaluation process. Having an observer record the evaluator’s verbal comments adds to the over- head of the session, but reduces the evaluator’s workload and allows greater focusing on the review process. If the same observer is used for all evaluation ses- sions, session results should be available faster because the observer only needs to understand and organize one set of personal notes, not multiple reports or reviews written by all session participants. The observer may answer questions and provide hints to the evaluator, as necessary. Evaluators not familiar with the system’s content will occasionally need advice. Also, precious time will not be wasted by the evaluator’s struggling with the interface’s mechanics. Observer comments should be restricted to situations where the evaluator is clearly con- fused or in trouble.

MAXIM Not even the most brilliantly conceived and ingenious computer sys- tem can do all that it was designed to do — or even a small part of what it was designed to do — unless the brilliance of its operation and purpose is matched by the cunning simplicity of its user interface (Hicks and Essinger, 1991).

Finally, to minimize evaluator fatigue, restrict the length of a session to about 2 hours. For large or complicated interfaces that require more time to evaluate, it is better to split the session into several sessions, each concentrating on a part of the interface.

After the session . When all evaluator and/or observer notes have been compiled, hold a debriefing session, no more than 2 hours in length. Include all observers and design team members. Have each evaluator present the problems detected and the heuristic each violated. Assemble a composite list of usability problems (if one has not yet been compiled). Solicit and discuss design suggestions for improving the problematic aspects of the system.

After the debriefing session, form the composite list of violations into a ratings form. Request the evaluators to assign severity ratings to each violation on a scale ranging from “usability catastrophe” to “not a usability problem,” as shown in Table 14.2 It is difficult to obtain these estimates during the evaluation process, where the focus is on finding problems. Then, analyze the results and establish a program to correct the necessary violations and deficiencies. The ratings will

784 Part 2: The User Interface Design Process