Test, Test, and Retest 801

Step 14: Test, Test, and Retest 801

A usability test can be conducted in a variety of places ranging from a formal usabil- ity laboratory to the participant’s actual work location. Tests that are undertaken in the user’s own environment are referred to as field studies. Evaluations undertaken some- where else are called controlled studies. Controlled studies may be formal, being per- formed in a specially-designed laboratory, or informal, occurring in a conference room or office.

Usability laboratory . A laboratory is a fixed specially constructed facility designed

and fitted for conducting formal, controlled tests. The advantages of a laboratory from a data-collection perspective are

It can be designed with one-way mirrors for observers and note takers. More people are able to participate in, or simply watch, the test. The impor- tance of usability testing can actually be demonstrated to other visiting per- sonnel of the organization whose system is being tested.

It is easily equipped with recording devices such as video cameras and video monitors or projectors.

The physical environment — lighting, temperature, and so on — can be more precisely controlled. Actually work environmental conditions can be simulated.

Distractions and disturbances can be minimized. If the test is being held in a usability laboratory, the test facility should resem-

ble the location where the system will be used. The biggest disadvantage of a lab- oratory is its construction cost. Usability labs may be available for rent, however. Its fixed location may not be convenient for all participants, though.

Office or conference room . Informal controlled tests can be held in an office, con-

ference room, or a hotel meeting room. To collect test data, laboratories are now available in portable units that can be easily shipped and set up at remote facili- ties. Portable labs now possess most of the components incorporated within spe- cially constructed labs. The benefits of a portable lab, according to Koyani (2006), are as follows:

Scheduling tests is easier. Meeting rooms are easier to locate and reserve.

More flexibility exists in establishing a test location. The test can be held at locations close to the user’s workplace.

Renting a formal laboratory usually costs more than to rent of portable lab.

An informal setting is satisfactory for collecting qualitative data.

Some usability specialists consider a fixed laboratory an artificial environ- ment and prefer a more natural environment.

Field studies . An evaluation may also be conducted at the participant’s workplace.

An advantage is that information about the environment, including distractions and interruptions can be collected. Disadvantages include being difficult to arrange and set up; test disruptions because of the participant being called away to address other issues or problems; distractions for adjacent workers, manage- ment objections; and the workplace being unsuitable for visitors.

802 Part 2: The User Interface Design Process

Unmoderated remote testing . Recently there has been an interest in another form of testing called unmoderated remote testing. This method asks participants to complete prescribed tasks remotely, either at home or in another non-laboratory site, while performance measures are collected. An evaluator, or observer, does not physically monitor task performance. Schulte-Mecklenbeck and Huber (2003) compared Web-based opened-ended tasks performed in a lab versus at home by different groups of users. Lab users took twice as much time and used twice as many clicks as the home users. The authors postulated that lab users may have felt more pressure to perform well and/or perceived the lab evaluation as “more important.”

Tullis et al. (2002) performed a similar study using “closed-ended” tasks (the results being either right or wrong). No differences between the success rates and completion times were found between the two groups. They concluded that remote testing worked well and had benefits that complimented the lab study. They recommended a combination of both remote and lab testing to cover the range of design issues. Remote testing deserves additional study.