questions to the subject, if they were being too quiet or seemed confused, so the administrator
could pinpoint in the notes what was the cause of the confusion.
When all the runs ended, the administrator interviewed the subject. The subject was generally
asked to rate his or her own performance and answer a few questions about the experience. The
subject was then asked for any other comments he or she may have.
We analyzed this data to determine performance measures. Performance measures are implicit
measures of the quality of the user interaction provided to users. Under ordinary circumstances,
users who were given usable interfaces could be expected to perform better at their tasks than
those who were given poor interfaces.
There were a few ways we inferred situation awareness from the test data. First, we counted the
number of collisions that occurred with the environment. A user with good surroundings
awareness should hit fewer obstacles than an operator with poor surroundings awareness. We
also analyzed the percentage of the arena covered or the time to complete the task, depending
on the study. A person with good location awareness should not unknowingly backtrack over
places they have already been. Therefore, they should be able to cover more area in the same
amount of time than an operator with poor SA, who might continually traverse the same area
over and over. Likewise, if a person has good SA, it stands to reason that they should complete
the task at hand quicker than someone with poor SA. An operator with poor SA might be
confused and pause to try to figure out what is perplexing them.
Other implicit measures were also taken. These measurements often came from the subjects
think-aloud comments. These comments can give valuable insight into whether or not a subject
is confused, or if they recognize a landmark. For example, users would often freely admit to a
loss of location awareness by saying “I am totally lost.” or “I don’t know if I’ve been here
before” (speaking as a “virtual extension” of the robot).
Some of our studies also used explicit techniques to determine SA. At the end of their task time,
the participant was asked to return to a previously found target. Often this target was a victim in
the arena or the location they started from. The user had no prior knowledge that they would be
17
Comentarios a estos manuales