The vast majority of workplaces and educational settings have been forced to adopt either a virtual or a hybrid (jointly virtual and in-person) setting. Collaborative work in particular has been inhibited by the necessity of social distancing, as multi-faceted teams of facilitators and participants are now forced to conduct their work while geographically separated. For example, UX Researchers, who analyze consumer behavior and assess usability, are often required to observe in-person interactions involving a test participant, a test facilitator, and a computer that displays the website, system, or other remote platform being assessed. Without the possibility of such face-to-face facilitation, UX researchers have turned to virtual environments to continue their work, such as web-based conferencing and screen-sharing conditions, or immersive virtual reality (VR)-based conditions.
The Clemson University Departments of Industrial Engineering and Civil Engineering have conducted research that investigates the efficacy of collaborative virtual reality systems for moderated remote usability testing. Led by Dr. Kapil Chalil Madathil and Dr. Joel S. Greenstein, this research was aimed at determining the potentiality of implementing virtual, world-based approaches in replacement of traditional, in-person lab settings to adequately perform synchronous usability testing.
By comparing the experiences of test participants and facilitators throughout three moderated remote usability testing conditions–a traditional lab-based environment, a virtual video conferencing and screen-sharing environment (orchestrated through the online platform, WebEx), and a collaborative virtual-reality-based environment. Dr. Chalil Madathil and Dr. Greenstein compiled a group of 36 remote usability test participants and 12 remote usability test facilitators, the former of which was split into three sections in compliance with the three conditions being compared. The traditional lab-based environment consisted of a participant and facilitator gathered together in a laboratory setting, while a UX researcher observes interactions and the shared thought processes of both participant and facilitator. Within the WebEx Server environment, UX researchers, test facilitators, and test participants were separated into three different geographic locations; the facilitator and participant were connected through a shared web browser. While the VR-based environment also encompassed geographically separated parties, it provided a three-dimensional immersive laboratory in which participants and facilitators shared not only a web browser and a whiteboard but also text and audio chat tools for communication. These tools made it easier for the facilitator to record the participant’s concerns with the design of a given web application’s interface.
The experimental design of this research combined within-subjects observations, concerning test participants’ interactions with the web application, and between-subjects observations, concerning the test participants’ interactions with the remote usability testing environment. The tests were controlled through the provision of consent forms, pre-test questionnaires, brief preliminary training, and post-task retrospective think-aloud sessions that allowed participants to voice their concerns while interacting with the remote interface. Moreover, a heuristic evaluation was conducted to rate the severity of the usability problems identified within the tests.
Given all of these reported advantages, as well as the fact that the virtual alternatives of WebEx-based and VR-based conditions alleviate travel time and cost, improve turnaround time for user-centered iterative product development, and connect test participants, facilitators, and researchers that may be geographically separated, this research identified the condition of immersive virtual reality as a good alternative to traditional lab-based usability testing. These factors are especially crucial within the era of COVID-19, as in-person lab-based conditions are significantly less feasible. However, this research study was not without limitations; this was a very small research study, comprising 48 individuals from one university. Although the VR environment performs as well as the other two environments, the team identified a few issues within the virtual reality system that need to be further improved upon. Within the prototype interface being tested, participants experienced a delay while interacting with the interface, and facilitators were unable to monitor the mouse movements of the test participants. If further versions of this interface resolve these disadvantages, collaborative VR-based conditions could provide the most viable and accessible remote usability testing under unprecedented conditions.
The results of this research indicated that there are no significant differences between the three remote usability testing conditions concerning the time taken to complete tasks, the total number of identified usability defects, or the severity of such defects. However, participants and facilitators reported decreased workload and decreased confusion within the traditional lab-based condition, perhaps due to the fact that the WebEx condition and the collaborative virtual reality condition contained time delays and additional preparatory sessions. Higher reports of workload and participation were prescribed to the WebEx and collaborative VR conditions, most likely stemming from higher levels of interaction with the computer and relatively slow response times. Despite these disadvantages, only test participants within the traditional lab-based condition recounted feelings of pressure to avoid mistakes, possibly due to the in-person presence of the test facilitator. Additionally, the collaborative VR-based condition outscored the WebEx-based condition in terms of involvement and immersion. Because the VR-based condition provides a multisensory immersive experience and is characterized by avatars, test participants and facilitators are supplied with simultaneous visual and auditory feedback. Moreover, a collaborative VR-based condition may be the best option for security-conscious organizations, as this condition allows the facilitator and participant to collectively view only the web browser without sharing any information outside of the browser. The VR-based condition also does not require the installation of any client applications to enable screen sharing and chat functionalities, as these are built into the VR platform.
This work has been published in the Applied Ergonomics Journal.
