Guidelines for Testing Mobile Augmented-Reality Apps

The number of mobile augmented-reality (AR) applications and their popularity are rapidly increasing. It is only a matter of time until companies in various domains, including health, ecommerce, education, and gaming, incorporate AR features into their mobile apps.

As with any digital product, AR apps benefit from user testing before and after launch.  While conducting an AR usability study has many similarities with other mobile studies, certain factors can impact the quality of the study and of the findings. For example, some AR apps involve (sometimes intense) physical movement, while others require the participant to share a camera capture of their surroundings with the facilitator.

We recently ran a usability-testing study of mobile AR applications. The study involved 7 in-person sessions and 4 remote moderated sessions. In this article we share the lessons we learned about setting up such a study. In particular, we focus on the following main components:

Tasks: Wording and Order  

Since users are often new to AR technology, tasks should be clear and straightforward to reduce potential confusion or misunderstanding.

For example, a task in our study required users to “download the application ARLOOPA and take a photo of a few animals in the room.” This task was intended to get the participant to use the AR feature to add a few virtual animals to the user’s environment. However, the participant misunderstood the task and to the facilitator’s surprise, tried using the app to take some pictures of his pets (a cute dog and fish tank). The facilitator had to patiently redirect the participant and ask him to add some virtual animals to the room. Eventually, the user added an AR dinosaur to the scene.

You should be ready to provide more details than in a regular user test and may even have to guide participants through the initial steps of the task — for example, you may need to help them identify the AR feature within the application.

Alternate tasks that require physical activity with tasks that do not. Even participants who are physically fit may get tired after doing several physically demanding tasks in a row. Alternatively, if you are testing only physical tasks, consider offering one or more breaks to users during the session. Make sure you have water on hand, as well.

Consider categorizing the tasks based on priority for the study, need for physical activity, and space requirements, to make it easy to select the appropriate tasks for each participant.

Study Setting

Some AR apps require ample or open space to function. If you are running an in-person study, make sure that there is enough space in the testing room for the tasks to be accomplished without participants bumping into objects. If your study is remote, depending on the types of apps you’re testing, you might need to let participants know in advance that they will need to have plenty of space around them.

While working with an AR app, participants typically focus on the task and the AR objects and may disregard their surroundings. Thus, prevent hazardous situations by thinking ahead about potential risk factors, such as sharp objects, moving furniture such as chairs, or slippery surfaces. Also, keep in mind that some activities are designed for outdoor setting and might not work properly in a closed room.

If you’re testing applications where the device needs to be far away from the participants, you may also need to have a device stand to enable participants to still see their device’s screen while interacting with the AR app.

Piloting the test beforehand, while recommended for any usability study, is crucial for AR studies because it can help you determine not only whether your tasks are clear, but also whether your room is safe and, also, how long the study session should be (see below).

Session Duration

You might need to allocate a longer time to an AR-testing session compared to a regular mobile-testing session, as the participants are likely to be less familiar with the AR technology and it might take them longer to complete the tasks.

Moreover, some AR apps can take a long time to download, and that can also increase session time if participants will need to install apps on their devices during the session.

You might consider sending study participants a list of apps to download beforehand. However, the danger of this approach is that, in a desire to “do well” during the test, participants will play with the apps before the session and, thus, contaminate your findings. Therefore, if you do decide on sending apps to users in advance, make sure you do it only shortly before the session (15-20 minutes before).

Alternatively, you might ask the participant to start the download at the very beginning of the session, after you introduced yourself, and then, as the apps load, you can talk about the study goals and explain the protocol to the participant.

To save time, you can also ask participants to sign consent forms ahead of time. Make sure, however, that participants do understand that they will be recorded and how their data will be used.

Recording Equipment

Often AR activities require participants to move around.  These interactions with the physical environment can uncover important limitations or opportunities in the AR interface.

If your application involves such interactions with the physical environment, it’s a good idea to run the study in person and record both the participant’s phone screen and their movement in the testing room.  In mobile-usability studies, we often use document cameras to record participant’s fingers as they are interacting with their phones. However, depending on the study’s goal, that may not be enough for your AR study.

While the phone screen can be recorded using Zoom or another similar software, we recommend that you use one or two tripod-mounted cameras to capture participants’ interactions with the physical environment. Have backup batteries and memory cards for these cameras. Also, doublecheck the recording limitations of the camera. Often DSLR cameras have a 29–59-minute recording-duration limit, which can interrupt the process of usability testing. If this is the case, the facilitator will have to move to the camera and restart the recording, which can be distracting for participants and impact their natural workflow.

Participant movement in the room can also impact the quality of your audio recording.  Make sure that the participant’s voice can be heard and recorded when the camera or the phone are far from the participant — for example by having the participant use a wearable microphone.

Note that some AR experiences use storytelling and narration throughout the experience. Thus, these narrations might also be recorded in parallel to the participant’s comments, which will make hearing the participant’s voice difficult. Either adjust the audio configuration of your recording software so it prioritizes human voices over the device’s output or rely on the external camera for reviewing the participant’s comments.

Choosing the Right Participants

AR apps can involve a lot of movement; some are even intended as fitness games. Based on the goal of the study and the type of tasks, your screening criteria should include factors such as participants’ age, health conditions, or physical ability. You do not want to jeopardize participants’ health, especially since some participants may feel pressured to do tasks that they are not totally comfortable with during a user test.

During the test, pay close attention to participants’ reactions and look for any signs of discomfort or other unforeseen dangers. Should such signs occur, stop the activities immediately.

Even minor details such as wearing glasses can impact the performance of the participant throughout an AR study. For instance, if the participant is playing a physical-exercise game and the device needs to be placed farther away, they may not be able to read instructions because they may not have their glasses on (or may not want to wear them during the physical activity).

Informed Consent

It’s always a good idea to let participants know what to expect during the study, especially if doing so does not risk compromising the validity of your research findings.

Therefore, before the study, inform participants of whether:

Allow users to withdraw from the study — before the study or during the study. And, if your tasks involve physical movement, remind them to charge their phones before the session — a plugged-in phone can be a tripping hazard.

If possible, have an observer or another person who could help the facilitator during the session. While the facilitator is focused on moderating the session, the helper can take notes, monitor the cameras, change any memory cards, or restart the recording as needed.

During the Session

Creating a customized checklist of the steps to take during the session can help you keep track of the many moving parts of an AR study.

The rule of thumb is to be calm and patient with the participants — even more so during an AR study than in a regular user test. Participants are often new to the world of AR and might not be familiar with AR patterns. This unfamiliarity will make some of the tasks confusing to them. Resist the urge to rush people through the activities and give them the chance to explore the app as they would do in their natural setting.

Remember that some people may not even know what AR is. You might occasionally need to guide them to move the test along. Make a note of all the situations where participants needed help, as those are improvement opportunities.

Mobile AR technology has a lot of potential in various domains and proper usability testing can ensure the success of the product and the return on investment. While conducting an AR-usability study can seem a complex process at first glance, with proper planning and preparation you can achieve your research goals. A significant part of this process is understanding your participants and putting yourself in their shoes, as many will not be familiar with AR technologies or may not be physically able to perform certain activities Explain your tasks clearly, using words that they can understand. Be aware of the setting and of potential safety hazards. Use an external camera to record the participant’s movements. Pilot the test to make sure that the study room or setup and the study duration are reasonable and that the tasks are all easy to comprehend. And last but not least, be patient with your participants, explain what’s going to be required of them during the session, and resist the urge to rush through the tasks.