Explore the advantages and disadvantages of usability studies

Now that you know the basics of moderated and unmoderated usability studies, let’s dig into the details and explore the benefits and limitations of each type of study.

Remember, moderated usability studies have a person, known as a moderator, guide participants through the study in real time. Let’s begin by exploring some advantages of conducting a moderated usability study.

• Build rapport. Moderated usability studies allow for rapport building between the moderator and participant, which can help the participant open up and share more feedback about the design. Building rapport is especially important if the design being tested deals with sensitive or personal issues. In addition, speaking live can make it easier for the researcher to understand the users’ backgrounds and empathize with their needs.
• Observe firsthand. Feedback and learnings can be more impactful when you see participants interacting with your prototype or product firsthand, instead of watching a recording or reading about the results. You can also bring in stakeholders to observe the study, either remotely or in an observation room, to get the same high-touch interaction with real users.
• Provide live guidance. The moderator guides each participant through the study live, which helps to ensure that participants complete the exact tasks that you want to get feedback about.
• Explain the user journey. The moderated setting gives you more time and opportunity to test ambiguous or complex problems with your product that could be misinterpreted if there wasn’t guidance. This is why moderated usability studies are often used early in the design process, when low-fidelity prototypes might be unclear to users.
• Control the conversation. A moderator keeps the flow of the session from being interrupted and helps participants stay focused, which makes the study more productive overall.
• Make language adjustments. The moderator can rephrase a prompt or question that a participant is not understanding. This is beneficial because it allows you to collect more information that you can act on to improve your design.
• Gain clarification. The moderator can ask participants to clarify the feedback they’re providing or to elaborate further (e.g., “tell me more about that”), which can lead to richer insights.

With all of these benefits in mind, it might seem like moderated usability studies are the way to go! However, it’s important to note that moderated usability studies have some disadvantages too.

• Influence or bias. Because a person is guiding the study, there’s a chance that the moderator can accidentally let their own thoughts or feelings influence the way that they ask questions or in the body language they show, which can affect the study’s results.
• Self-identification. If participants do not connect or identify with the moderator, they might have a hard time being open about their experience using the product, especially if the subject matter is personal or sensitive.
• Participant pool. Since a human moderates each session live, most moderated usability studies happen during standard working hours. Some populations are hard to reach during standard working hours, and therefore could be underrepresented in your study findings.
• Difficult scheduling. Because you are coordinating two people’s schedules (the moderator and the participant), it can be difficult to schedule moderated usability studies, and it can be hard to reschedule a participant who cancels or fails to show up.
• Reliable internet required. Internet connectivity issues for remote moderated sessions can throw off the tempo of the session, can make it harder for participants to communicate effectively, and can even cause the session to end early in some cases.
• High cost. Moderated usability studies are comparatively expensive because someone needs to spend time serving as the moderator for each individual session and because there is often a cost to rent space to host sessions in person.

Alternatively, unmoderated usability studies have participants test the prototypes without human guidance. An unmoderated study can be done by recording a participant’s screen or having the participant give feedback on tasks. Since there’s nobody to assist, it’s super important that your designs are clear and accessible to all users.

Let’s start with some advantages of conducting an unmoderated usability study.

• Natural product interactions. Because users are mostly unsupervised while completing tasks in the product, they’re more likely to use the product naturally and intuitively. This can lead to a more accurate understanding of the user’s experience in the real world.
• More participants. Unmoderated studies make it easier to include more participants because you don’t need to spend time guiding each participant live. You can also recruit participants in different time zones and with different availability, like outside of standard working hours. Keep in mind that you want to have a minimum of five participants, which is a large enough sample size to gain valuable feedback, and a maximum of eight participants, since there is often diminishing return on investment if additional participants are added to the study.
• Easier scheduling. Unmoderated usability studies allow participants to complete tasks on their own time and in their own space because there is no need for a moderator. If a participant does not show up or needs to reschedule the session, it’s easier to do so when it’s unmoderated, since the participant can do it on their own time.
• Inexpensive. Unmoderated studies are more cost-effective than moderated studies, since they don’t require a facilitator or physical location to conduct the study.
• More privacy. If the product involves sensitive or personal topics, an unmoderated setting can make participants feel more comfortable answering honestly and sharing their perspective, since they aren’t required to speak with someone face-to-face.
• Simple user journeys. Unmoderated usability studies work well to test straightforward user journeys and simple features of a product, since participants don’t need a lot of guidance to complete tasks.

Of course, unmoderated usability studies have some disadvantages too.

• No real-time support. If participants don’t understand the wording of the prompt or if they have technical difficulties, there is usually no one to provide support.
• No live follow-ups. Unmoderated usability studies don’t give the UX team an opportunity to ask participants real-time follow-up questions or to clarify their feedback. This can be limiting if the participant does not explain their problem in detail because there is no way to follow up while the UX team watches the recording of the session later.
• Potential distractions. There is little to no control over the environment to ensure you have the participant’s full attention. Without a moderator present, participants could multitask and not be focused on the key activities in the study.
• Possible privacy breaches. To test the use of your product in an unmoderated study, you might need to upload your designs to a testing platform, which participants access from their homes. This means there is less control over your intellectual property.
• Inaccessibility. The testing platform you use to conduct remote research might not be equally accessible to users with disabilities or people who use assistive technology.
• More data to analyze. Because there is no moderator to control the conversation of the session, the information you collect might be off topic or irrelevant to your research.
• Simple user journeys only. Unmoderated studies are not great for testing complex user journeys because if a user gets stuck with a complex prototype, there’s no one to guide them through the process.

Start testing

When you’re choosing whether to conduct a moderated or an unmoderated usability study, there are a lot of things to consider! Both can help you get the research data you need to improve your product; it’s just a matter of deciding which one will do it better. Keeping these advantages and disadvantages in mind can help you make the right choice for your specific project and research goals.

Sources: Google UX Design Professional Certificate

Coursera | Conduct UX Research and Test Early Concepts