Usability Testing

A comprehensive guide to usability testing by Userbility

A comprehensive guide to usability testing

Welcome to our complete guide on usability testing! 

In this post, we will explore the ins and outs of this valuable method for evaluating products with real users. 

Usability testing allows you to identify any issues or challenges that users may face and make necessary improvements, ultimately leading to a better overall user experience. 

We will share everything you need to know to run effective usability tests and gain actionable insights. 

Let’s dive in!

What is usability testing?

Usability testing is a method of evaluating the functionality and ease of use of a product, such as a website or app, by observing how real users go about completing the tasks you set them.

The goal of usability testing is to identify areas of confusion or difficulty for users and to make improvements to enhance the overall user experience. 

Usability testing can be conducted at different stages of development and is often repeated to ensure continuous improvement.

It’s important to note we often hear the term ‘website usability testing’ and ‘user testing’ used interchangeably with ‘usability testing’.

Whilst ‘website usability testing’ can be used interchangeably, ‘user testing’ cannot and has a different definition. We’ll explore this further down.

Why is usability testing important?

Studies have shown that for every $1 invested in improving your website usability, your website returns $10 to $100.

That’s an impressive 10-100X return on your investment.

Apart from the monetary returns, here are the top 3 reasons why usability testing is important: 

Reduction of risk

Usability tests can reduce risk by catching usability problems and bugs before development begins, as well as catching problems before a product is launched – reducing the risk of impacting critical business metrics.

Improved user experience

Usability testing allows you to gather insight and feedback on the experience and if the experience is exceeding users expectations, delivering satisfaction and in-turn, assisting you in developing personalised experiences to improve satisfaction.

Removal of bias

As usability tests involve users or customers familiar with your product, and aren’t the creators of the website or product, this provides an unbias opinion to what is and isn’t working.

You are not your customer.

The benefits of usability testing

Usability testing benefits

Whilst the importance of usability testing can be covered in three broad categories, these categories spill out into an array of benefits.

The benefits you receive from usability testing only expands as you continue to test and learn, and develop your user experience.

However, there are 5 primary benefits you can expect to experience:

Geting stakeholder buy-in

Knowing about a website problem and seeing the problem first-hand are two different things. Being able to see and hear someone’s frustration can be a persuasive technique to build change within your business.

Increased conversion rates

Usability testing can reveal friction points within your website or product experience. By not fixing these friction points, you may experience users abandoning their journey e.g. not continuing in the checkout process. Fixing these friction point create a more seamless experience and increase the likelihood conversion increases.

(We recommend you a/b test the changes or conduct a pre/post usability study).

Save time and money

Testing early and testing often can save design and development time during the design process of your product. This ultimately saves the business money in the long run.

Validate or invalidate designs

Receiving first hand data on which designs have the lowest and highest usability scores, can allow you to validate or invalidate design ideas for solving complex problems.

Provide an additional data point

Whilst usability testing can be a qualitative behavioural method or a quantitative behavioural method, it provides an additional data point that can be partnered with existing insights from other methods e.g. a/b testing results.

Different types of usability testing

Different types of usability testing

This guide covers usability testing, which is a broad topic and often confusing.

Before diving into specific types of usability testing, it’s important to understand the different categories of usability testing methods.

These methods can be broadly divided into three categories:

  1. Remote or in-person
  2. Unmoderated or moderated
  3. Qualitative or quantitative

Remote or in-person

Before you create a usability testing script, it’s important to decide how you’re going to conduct your usability test.

Usability tests can take the form of remote or in-person, with both having disadvantages and advantages when comparing them against each other.

As a general rule of thumb, in-person usability testing is best for physical products, whilst remote testing is inexpensive when compared to in-person.

RemoteIn-Person
Great for physical products✔
More affordable option✔
Can capture body language easily✔
Greater flexibility✔

Unmoderated or moderated

You’ve decided if your test will be remote or in-person. You now have to decide if the test will be unmoderated or moderated.

Much like remote or in-person testing, unmoderated and moderated testing follows a general rule of thumb:

Both practices aim to achieve the same objective, however, moderated testing has the advantage of a facilitator – someone who can steer the direction of the usability test.

Even though moderated testing has the advantage of a facilitator, it usually comes at a pretty penny.

UnmoderatedModerated
Can prompt in real-time✔
Low cost to setup✔
Can easily capture body language✔
Quicker turn-around for results✔
Easier to test prototypes✔
Greater flexibility✔

Qualitative or quantitative

The observations and insights you collect from user research will fall into two buckets: qualitative or quantitative.

The objective of your usability test will determine if your study will be qualitative, quantitative or both.

For example, if you aim to understand perceptions, behaviours and emotions, your test will be qualitative based.

If your objective is to quantify the usability of a website or product, your test will be quantitative based.

QualitativeQuantitative
What, where, when✔
Why✔
Numerical data, statistics, percentages✔
Reactions, perceptions, behaviours✔
Quantify usability✔
Insights on emotions and behaviour✔

What usability testing is not

There are several user research methods that aim to reveal customer insight or have a direct impact on the user experience, but don’t qualify as usability testing.

This is because they don’t explicitly focus on the ease of use of product, invalidate/validate a design or replicate the user experience:

Surveys

A great methodology to understand the why behind behaviours and user actions, surveys aren’t usability testing. The main reason is because, surveys don’t allow you to observe users or visitors whilst they use your product or website.

Nonetheless, surveys are a great methodology to use in conjunction with usability tests to understand the why behind user behaviour.

Heatmaps

Providing a representation of user navigation and user site interaction, heatmapping provides a visual way to reveal the most interacted (hotspots) and least interacted (coolspots) of your website or product.

Heatmaps aren’t a substitute to good usability testing, but can provide support for or against observations in your usability test.

Focus-Groups

Whilst a usability test aims to understand how people use a product, focus groups aim to understand and gather opinions from a group of people about a product.

There is a similarity in that focus groups, much like moderated testing, have a facilitator. We’d recommend using focus groups to validate or invalidate if people want your product.

A/B Testing

A/B testing is a great methodology to combine with results from a usability test, as they’ll allow you to test two versions against each other and see the impact on business metrics.

For example, you could a/b test two designs, one with better usability than the other and see which has a greater impact on conversion rate.

When should you conduct usability testing?

There are 4 main intervals for when usability testing should be conducted. 

They are:

  • After creating a wireframe or prototype
  • Before the launch of your website or product
  • Regularly, after the launch of your product
  • In uncertain times

Testing the usability of your prototype

Testing your prototype early on is a type of formative testing, as the usability test helps you gather feedback and insights early in the design process.

Testing your prototype early and often before passing it on to development can help reduce time building the product and save money by removing any usability bugs early on.

Before you launch your website or product

Your prototype successfully passed the usability testing and has been built. Now is a great time to conduct another round of usability testing.

This type of usability testing is known as summative testing and aims to prevent any major usability issues by evaluating if participants can effectively complete their goals with the product.

Regular intervals after your product launch

It’s important to conduct usability testing regularly to ensure your product continues to meet the needs and goals of your users.

Regular testing will allow you to continue capturing any usability issues as well as provide a steady stream of user and customer insights on your product’s performance.

Likewise, conducting regular tests will allow you to benchmark your product over time as your product and users change.

During times of uncertainty

When there’s disagreement between teams or a lack of clarity on what’s the best course of action to take with a design, usability testing can be the shining light.

Well structured and thought-out user research always provides insights that can be used and provide direction for the next course of action.

What do the experts say?

When you have a more developed prototype or a built product / service. You need to have something that functions in the way you intend it to (or as close as possible). This could be testing a new feature, or e2e (end-to-end) on a website.

Sarah O'Donnell - User Researcher - AND Digital Tweet

What's the difference between user testing and usability testing?

Difference between user testing and usability testing

The term user testing and usability testing are often used interchangeably within the industry – confusing designers, product managers and developers alike.

We’re here to argue there are a few core differences between the two.

User Testing

User testing can be seen as an encompassing term for numerous user research methods that aim to test whether your target audience needs your solution or service.

These user testing methods compliment usability tests in order to improve the user experience.

For example, combining on-site surveys, usability test results and session replays would provide adequate data to create an a/b test.

Likewise, user testing methods can be combined with usability tests to understand more from test participants.

For example, adding a post-test survey to a usability test.

Usability Testing

Unlike user testing, usability testing is a singular research method with different combinations of execution.

The different combinations of execution are:

  • Remote unmoderated or remote moderated
  • In-person unmoderated or in-person moderated

This allows you to execute usability testing depending on your budget, access to tools & resources OR what usability test you have the most time for.

In another chapter, we’ll explore the core differences and benefits of each way of executing usability tests, for now, here is a table that summarises the core differences between user testing and usability testing.

User TestingUsability Testing
Focuses on ease of use✔
Focuses on product adoption✔
Invalidates/validates a design✔
Invalidates/validates an idea✔
Different ways of execution✔
A mixture of methodologies✔

Best practices to start usability testing

You’ve made it this far and are keen to start your first usability test.

Before starting your first usability test, it’s best to know some of the hiccups you can encounter and best practices to follow.

#1 Create a usability test plan

Before writing out your testing script, you need to have a test plan in place to ensure you get the most out of your time.

Some of the most common things to cover in your usability test plan, include:

  • Objective and questions: what’s the objective of your usability test and what are the questions you need to answer in order to achieve your objective? e.g. Objective: To understand if it’s easy or difficult to add a product to cart. Question: how long does it take someone to add a product to cart? 
  • Type of usability test: will the usability test be conducted remotely or in-person? Will the usability test be moderated or unmoderated? These are questions to understand the benefits and drawbacks to.
  • Metrics: what metrics are going to be collected and how are they going to be measured? It’s important to match the metrics you measure to the objective of your usability test.
  • Participant recruitment: who will take part in your usability test? How will you recruit your participants and do they match your real website/product users?

#2 Recruit your real users/visitors

It’s important to recruit the real users of your website or product when conducting a usability test.

The reason for this is to ensure the people conducting the test display similar behaviours to those who would use your website or product.

This ensures your website or product allows your users to achieve their goals effectively and efficiently, your website or product is easy to learn for first time users and easily remembered for repeat users, and your website or product provides a high level of user satisfaction.

By not using your real users, you wouldn’t get an accurate representation. 

#3 Conduct a pilot test & QA the instructions

An important step to capture any hiccups in your test. Conducting a pilot test allows you to see how people interpret the scenario, tasks and questions and allows you to make any important amendments before sending the test to more participants.

#4 Take a note of observations & insights

Because the purpose of usability testing is to uncover any difficulties or challenges that users may experience, it’s best to classify findings into two categories: observations and insights.

You can classify your observations based on the level of severity and frequency.

When an observation has a high level of frequency, it can be deemed an insight.

Tip: utilise the playback speed settings to speed up the analysis e.g. 1.5x.

Also note, it’s important to watch the recordings first in 1x speed to catch any pauses or moments of delay from the participants.

Frequently Asked Questions about Usability Testing

Questions about usability testing

Yes, it is possible to automate some aspects of usability testing. For example, there are tools available that can automatically evaluate the usability of a website or app by analyzing elements such as navigation, layout, and content organization. 

Alternatively, creating a usability study that runs 1 month would allow you to automate your testing over a certain time period.

It is important to supplement automated testing with manual testing by real people to get a complete understanding of the user experience.

Usability testing can involve both qualitative and quantitative data.

A qualitative usability test would focus on understand the perceptions, emotions and and behaviours of test participants, when they use your product or website.

A quantitative usability would primarily focus on quantifying the usability of your website or product by measuring aspects of the test e.g. time on task, task success and error rate and measuring SUS.

Regardless of the type of usability test you run, both types of data can be useful in understanding the user experience and identifying areas for improvement.

Usability testing is important because it allows you to identify and address any issues or challenges that users may face when using your website or product.

 This is important because poor usability can lead to a frustrating user experience, which can cause users to abandon a product or service. 

On the other hand, good usability can lead to a positive user experience, which can increase customer satisfaction and loyalty. 

Usability testing is a type of functional testing, which means that it focuses on the functionality of a product and how well it performs the tasks it was designed to do. 

Usability testing is a specific type of functional testing that focuses on the ease of use and user-friendliness of a product.

The four principles of usability testing are:

  1. Learnability: This refers to how easy it is for users to learn and become proficient at using a website or product.

  2. Efficiency: This refers to how quickly and efficiently users can complete tasks on a website or product.

  3. Memorability: This refers to how well users can remember how to use a website or product after not using it for a period of time.

  4. Errors: This refers to how well a product handles and recovers from errors made by the user.

These principles are often used as a framework to evaluate the usability of a website or product and to identify areas for improvement.

The purpose of usability testing is to identify any issues or problems users may have with your website or product, and to identify areas for improvement, in order to deliver a delightful user experience.

There are several ways to perform a usability test, but a common method is as follows:

  1. Define the goals and objectives of the test: Determine what specific aspects of the product you want to evaluate and what information you hope to gather from the test.

  2. Recruit test participants: Identify a group of representative users who will participate in the test.

  3. Prepare the test environment: Set up a testing area and ensure that all necessary equipment and materials, such as a computer and the product being tested, are ready.

  4. Conduct the test: Have each participant complete a set of predefined tasks while being observed and/or recorded. Be sure to give clear instructions, and let the user complete the task as they want.

  5. Collect and analyse data: Gather data from the test, such as observations of user behaviour, task completion times, and any comments or feedback provided by the participants. Analyze the data to identify patterns and trends.

  6. Report and act on findings: Use the data collected to create a report that summarizes the findings and recommendations for improving the product. Share the report with the relevant stakeholders and work on incorporating the recommendations into future designs or version of the product.

It’s also important to mention that there are different types of usability testing methods, like remote, moderated, unmoderated, etc. each one is suitable for different stages of the product development.

Here are a couple examples of usability tests:

Accessibility refers to the design of products, devices, services or environments, so that they can be used by people with disabilities.

Some considerations include: providing alternative text for images, using larger fonts or even caption videos.

Usability, refers to how ‘usable’ the design of the website, product, device or service is and primarily looks at how well a website or product can used by a wide range of people to achieve their goals.

The Comprehensive Usability Testing Guide

You’ve made it to the end – congrats! 🎉

These additional 6 chapters cover the basics of usability testing and will provide you with a strong foundation of knowledge in this area. 

Using this guide, you’ll develop the fundamental skills to conduct effective remote, in-person and moderated/unmoderated remote usability tests.

Chapter 1: Usability Testing Methods

Before conducting your first usability test, you need to know about the methods you have available.

Coming soon.

Chapter 2: Remote Usability Testing

Learn the ins and outs of conducting  successful remote usability test.

Coming soon.

Usability Testing Script Writing

Chapter 3: Usability Testing Script

Getting the most out of the usability tests involves a good usability testing script.

Coming soon.

Chapter 4: Usability Testing Questions

Incorporating great questions within your testing will allow you to extract additional understanding.

Coming soon.

Usability Testing Principles

Chapter 5: Usability Testing Principles

Understand the fundamental principles to usability testing.

Coming soon.

Usability Testing Checklist

Chapter 6: Testing Templates & Checklists

Testing templates to help you get started with your next test.

Coming soon.

Chapter 7: Analysing & Reporting Test Findings

How to analyse test results and report your findings to stakeholders.

Coming soon.

Userbility Footer Logo

Copyright © 2023

Made with ☕ in Manchester

Resources

Blog

Get Paid to Test

User Testing for Australia

Usability Testing Guide

About Us & Help

About Us

FAQ

Privacy

Term & Conditions

Follow Us

Facebook-f Twitter Instagram Linkedin