When I was at AT&T, our user testing evolved through three phases:
First, we started out using third party labs to do our testing for us. (expensive)
Second, as we began building out our UX Research Team, we rented the lab space and paid for subject recruiting but handled the testing in-team. (less expensive, but still pricey) One of the big benefits of that however was that learnings remained in team. Sometimes there are findings that might be picked up that weren't part of the test plan, but can be cross leveraged in something else the team is working on. Those kinds of things are lost when farming out the process.
And then thirdly, we graduated to creating small labs onsite in Atlanta, Seattle, and Dallas, where we had UXR team members located. We still hired out recruitment, but handled the rest. This worked beautifully, as the knowledge was retained in-house, and we were able to spin up tests more quickly (within a week) to accommodate rapid iterative development timelines, as well as being more budget friendly.
Flash forward to 2020 and Covid lockdowns. I was standing up a new UX Research Team
in my role at Western Digital. Between slim budgets and everything needing to be remote, we gave UserTesting.com a try. What a godsend for the times!
We're now passed that period and in talking to teams at different companies, I have found that some people have pretty strong opinions about which is best. So having done both, here's my take on the subject.
Both methods have their own set of benefits and drawbacks, and it's important to consider which one is the right fit for your specific testing needs.
Let's start by exploring the benefits of using UserTesting.com. First and foremost, this platform and others like it allow you to reach a large and diverse pool of participants from all over the world. This means that you can gather feedback from a wide range of individuals, which can be especially valuable if you have a diverse user base. This was the case at Western Digital, where eCommerce is present in dozens of countries. Additionally, using a remote platform is often more cost-effective than bringing people into a lab, as you don't have to worry about overhead costs such as renting a physical space and testing subject payments. But beyond reach and cost is the speed element. You can spin up a test same day for quick response.
On the other hand, there are some drawbacks to using an online platform. One of the main concerns is the lack of control over the testing environment. When participants are using their own devices in their own homes, there are more variables that can impact the testing process, such as internet connection or distractions. Additionally, it can be more difficult to establish a personal connection with participants when using an online platform, which might result in less detailed feedback.
Someone I spoke with recently likened online test subjects to Uber drivers, professional testers just waiting for the next gig. I wouldn't rule out UserTesting.com on that basis though. Sometimes you just need a quick pair of unbiased eyeballs on a piece of functionality to validate something or break a design/development logjam. And in the case of Western Digital, they're selling internal hard drives, so looking for tech savvy end users anyway. So I guess you need to balance the type of project and targeted end user to decide if that impacts your decision.
Now, let's take a look at the benefits of bringing people into a user testing lab. One of the main advantages is the level of control you have over the testing environment. Additionally, having participants in a lab setting allows for more personal interaction, which can result in more in-depth discussions and insights.
However, there are also drawbacks to using a user testing lab. First and foremost, this method can be more costly and time-consuming if you don't have your own lab, as you have to account for expenses like hiring a third party or renting a space and compensating participants for their time. It's also an unfamiliar environment, so feels more clinical to a test subject than doing the test on their own computer from home or office. Additionally, you might have a limited pool of local participants, which could result in a less diverse range of feedback. That's why we had three labs in various parts of the country at AT&T.
If you're looking for cost-effective, quick and diverse feedback, an online platform like UserTesting.com might be the best option for you.
If you value control over the testing environment, looking for more in-depth personal interactions, and doing larger scale testing such as end to end baselining, bringing people into a user testing lab would be the better choice.
Ultimately, which is best will depend on your specific testing needs and resources. Let budgets, timelines, and project size determine your choice. I think it's important to have both options in your toolbox.