Remote Usability Testing
I just finished up a rather intensive research phase for a client recently. One of the goals of the research was go find out how peope are finding information at work. The client is rather large and has employees all over the US, Europe, and Asia. We worked with the business units to determine an appropriate sample size and representation.
Over the course of two weeks, we interviewed close 42 participants. About half of the participants were within an hour from us. For those participants we conducted face-to-face observational interviews. But the rest were located across the US and in Europe. We couldn’t have done this without remote methods.
For the remote observational interviews, we used a combination of SameTime (it’s a kind of instant messenger) and a standard phone call (VoIP). While scheduling participants in Europe was a challenge, the remote observation method worked really well. We were able to see the participants’ native environments, watching them use their bookmarks, browse shared drive spaces, launch applications, sign-in, etc.
Remote methods allowed us to observe there native environments without either one of us having to travel. It also gave us the ability to do a much larger sampling – especially with geographically dispersed participants.
Remote methods aren’t without their drawbacks, however. Most of the remote methods out there today like Ethnio and Morea only work on Windows and require an installed application. I’m a bit aprehensive asking a participant to install something on their machine so we can observe them. I like to leave their system untouched. Fortunately, in this case, the client already has a standard application installed that allows for remote observation. So, it wasn’t an issue.
Anyone who knows me knows that I’m first and foremost and advocate for face-to-face methods. I’ve just found that the native environment will reveal details of how and why that you simply cannot get with a survey, or sterile lab environment. But if you can’t do a face-to-face, then remote with the native environment is your next best thing.
I agree with you about the value of remote testing is really about users in their native habitat. I have been using Ethnio for quite some time and I have discovered the same things about remote testing. You will learn details from remote testing that you can’t learn from in-person testing. Remote testing should be part of your research mix, which includes face-to-face, surveys and etc. I really like the speed and flexibility of remote testing since it provides a lot of value to agile development.
I work within a usability center for a multinational firm and we frequently rely on remote usability testing. Adding to a few of the benefits of remote testing: multiple observers, can be cost saving for all parties and fits to the testers schedule. We run remote tests over Sametime and phone as you described. We can then have multiple observers including key stakeholders, which is great to resolve any differences related to usability. (just make sure everyone is on mute – odd and unexpected comments can send a tester into a tailspin) It goes without saying you can save a lot of $$ when you don’t have to be a globetrotter, even though I wouldn’t mind touring the world. I also agree with Brian about a native habitat, it opens up the user for great one-off comments etc. Some drawbacks are poor connections and the occasional translation issue which would be easier to resolve in person.