Arneo deciphers : Remote user testing

Preparing the study

From a methodological point of view, whatever the test format, you'll need to draw up a test protocol detailing how the study will work, and above all, imagining the test scenarios.
The face-to-face format allows for a fairly long scenario, with a facilitator to guide the test and the instructions printed out for the participant to refer to.
For the remote format, the idea is to be more concise and to break down the tasks into finer micro instructions. The aim is to limit the cognitive load and encourage independent completion of the task.
To give you an idea, where I used to have 4 or 5 scenarios to test in the classroom, we'll have between 10 and 15 instructions to present to remote testers.

Data collection

For the face-to-face test, I used to prepare my semi-directive interview guide and observation grid so that I could record my findings directly during the session with the participant. In this format, the facilitator has a major role to play in ensuring that the test runs smoothly, since he or she has to be able to adapt to each personality and mobilize softskills borrowed from interviewing techniques:

• Observe, listen actively without judging,
• Bounce back and reformulate while maintaining a neutral posture
• Putting the participant at ease

For remote testing, once the study has been launched and is therefore open to testers who wish to connect to it, there's nothing more to do! Finally, I'd still advise you to check at least the incoming entries to make sure that everything is going according to plan.

Data processing

In my opinion, the biggest difference lies in this stage.
In a face-to-face setting, we have already listened to and actively observed our testers in front of the interface. We've also been able to go back over certain points in the course and dig deeper into certain elements with them. So it's more natural to remember things, to identify the right moments to come back to, and we already have an idea of the study's insights as soon as the test is carried out.

In the case of remote tests, it's a bit of a blind man's game... we discover the videos of the participations with fresh eyes, which sometimes means we have to watch certain passages over and over again to fully understand what happened. And while it's true that data processing is made easier thanks to the Testapic solution, which lets you tag, sort and organize the observed findings into folders etc., I have to admit that I greatly missed not being able to chat with the participant. Not only to be able to check my understanding of his observations, but also because I found that the pleasure of interaction was lost.

Reporting the results

The aim is the same for a face-to-face or laboratory test: to synthesize the results of the study with findings and proposed recommendations in a deliverable to be handed over and presented to the customer. There's no debate about it.

In short, the advantages and disadvantages of each procurement method

To conclude, I'd say that my opinion isn't really clear-cut on the subject. It's like anything else: each format has its advantages and disadvantages. Nevertheless, my preference remains for the face-to-face mode. However, I understand the advantages of both formats. I'd also recommend the "remote" user test format, for simple courses such as an e-commerce site, for example. On the other hand, the "face-to-face" format remains the ideal solution for more complex projects, such as business interfaces, which merit rigorous investigation, since it allows for greater empathy with the participants.

In all cases, I'd always recommend carrying out a user research phase if possible.

In fact, this is an approach we systematically promote at ARNEO, whatever the project, to ensure that our customers receive the most optimized paths possible, as they have first been screened by end-users.