User Experience Insights

How to Recruit Usability Test Participants

First, a little digression on sample size and test validity

Usability tests are only valid if the participants (test subjects) are drawn from a representative sample. Sampling allows the researcher (in this case, the usability test administrator) to generalize findings from a subset onto the general population. Most people are familiar with random sampling, but this is not the best method for recruiting pariticipants in a usability test, because visitors to a particular site or users of a particular online product tend to have key things in common—what I call defining characteristics. These are the characteristics that the participants must have in common for the test results to be relevant and so that you can generalize the results.

The sampling method for a usability test needs to balance the cost of adhering to the method against a) the level of precision needed for the study, and b) the time it takes to analyze the study data. While some usability testing specialists might advocate seeking only statistically significant results, I advocate using the most cost efficient, yet sufficient number of participants. Jakob Nielsen and other usability experts found that over 80% of a site or product's functional deficiencies can be found with as few as 5-7 participants. Due to their cost- and time-effectiveness, I advocate using this number of participants for "initial assessment" tests (early in the design process). If time is available, and for the sake of garnering more thorough feedback, a larger number of participants may be used for "ongoing validation" tests (after the Website or product has been developed and deployed).

Finally, on to talking about finding participants

Given that you need only 5-7 participants for your initial test, it is vital that you solicit participants that have all of the "defining characteristics" of the real Website or product users. Examples of three defining characteristics are:

The specific set of defining characteristics will vary from application to application. To determine them with confidence, you will need to speak to people in your organization that are closest to the users, if not the users themselves. (Be careful not to just take your management's opinions of key user characteristics; make sure that these characteristics are substantiated with real customer or prospective customer data). False assumption really can harm you; take care to avoid them.

In general, test participants should have had no prior exposure to the Website or product being tested (i.e., have never seen or used it). If internal people "know too much" or may potentially give biased feedback, you will need to solicit people outside the company. If you're planning a series of tests for an existing product that does require participants that have previously used the site or product, sales/marketing customer lists are probably your best resource. Sales managers are often willing to contact clients, and ask them if they know users with the defining characteristics who would be willing to participate in your test.

If the product is new, it is usually necessary to use the following methods to find test participants:

This should give you enough guidelines to get started. Remember, you can use fewer participants, but, if you want to get results that you can confidently communicate to your development group, just make sure you pick your participants from people "swimming in the right participant pool." Also, remember that perseverance pays off—you may sometimes have to actively recruit 12-15 people to find 5-7 who are available to participate in your test.


	How to Recruit Usability Test Participants First, a little digression on sample size and test validity Usability tests are only valid if the participants (test subjects) are drawn from a representative sample. Sampling allows the researcher (in this case, the usability test administrator) to generalize findings from a subset onto the general population. Most people are familiar with random sampling, but this is not the best method for recruiting pariticipants in a usability test, because visitors to a particular site or users of a particular online product tend to have key things in common—what I call defining characteristics. These are the characteristics that the participants must have in common for the test results to be relevant and so that you can generalize the results. The sampling method for a usability test needs to balance the cost of adhering to the method against a) the level of precision needed for the study, and b) the time it takes to analyze the study data. While some usability testing specialists might advocate seeking only statistically significant results, I advocate using the most cost efficient, yet sufficient number of participants. Jakob Nielsen and other usability experts found that over 80% of a site or product's functional deficiencies can be found with as few as 5-7 participants. Due to their cost- and time-effectiveness, I advocate using this number of participants for "initial assessment" tests (early in the design process). If time is available, and for the sake of garnering more thorough feedback, a larger number of participants may be used for "ongoing validation" tests (after the Website or product has been developed and deployed). Finally, on to talking about finding participants Given that you need only 5-7 participants for your initial test, it is vital that you solicit participants that have all of the "defining characteristics" of the real Website or product users. Examples of three defining characteristics are: Primary job description/duties Length of experience with the Web (total number of years used; hours used per day) Length of time using relevant application (e.g., SABRE airline reservation system for a travel booking software product) The specific set of defining characteristics will vary from application to application. To determine them with confidence, you will need to speak to people in your organization that are closest to the users, if not the users themselves. (Be careful not to just take your management's opinions of key user characteristics; make sure that these characteristics are substantiated with real customer or prospective customer data). False assumption really can harm you; take care to avoid them. Regarding previous exposure to the Website or product In general, test participants should have had no prior exposure to the Website or product being tested (i.e., have never seen or used it). If internal people "know too much" or may potentially give biased feedback, you will need to solicit people outside the company. If you're planning a series of tests for an existing product that does require participants that have previously used the site or product, sales/marketing customer lists are probably your best resource. Sales managers are often willing to contact clients, and ask them if they know users with the defining characteristics who would be willing to participate in your test. If the product is new, it is usually necessary to use the following methods to find test participants: Contact temporary employment agencies (especially if you're seeking less technically-savvy, younger or administrative-type participants). Contact market research organizations (since they likely already know the key characteristics of their research participants). Contact professional groups in the applicable field (e.g., a marketing association if you are testing a marketing-oriented product). Use personal contacts. The old fashioned fall-back source. The key here is to ensure that the participants are not chosen just because you know or like them; you must objectively ask yourself (or them) whether they have the defining characteristics you seek. If so, great; if not, start looking elsewhere. This should give you enough guidelines to get started. Remember, you can use fewer participants, but, if you want to get results that you can confidently communicate to your development group, just make sure you pick your participants from people "swimming in the right participant pool." Also, remember that perseverance pays off—you may sometimes have to actively recruit 12-15 people to find 5-7 who are available to participate in your test. °°° Mark D. Hall San Diego, CA, USA August 12, 2002 Want to comment on this article? Please write Mark.

	< Back to Newsletter Home

	Copyright 2002 HallMark Consulting All rights reserved