Pre-testing and piloting questionnaires | Survey Methods Handbook

In this chapter…

This chapter explains how to find problems in questions, instruments and field procedures before full data collection.

By the end of this chapter, you should be able to…

distinguish expert review, cognitive testing, technical testing and field piloting
select pretesting methods that match the likely source of error
use evidence from testing to revise and retest an instrument

In this chapter

By the end of this chapter, you should be able to:

explain why pre-testing and piloting of questionnaires is advisable;
distinguish between developmental trials and experiments, cognitive testing of questions and instruments, and rehearsal pilots;
describe which approach to piloting and pre-testing is most appropriate in a given circumstance;
outline the principal methods of each approach to pre-testing and piloting.

Why pre-test and pilot questionnaires?

Unless a questionnaire consists entirely of previously tested and validated questions which have been successfully used together before on similar survey populations, it is advisable to go through a process of pre-testing and piloting. This process helps to ensure face and content validity (see Chapter 2), and may indicate the need for re-wording, re-formatting or other refinements to the questionnaire itself or to the proposed conduct of the survey. Pre-testing and piloting is also recommended when previously validated questions or questionnaires are to be used in a new setting, with a new type of respondent or using a new method of administration – questions that work well in interviewer-administered questionnaires will not always work so well in self-completion questionnaires and vice versa. Make provision in the budget and timetable for planning and conducting pre-tests and pilots, and interpreting, feeding back and acting upon the results.

Different types of pre-test and pilot

There are broadly three types of pre-test or pilot, each with a different purpose:-

Developmental trials and experiments
to explore new topic areas;
to test the feasibility of novel methods;
to develop the survey instruments;
to focus on particular problem areas.
Cognitive tests of questions and instruments
to check how respondents cope with the questionnaire and instructions;
to check whether respondents understand the questions as intended;
to check how the respondents set about answering the questions.
Rehearsal pilots
to test the survey procedures overall, rather than just the questionnaire, under ‘main survey’ conditions;
to detect and remove minor ‘glitches’;
to provide better estimates on which to base sample size calculations (see Chapter 12);
to estimate the rate of and speed of response to the main survey (see Chapter 10);
to check timings;
to smooth co-ordination and establish systems and routines conditions and thereby to inform planning of the survey management process (see Chapter 11).

For an important survey in an area which has not been previously explored, all three types of pre-test and pilot work may be required. Exploratory work, or developmental approaches, often using qualitative methods such as in-depth interviews, can be an important aid to mapping topics and content area (see Chapter 5). In situations when the basic topic and approach are clear, the priorities will be cognitive testing and rehearsal pilots. These two approaches do different jobs and are not interchangeable. If it seems that administration of the survey will not be problematical, then the priority should be cognitive testing of questions, questionnaires and other survey documents (e.g. covering letter and any supplementary information; reminder letters), especially when the questionnaire contains questions or instructions not previously tested in the population of interest. Cognitive testing is particularly important in the case of self-completion questionnaires, since there is no opportunity to obtain feedback from interviewers on how the respondents reacted to and interpreted the questions, and there is no possibility of skilled interviewing making up for poor questionnaire design.

Cognitive pre-testing

Cognitive pre-testing methods (e.g. Schwarz et al., 1996; Schwarz and Sudman, 1996) are about uncovering what goes on in the minds of respondents when they receive the survey ‘package’ – for postal surveys, this package is likely to consist of a covering letter, questionnaire and a return envelope.

A weak link in the process of conducting postal or other self-completion surveys is the task of persuading the respondent to complete and return the questionnaire, to yield a high response rate. If respondent are to return the questionnaire, they must: open the package; read the introductory message (in the covering letter); start filling in the questionnaire; understand the questions as the researcher intends; answer the questions completely and in the way intended by the researcher; return the completed questionnaire. Many factors can affect the chance that all this will happen as planned (Dillman, 1978; Brown et al., 1989). Cognitive testing can give clues to how well different aspects of the postal package (e.g. out-going and return envelope; stationery and letterhead; size and weight of the package; appearance and presentation of the contents) are working – whether they attract the interest and curiosity of the respondent, or act as a turn-off. Cognitive testing can also indicate whether the covering letter is fulfilling the aims of communicating with the respondent regarding: the sponsorship and aims of the study; the way in which sampled individuals have been selected; the importance of and implications of participating; how data will be handled; how confidentiality will be maintained; how and in what form results will be made available; how respondents can make contact with those conducting the survey.

Comprehension of questions may also be a problem particularly if the survey researcher has little experience of communicating with the types of people who it is hoped will respond. Inexperienced designers may also misjudge whether respondents will be willing and able to give adequate answers. Cognitive testing can indicate how respondents are interpreting questions, response categories and instructions and how they are going about formulating their answers.

The main cognitive testing techniques require direct interaction between a researcher, or specially trained interviewers, and test respondents. The test respondents should be drawn from the same population as the subjects for the main survey. There is no hard-and-fast rule about sample size; samples are usually relatively small (generally not more than 20). However, in contrast to the random, probability sampling procedures to be employed in the main survey, the sample is purposively selected to ensure representation of groups who may have different problems with responding (e.g. to ensure a good spread of age groups, levels of education and literacy). The tests can take place in the respondents’ own homes or at a central location. A small fee and / or travel expenses are usually paid to respondents, because a significant effort is required of them.

Each test respondent is asked to go through a procedure that mimics as closely as possible what self-completion respondents will be required to do – open the package; read the covering letter; fill in the questionnaire; place the completed questionnaire in the return envelope. If desired, the procedure can be confined to completing the questionnaire or key parts thereof. Test respondents provide feedback to the researcher or interviewer, which is then summarised into action points for revising the survey documents and procedures.

One approach to providing this feedback is the concurrent ‘think aloud’ approach, in which respondents are asked to verbalise their reactions and thought processes as they go through the required steps, while the researcher or interviewer takes notes or tape records these comments. However, not all test respondents can cope with verbalising their thought processes as they go along. An alternative approach is the ‘debriefing interview’, using retrospective probes. The test respondent completes the required steps in self-completion mode, unobtrusively observed by the researcher or interviewer (who may take notes). Respondents are then asked taken through the steps again, with the interviewer prompting with directive probes such as:-

What did you think when you first saw / first opened the package?

What did the letter tell you?

Did you happen to notice this instruction before the question?

What did you think you needed to do to answer that question?

What sorts of things were you thinking about when you answered that question?

I noticed that you hesitated before you answered that question. Why was that?

Once again, the interviewer takes note on and/or tape records the responses to these probing questions.

Sometimes there will not be the time or resources to carry out cognitive testing. A cheaper and quicker alternative is review by an expert panel. This is a technique that can be used when the survey researchers have access to colleagues experienced in survey work, but who are not directly involved in the current survey. The proposed documents and procedures are sent to panel members for review. Written comments may be given or an informal seminar may be arranged at which the designers and panel members exchange views and suggestions for improvement. Expert panels may also be used as a preliminary to cognitive testing. At the very minimum, even if none of the formal procedures described above are used, show the questionnaire and any accompanying documentation to colleagues, family and/or friends; a fresh eye can often pick out ambiguous or confusing questions or instructions.

Following whatever pre-testing approach is adopted, the questionnaire should be modified (if necessary) to take account of the feedback. If extensive modifications need to be made, it may be advisable to repeat the pre-test.

Rehearsal pilots

Once a final draft questionnaire is available, the whole survey procedure should be pilot tested, ideally with a larger sample (often 30-100, though the actual sample size will generally be dictated by resources). The sample used should again represent the variation in the types of respondent and respondent circumstances that will be met with on the main survey. In the pilot test, it is desirable to match methods of administration to ‘real life’ circumstances (for example, if the questionnaire is intended for self-completion in a doctor’s surgery, the pilot test should be carried out under those conditions). The rehearsal pilot study should give a feel for:-

How the survey procedures work as a whole;
How well questions seem to work; poor questions may be indicated by:-
frequent failure to respond to the question ;
inappropriate responses;
inconsistent responses;
lack of spread of responses;
an apparent need for new response categories.
Likely response rates.
Likely non-response bias.

Once again, the questionnaire and data collection procedures should be refined on the basis of the findings from the pilot test. As with the pre-test phase, if extensive changes are required, it may be necessary to repeat the pilot test.

Finally, when all the changes to the questionnaire have been made, it should be carefully proof-read before being printed and distributed.

Summary of key points

Unless a questionnaire consists entirely of questions that have previously been used and validated in a very similar population and using the same mode of administration , it is advisable to carry out pre-tests and pilot tests.
Tests of this nature may help to indicate the need for re-wording of questions and/or response categories, altering the format of the questionnaire, or changing other aspects of the conduct of the survey. Time and resources for pre-testing and piloting should be built into the survey plan.
There are three basic types of pre-test and pilot – developmental trials, cognitive testing of questions and instruments, and rehearsal pilots. They are not interchangeable.
Cognitive testing aims to gain an understanding of how survey respondents react to the survey ‘package’, including covering letters and questionnaires. The two main techniques of cognitive testing are ‘think-aloud’ interviews and directed probing (‘debriefing interviews’).
Rehearsal pilots are used to gain an impression of how the main survey will work in ‘real life’, including how the survey procedures as a whole are likely to work, to give some feel for how well questions work (though most question defects may remain undetected), and the likely magnitude and pattern of response rates and non-response bias.

References

Brown TL, Decker DJ and Connelly NA. Response to mail surveys on resource-based recreation topics: A behavioral model and an empirical analysis. Leisure Sciences 1989;11; 9-110.

Dillman DA. Mail and telephone surveys: The total design method, New York: John Wiley and Sons, Inc, 1978.

Schwarz N, Bradburn NM and Sudman S. Thinking about answers: the application of cognitive processes to survey methodology. San Francisco: Jossey-Bass Publishers, 1996.

Schwarz N and Sudman S. Answering questions: methodology for determining cognitive and communicative processes in survey research. San Francisco: Jossey-Bass Publishers, 1996.

09 Pre-testing and piloting questionnaires