Choosing the method of data collection | Survey Methods Handbook

In this chapter…

This chapter compares the practical and methodological implications of common survey collection modes.

By the end of this chapter, you should be able to…

compare self-administered and interviewer-administered modes
select a mode that fits the population, topic, frame and resources
anticipate coverage, nonresponse and mode effects

In this chapter

By the end of this chapter, you should be able to:

list the principal modes of survey data collection;
describe the key factors to be taken into account in choosing a mode of administration;
describe the main advantages and disadvantages of face-to-face interviews, telephone interviews and self-completion (including postal) questionnaires.

Modes of administration for survey data collection

In quantitative survey research, the most versatile and commonly used modes of data collection are self-completion questionnaires, face-to-face interviews (sometimes termed ‘field interviews’) and telephone interviews. These can be used in various combinations, and all can be computer-assisted (the acronyms CASI, CAPI and CATI may be found in the survey literature to denote computer-assisted self-interviewing, personal interviewing and telephone interviewing respectively). Self-completion methods are very popular, largely because they are cheaper and require less specialised resources than any form of interviewing. Postal administration is probably the most common method of administering self-completion questionnaires, but paper-based self-completion questionnaires can also be administered directly to ‘captive audiences’, for example people in a waiting room or class room; this approach allows for some supervision and assistance of respondents. There is also increasing interest in and use of other forms of self-administration. Increased access to IT and greater IT literacy have stimulated interest in electronic delivery of questionnaires, by e-mail or over the internet. Fax delivery has also been explored. Multimedia delivery, including audio- and video-supported delivery, and computer-assisted self interviewing (CASI) are also growing in popularity.

Each mode of delivery poses its own challenges for the survey designer, though there may be commonalities. No one mode of data collection has been shown to be superior in all respects, as can be seen from Table 2. Rather, the most appropriate method is likely to depend on the study population, survey topic, and nature and complexity of data to be collected. Resource constraints will also influence the choice of method. In particular, given the statistical need in quantitative survey work to accumulate relatively large samples, interview approaches are likely to be too resource-intensive and expensive for many survey researchers. In this course and handbook, therefore, we concentrate on postal and other self-completion surveys.

Main advantages of self-completion questionnaires

The key advantages of self-completion questionnaires, especially postal surveys, over interview approaches (Mangione, 1995) are:-

Cost – postal surveys are relatively inexpensive. In interviewer-administered surveys, interviewers’ salaries and travel expenses must be paid. Geographical clustering (a two-stage sampling process in which geographical units – such as electoral wards – are initially sampled and then a sample of population units – such as households – are selected within each sampled geographical area) may be necessary to avoid large travel costs; this may result in less precision for a given sample size. By contrast, postal surveys can cover widely dispersed populations without increasing study costs. Similarly, in ‘captive audience’ self-completion surveys, data can be collected simultaneously from a large number of respondents.
Coverage – provided there is an appropriate sampling frame (this has been covered in Chapter 12), a large number of respondents, even where dispersed over a wide geographic area, can be surveyed in a relatively short time (by comparison with face-to-face interviews); this is especially true amongst those who are reluctant to be interviewed in person or on the telephone. Postal surveys may also be quicker than interviewer-administered surveys, although time must be allowed for late returns and follow-up attempts.
Less demanding of special resources – postal surveys are easier to implement and require fewer personnel (by comparison with all forms of interview survey) and minimal equipment (compared with telephone interviews and computer-assisted approaches).
Lack of interviewer effects – the respondent is “insulated from the expectations of the interviewer” (Mangione, 1995); the potential for interviewer-induced bias and error, discussed in the Further Reading file ‘Measurement and error in survey research’, is eliminated.
Respondent convenience – respondents can complete the questionnaire at a time that is convenient, and can take their time in answering; for these reasons, better thought-out answers may be given.
Provision of visual stimuli – interviews, especially telephone interviews, rely primarily on auditory input, whereas postal and other self-completion questionnaires provide visual input, which can be an advantage in presenting multiple-response type questions.
Ease of consulting appropriate information sources – because respondents are not under time pressure to respond, but can answer questions at a convenient time and location, it may be easier to look up any information required to answer questions in an accurate and complete manner (for example, to check pay slips for income details, or pill bottles for names of medication). However, this ability to consult documentary sources, or to ask other people for information, can also be a disadvantage, especially in respect of knowledge questions.
Minimisation of ‘history’ effects - it can generally be assumed that all subjects receive the mailed questionnaire at approximately the same time, therefore contextual ‘history’ effects (e.g. the influence of breaking news or of a campaign or product launch) which may influence subjects' experiences, opinions or attitudes are minimised for the total sample. This would not be true, however, if a proportion of the sample were away from home when the questionnaire arrived.
Context gives meaning to questions – within a questionnaire, meaning is inferred not just from each individual question in isolation, but from the context in which each question is posed. In a self-completion questionnaire, it is easier to review preceding and following questions, and to view questions and response categories as a coherent whole. Thus, it is argued, context effects (for example, due to ordering of questions and of response categories) tend to be less marked in self-completion questionnaires compared with interviewer-administered surveys. There is some empirical evidence to support this assertion. However, further investigations are needed, especially in respect of internet surveys, where only one screen is visible at a given time, and flicking back and forth may be more difficult than in a paper-based questionnaire.
Privacy – the greater anonymity afforded by self-completion questionnaires may reduce the risk of social desirability bias; self-completion approaches may be more suitable for surveys on embarrassing or sensitive topics.

Main disadvantages of self-completion questionnaires

The principal disadvantages of self-completion approaches are:-

Lack of availability of suitable and unbiased sampling frames – a pre-requisite for a postal survey is an explicit listing of members of the underlying population from which to draw the sample, and full and up-to-date contact details for those individuals. As discussed in Chapter 12, for certain populations, there may be no such list, or the information may be inaccurate, biased or in other ways flawed (though this weakness also applies to other modes of administration). There is a particular risk of bias at present for e-mail and internet surveys, particularly of general populations, since access to the internet and possession of an e-mail address is not uniform across the population.
Potentially lower response rates (with implications for precision of estimates derived from the achieved sample) – in an interviewer-administered survey, the burden of recording answers falls on the interviewer, while in a self-completion questionnaire this burden is transferred to the respondent. A skilful interviewer can engage the interest and attention of a potentially reluctant respondent, thus reducing non-participation and non-completion rates. As a general rule, therefore, response rates to interviewer-administered surveys are generally higher, and a greater volume of data can be collected from a given individual by this means. However, careful attention to questionnaire design and administration can yield highly satisfactory response rates to self-completion surveys (this will be discussed further in Chapter 6, 8 and 10).
Greater risk of non-response bias – with postal surveys, there is a greater likelihood that respondents may differ significantly from non-respondents (the most obvious bias being in favour of better-educated and more literate individuals), so estimates based on the former may be biased (Moser and Kalton, 1971). While non-response bias is an issue in all approaches, the problem is larger in postal surveys, where relatively lower response rates are common.
Poorer sample control – in a self-completion questionnaire, especially with postal, e-mail or internet delivery, there is no easy means of ensuring that the questionnaire is actually completed by the intended respondent, or that responses are not influenced by other household members or colleagues.

Greater risk of response error – in an interviewer-administered survey, the interviewer can clarify questions, prompt and probe for additional information. In a self-completion questionnaire, the questions and associated response categories and instructions must stand alone, and the risk of a respondent misreading or misunderstanding a question is greater. Skip patterns, branching and filter questions are generally handled more easily by interviewers than by respondents to self-completion questionnaires. “Primacy” effects, whereby respondents select the first response that seems applicable, without considering the full range of alternatives, may be more common in self-completion questionnaires (Krosnick, 2000).
Less complete responses – again, because the burden of recording answers falls on the respondent, responses in self-completion questionnaires are typically briefer and less detailed than in interviewer-administered surveys. Moreover, there is often a subtle bias with respect to education and literacy levels, with more articulate, more literate and better educated respondents providing fuller answers.
Inappropriateness for certain populations and sub-groups – clearly self-completion questionnaires are not a feasible means of data collection for illiterate or pre-literate populations. In multi-cultural settings, significant resources required to translate and validate questionnaires into different languages. For languages for which there is no written representation, this approach if obviously very difficult.

Summary of key points

The main modes of administration for survey data collection are face-to-face interviews, telephone interviews and self-completion questionnaires (including postal surveys, ‘captive audience’ surveys, e-mail and internet surveys). All modes may be computer-assisted.
In choosing a method, a range of factors need to be taken into account. These include: sample control; likely response rates; quantity of data to be collected; potential threats to validity and reliability of data; potential sources and magnitude of bias; resource implications.
Each method has its strengths and weaknesses, and no single method is superior in all respects. The choice of method needs to be made on a survey-by-survey basis, balancing aspects of quality and cost.

Table 2 Advantages and disadvantages of different modes of questionnaire administration (adapted from De Vaus, 1991, after Dillman, 1978)

	Face-to-face interviews	Telephone interviews	Postal questionnaires
*Response rates*
General population samples	Usually best	Usually lower than face-to-face	Poor to good
Special population samples	Usually good	Satisfactory to best	Satisfactory to good
*Representative samples*
Avoidance of refusal bias	Depends on good interviewer technique	Depends on good interviewer technique	Poor
Control over who completes the questionnaire	Good	Moderate	Poor to good
Gaining access to a named selected person	Good	Good for those with telephones	Poor to good
Locating the named selected person	Good	Good	Good
*Ability to handle:*
Long questionnaires	Good	Moderate	Satisfactory to poor
Complex questions	Good	Moderate	Moderate to poor
Boring questions	Good	Moderate	Poor
Item non-response	Good	Good	Moderate
Filter questions	Good	Good	Moderate to poor
Question sequence control	Good	Good	Poor
Open-ended questions	Good	Good	Poor
*Quality of answers*
Minimise social desirability responses	Poor	Moderate	Satisfactory
*Ability to avoid distortion due to:*
Interviewer characteristics	Poor	Moderate	Good
Interviewer’s opinions	Moderate	Moderate	Good
Influence of other people	Moderate	Good	Poor
Allows opportunities to consult	Moderate	Poor	Good
*Implementing the survey*
Ease of finding suitable staff	Poor	Moderate	Good
Speed	Poor	Good	Poor
Cost	Poor	Moderate	Good

References

de Vaus DA. Surveys in social research, 3rd edition. London: UCL Press Ltd, 1991.

Dillman DA. Mail and telephone surveys: The total design method, New York: John Wiley and Sons, Inc, 1978.

Krosnick JA. The threat of satisficing in surveys: the shortcuts respondents take in answeringquestions. Survey Methods Centre Newsletter 2000, 20, 4-8.

Mangione TW. Mail surveys - improving the quality. Thousand Oaks: Sage Publications (Applied Social Research Methods Series, Volume 40), 1995.

Moser CA and Kalton G. Survey methods in social investigation. London: Gower, 1971.

03 Choosing the method of data collection