Survey Methods Handbook · Chapter 12 of 15

12 Sampling for survey research

In this chapter…

This chapter connects the target population, sampling frame, selection method and achievable precision.

By the end of this chapter, you should be able to…

  • define the population and sampling units precisely
  • compare probability and non-probability selection approaches
  • recognise coverage, selection, clustering and weighting implications

In this chapter

By the end of this chapter, you should be able to:-

  • state why surveys are generally based on samples rather than complete censuses;
  • distinguish between probability and non-probability sampling methods, and demonstrate why the former are to be preferred in quantitative social surveys;
  • identify the factors influencing the precision of estimates based on samples;
  • define what is meant by an ‘adequate sample’;
  • state what is meant by ‘target population’ and ‘sampling frame’ and identify the features of an ideal sampling frame;
  • explain how sampling bias may be minimised;
  • identify what factors need to be taken into account in determining sample size, and carry out simple sample size calculations.

Why do we sample?

Surveys aim to draw conclusions (make inferences) about defined populations of units (individuals, households, institutions etc). Sampling is an efficient basis for doing this, offering enormous savings of time and resources compared with a census of all population members. For many populations a complete census would in any case be impracticable. When the data collected from a sample are summarised statistically, the prevalence and incidence of attributes, attitudes, behaviour and so on, in the population can be estimated.

While offering these important advantages, estimates based on samples are always subject to sampling variability (variance). This variability occurs even if samples drawn to the same sample design are repeatedly drawn from the same population (assuming no population change in the meantime). However, provided that samples are drawn using a procedure that gives each member of the population a calculable probability of selection (probability or random sampling), the extent of this variability in estimates can be calculated. That in turn enables us to draw confidence limits around any estimate, giving a range of values within which the true population value is likely to fall. For example, we can say that there is a 95% probability that the true value for the population lies between value X and value Y.  Use of non-probability sampling methods, on the other hand, is prone to cause systematic biases in the results that cannot be quantified or allowed for. When using non-probability methods, it is only possible to calculate confidence limits around estimates by simply assuming that the sampling method is equivalent to probability sampling (which it is not).

Factors influencing the precision of sample-based estimates

For a sample-based estimate to be useful, it needs to be sufficiently precise for the purposes for which it is to be used. Very few applications need absolute precision and often users can tolerate estimates being fairly approximate. Their view of what can be tolerated is generally strongly influenced by the cost of improving precision. The precision of estimates is determined mainly by two factors, of which the first is outside the control of the sample designer, but the second is in principle within his or her control.

Population variance

The first factor influencing the precision of estimates is the degree of variability within the population sampled (the population variance). Consider a survey conducted to estimate the average height in centimetres of members of a large population. If all members of the population are in fact exactly the same height (no variability), samples consisting of just one person will always give the same (correct) answer. On the other hand if the large population consists of a collection of men, women, Swedes, Japanese, pygmies and so on the variability is high. Then even random samples of 1000 persons drawn from it will not always give an estimate of average height that coincides exactly with the true average height of the population.

In real survey life, the situation is more complicated, since no survey is done to estimate just one feature (or parameter) of the population, such as average height.  It will probably be intended also to measure quite different features, such as population members’ mean age, or what their ethnic group is. Unfortunately not all characteristics have the same population variance, so a given sample design and size will provide differing degrees of precision for different estimates.


Sample size

Underlying the example just given is the idea that, for a variable population, we may need quite a large sample to obtain precise estimates (i.e. narrow confidence intervals). So the second determinant of the precision of sample-based estimates is the number of population members included in the sample (i.e. the sample size).  If we know (or can make a good guess at) the population variance of (say) height and we can specify how precise we want our estimate of mean height for the population to be, we can use a formula to calculate how large a sample will be required.

In situations where we are taking a relatively small sample from a large population, it is the absolute size of the sample that matters, not the proportion of the population that is included in the sample. Thus for example, to provide results of the same level of precision for the population of Scotland and the population of England, equal-sized samples are generally required, despite the population of England being much greater than the population of Scotland. This is not what most people new to sampling would assume!  In practice, the proportion of the population included in the sample (the sampling fraction) only starts to matter when it is as high as 1 in 10 (10%) or more. Where that is the case, a simple finite population correction enables us to adjust our estimate of sampling precision (in the remainder of this chapter, we shall assume that there is no need to apply a finite population correction).

For simple random sampling (see below) and most other sample designs,  therefore, the larger the sample, the more precise will be estimates based on it. Unfortunately, precision does not increase directly in proportion to sample size, but only with the square root of sample size (see Table 13). So, by taking a sample four times as large, we only increase the precision (reduce the size of the standard error and thereby narrow the confidence limits around a sample-based estimate) by a factor of two.  But for a sample four times the size, the cost of data collection and other operations will increase by a factor of around four (since most of the costs are direct, and economies of scale do not apply). Therefore survey researchers and research sponsors need to think hard about the best trade-off point between cost and sample size.


Table 13 Sample size and the precision of population estimates

Sample-based estimate of % of smokers in group

Sample size for group of interest

Range within which true value lies (95% confidence interval)

30%

10

1.0-59.0% (range 58)

30%

100

20.8-39.2% (range 18)

30%

1000

27.1-32.9% (range 6)

30%

10,000

29.1-30.9% (range 2)

 

So far we have implicitly been talking about estimates based on the total sample obtained and referring to the whole population sampled.  In most surveys, however, estimates based on the whole sample are certainly not the only estimates required. Typically this happens because much interest focuses on differences between population sub-groups, which are of course represented by sample sub-groups. Comparisons may be required between just two sub-groups (say men and women) or between more sub-groups (say persons falling into each of 4 age groups, or men and women separately falling into each of four age groups, giving 8 sub-groups in all).

Though all the groups will in practice be covered by the same survey operation, this desire to compare groups and to make inferences about specific sub-groups is statistically equivalent to requiring parallel surveys of each of the population sub-groups of interest. All the principles discussed above then apply to the sample size required in each sub-group. So, if estimates are required for N different sub-groups at the level of precision originally specified for estimates for the whole population, then the total sample size required will (as a first approximation) be something like N times as large. If the numbers falling into different subgroups are unequal, an even larger sample size may be needed, so as to ensure sufficient numbers in the smallest group to provide the required precision of estimates based on that sub-group.

This may be discouraging news for those hoping to get useful estimates for many minority population sub-groups from the same modest-sized sample selected with equal probability. However, there are some countervailing factors. First, within-subgroup variability is in some cases (e.g. age by sex sub-groups) likely to be less than is variability across the whole sample, so that the precision given by a given sample size will increase in the case of estimates based on sub-groups.

Second, it is often possible to devise probability sampling designs that deliberately over-represent particular small but important sub-groups (though, for a fixed total cost, such over-representation of these specific sub-groups will have to be at the expense of other sub-groups). In general, however, it is true to say that analysts of the survey data will often be faced with the dilemma that the detailed analyses, involving comparison of those sub-group estimates which they consider would be most meaningful and revealing are in fact disabled by the fact that sample sub-group numbers are small, so that the resulting estimates are too imprecise to be useful (i.e. their study is under-powered to detect differences between sub-groups).

Definition of an adequate sample

An adequate sample should satisfy the following requirements:-

  • The sample should be drawn using a method which gives every member of the target population a calculable and non-zero chance of being selected; this ensures that it will be unbiased in its representation of the population and that confidence intervals can be calculated around estimates. But note that bias can creep back in through non-response.
  • The sample must be sufficiently large to make inferences about the underlying population and about important population subgroups that have acceptable margins of random variability (see preceding section).

To achieve an adequate sample, the survey researcher needs to:-

  • state the objectives of the survey clearly and precisely;
  • define the population to be surveyed explicitly, in terms of inclusion and exclusion criteria;
  • choose a sampling frame that is appropriate to the defined study population;
  • specify rigorous and objective sample selection methods (preferably probability sampling methods);
  • determine the required achieved sample size, taking into account the likely variation in the characteristics of interest in the population, the size of differences between sub-groups that s/he wishes to be able to detect, and the level of confidence s/he wishes to have in estimates of population values derived from the sample;
  • contact more than the required sample, to allow for the losses that are expected due to the occurrence of ineligibles (those who do not fit population criteria; those who have died) and to non-response (for example, non-contacts, refusals).

Some sampling terms

Population - the complete set of units from which a sample is selected and to which the sample-based results will apply.

Units or elements – the elements of which the population is composed, some of which are selected as the sample:-

  • units could be patients, or members of the public, or hospitals, or shops, but also events such as births or attendances at a clinic or visits to a leisure centre; sampling of time-points is also possible;
  • a precise operational definition of the population and its constituent units is crucial (i.e. explicit eligibility – inclusion and exclusion – criteria);
  • there could be different sampling units at different stages of selection (for example, the first stage could involve sampling of schools within a defined geographical area, followed by sampling of classes within the selected schools, and finally children within those classes);
  • the sampling unit may or may not be identical with the survey respondent (for example, the sampling unit could be the seat in the waiting area, but the survey respondent would be the patient occupying that seat; or the sampling unit could be the household, with data being collected at the level of the household as a whole and in respect of individual members of that household).

Sampling frame – a listing of all the units (elements) in the population that are eligible to be sampled:-

  • the sampling frame should ideally correspond exactly to the target population;
  • in practice, the sampling frame used may itself be a sample (adequate or inadequate) of the real target population – for example,  ‘List of elderly people registered with these general practice’ as a substitute for ‘List of all elderly persons residing in this PCT area’;
  • populations and sampling frames may be implicit because no physical listing exists – for example, patients attending an STD clinic in a given time period, considered as a sample from the population who have attended the clinic in the recent past and (barring major changes) will attend in the immediate future.

Defining the target population

The definition of the target population should relate explicitly to the research aims.  It should specify inclusion criteria - for example, adults using the swimming pool at X leisure centre - and exclusion criteria - for example: those attending for a second or subsequent time during the sampling period (although these should be included if a sample of visits, rather than a sample of users, is required); those accompanying children but not swimming themselves. The definition usually needs to include a time frame (for example, those attending during the months of June and July).

What is required in a sampling frame

The ideal sampling frame is a listing of population units that:-

  • matches the target population one-to-one;
  • contains complete, accurate and up-to-date identifying and tracing information for each unit;
  • is comprehensive (has no omissions);
  • has one entry only for each eligible unit;
  • contains no ineligible units;
  • contains information about each unit useful for sample stratification  (for example, age, sex etc.);
  • ideally, can be manipulated by computer;
  • is accessible and cleared for use in research (see the provisions of the Data Protection Act (1998)).  Note also that research ethics committees and Caldicott guardians are increasingly protective about the use of age-sex and morbidity registers, prescribing records and other clinical databases as a means of identifying and contacting potential research study participants.  Increasingly, they are insisting on a two-stage process, whereby those on the register are first contacted by the ‘responsible clinician’ and asked if they are willing to have their contact details released to the researchers (often on an ‘opt in’ basis).  Questionnaires may only be sent to those who agree to contact.  This two-stage approach obviously increases the potential for sample composition and non-response bias, since there has to be a positive response at two time points (see Angus et al., 2003);

In practice few sampling frames exactly satisfy all of these criteria. Because the sampling frame is so fundamental to the enquiry and because defects can affect the validity of the sample, it is often worth spending time and effort on improving it.


 

Sampling frames for the general population of households and persons

Many surveys require samples of households and/or persons selected from the general population. This presents problems different from those encountered when  sampling from lists (sampling frames) of named individuals.

To a researcher looking for a complete sampling frame of individuals in Great Britain (or geographically defined parts of Great Britain) the first idea that occurred might be the Census of Population. The Census does indeed aim to list all individuals present in Great Britain on Census Day, but there are two fatal objections to its use as a sampling frame. The first is that the Census is carried out only once every ten years and is not updated in between. The second is that stringent Census legislation forbids the release of any information about individuals to persons outside the Census Offices.

The next idea might be to use the Electoral Register, which lists the names and addresses of individuals aged 18 and over who are eligible to vote. It is updated annually by local authorities through a canvass of addresses (though the time lag between canvass and public availability of the list is more like 18 months). The Electoral Register has been traditionally used as a sampling frame for both households and individuals. Unfortunately, however, it has drawbacks that have recently become more severe.  The first is that only adults who are both qualified to and choose to register to vote are listed. Research done several years ago by the Office for National Statistics showed that the coverage of all individuals who should be registered was then about 94%. That does not sound too bad, but unfortunately non-registration is concentrated in groups that are often of prime interest to social researchers (one of many applications of Sod’s Law to surveys). The omissions include recent immigrants (to the area, if not the country) and foreign nationals (some of whom are not actually qualified to vote), certain poorly integrated ethnic minorities, young people living alone or in a household consisting entirely of young people, the homeless and rootless and individuals who reach the age of 18 during the life of an Electoral Register.  The biggest problem, however, is that recent legislation has led to the creation of two versions of the Electoral Register. The full version is used at elections, but the version that is in the public domain (and now available in electronic format) has the names of all individuals who do not wish their names and addresses to be publicly available removed. It can be assumed that those who take that option will be a significant proportion of all voters, will be higher in urban areas and will not be a random selection of all voters. This effectively disables the Electoral Register  as a serious sampling frame for individuals, ending a long era of survey practice; this has caused much concern and rethinking on the part of survey organisations.  A particular regret about this change has been expressed by those who conduct mail surveys; for them, a major attraction of the Electoral Register was that it contains the names of individuals, thereby facilitating personalisation of approach. In an era of ever-increasing “junk mail”, rates of response to mail surveys of the general population (always a weak point) are likely to be lower if a communication has to be addressed to “The Occupier” of an address, rather than to a named resident.

Another idea that might occur to a creative researcher is that almost every resident of Great Britain can now be reached by telephone. Therefore one might select a sample of named persons from the public telephone registers or directories (regularly updated copies of the registers are now held by various agencies, including in electronic format). In principle, this would be a way of sampling individuals for a mail as well as for a telephone interview survey, though addresses listed in telephone directories are not always complete and are generally not post-coded.  However, this approach is not without its problems also. Household ownership of fixed-line telephones did indeed peak at around 97% some years ago, but is now falling again as more users come to rely on mobile phones only. The use of public telephone directories as a sampling frame for the general public is in any case ruled out by the facts that something like a third of private numbers are now ex-directory (unlisted) and that unlisted telephone owners differ on average in important characteristics from owners whose numbers are listed (i.e. this represents an important source of potential sampling frame bias). 

This problem could until very recently be got around by a method known as random digit dialling (RDD). This requires information about the numeric ranges of numbers that are “in use” in a particular geographic area; moreover, repeated dialling of theoretically “in-range” numbers is required not just because there is often no reply, but also because a surprisingly high proportion of “in range” numbers are not domestic and/or in use.  However, with the increase in mobile telephony, there are also great technical problems in drawing RDD samples of individuals that include both fixed and mobile telephone users with equal probability (many users have both types of phone, but we do not wish to give them twice the chance of selection). Of course, those who intend to collect their data by mail or by face-to-face interviewing, rather than by telephone interviewing, could not in any case afford to do the enormous amount of dialling and redialling needed to even establish a sample.

The most widely used solution to all of these problems starts by identifying and sampling not individuals but rather addresses. The Postcode Address File (Small Users) (PAF) has very high coverage of addresses that receive modest amounts of mail and are therefore likely to be residential (by definition, it excludes large users receiving more than 50 items per day). The PAF is fully computerised, regularly updated by the Post Office, and copies can be purchased (electronic versions also). It has a geographically hierarchical structure, with addresses being listed within postcodes (each containing about 16 addresses on average), postcodes being grouped within postal sectors (about 2600 addresses each, in average) and so on. The great majority of PAF addresses turn out to be the residence of just one household (including one-person households). The remainder may cover accommodation occupied by several different households.  Of course, the homeless are excluded and those living in institutions may also be missed (if the place of residence receives in excess of 50 items of mail per day).

Companies specialising in “geo-demographic” data management hold copies of the PAF which they enhance, for example by adding information from the last Census or other sources about the average characteristics of small areas (such as postal sectors). They also have software that enables them (for a fee) to draw samples to user-supplied specifications. Virtually all major government household surveys rely on PAF address samples drawn in this way. However, once again there are some drawbacks. PAF Small Users contains about 12% of addresses, on average, that are non-domestic (small businesses etc), but unfortunately these cannot be reliably identified other than by investigation on the spot. In the case of a postal survey, therefore, a chunk of the non-returns are likely to be due to the ineligibility of the address. Also, PAF addresses do not include names of residents or any information about the address other than its geographical location, so questionnaires cannot be targeted to a specific, named individual.

In field interview surveys in which the PAF is used as the sampling frame, an interviewer visits the address, makes contact with a resident, asks about and lists households and individuals living at the address and uses a random procedure to select one household or one individual at that address (or in some cases they interview all adult residents). A number of recalls may be needed to achieve this outcome and of course the selected person may decline to take part.  Skilled field interviewers are able in part to overcome the disadvantage of not having a named person to ask for, but in the case of postal surveys researchers have to resort to addressing the questionnaire package to “The Occupier”. Because of the heterogeneity of the general population it is hard to find ways of appealing to target respondents for co-operation, though in a local survey some message such as “Attention: This package contains documents that could affect the lives of all who live in Middleton” and perhaps an envelope bearing the sponsor’s logo may have some favourable effect.


 

Avoiding sampling bias

It is important to ask:-

  • To what population (for example, professionals, patients, club members, institutions) do we intend our results should apply?
  • Does the sampling frame and selection procedure that we use to select our sample give all members of the in-scope population a known chance of selection? If not, how many and what types of population units are likely to be excluded?
  • Can we ensure, or reasonably assume, that the units to which we have access are a random sample of all  in-scope population units?

If the answer to any of the preceding three questions is ‘No’, we have a sampling frame bias (and it should be clear from the preceding section that these criticisms apply in greater or lesser extent to all the potential sampling frames for the general population).

It is also important to ask:-

  • do all members of the population have a known probability of selection (for example, an equal chance of being chosen)?
  • are some population units listed or available for selection several times?  This could arise if some individuals appear more than once on the population listing or visit the facility several times during sampling period.
  • if some population units are listed more than once, there is still bias even if they are not selected more than once (or at all). This is because population units of types that tend to be listed several times will be over-represented.

It is important to remember, first, that all statistical inference to population and hypothesis testing assume random selection.  We need to query whether our chosen selection process is truly random.  It could be non-random because of: concessions to convenience (for example, going for the ‘easy-to-find’ cases - in an interview survey, those people who are at home during daylight hours); a desire to include ‘interesting’ cases (for example, ‘We must include Mrs Miggins, her stories are fascinating’); a desire to exclude difficult or uninteresting cases (for example, ‘Let’s leave out Mr Meldrew, he’s so grumpy!’).  It is important to recognise that human beings cannot choose a random sample by judgement – there has to be a random selection procedure.

Secondly, the most perfectly constructed sample can be largely invalidated, for the purposes of drawing conclusions, if there is gross and differential non-response at the data collection stage. If sample members of a particular kind are less likely than average to respond, that is equivalent to under-sampling them (and relatively over-sampling other groups) in an uncontrolled way.

Sample selection methods

Probability (random) sampling methods include:-

  • simple random sampling;
  • systematic random sampling;
  • stratified random sampling;
  • multi-stage (clustered) random sampling.

Non-probability methods include:-

  • convenience sampling;
  • snowball  sampling;
  • quota sampling;
  • For a more detailed discussion, see Moser and Kalton (1971).

Probability sampling: selection

For probability sampling, each unit in the target population must have a calculable, non-zero probability of being selected.

  • Simple random sampling generally uses a paper-based random number table or a computerised random selection procedure. Each selection is made independently and each unit has equal probability of being selected.
  • Systematic random sampling uses a random start in the population listing, then selects every nth unit; this has the merit of being easier to implement when using a list as a sampling frame. It may lead to bias if the list is organised in some systematic manner. For example, in sampling staff from a series of departments, where each department has around 20 staff and the staff are listed in order of seniority within department, a low random start and a sampling interval of 10 will tend always to select one very senior staff member and one from the middle of the seniority list and to under-sample the remainder. Such risks can generally be removed through paying attention to how the list was compiled. With the common alphabetically or geographically ordered lists, there is generally no problem of this kind.
  • Stratified random sampling controls the composition of the sample in relevant respect(s); it can be applied at any stage of selection (but only if information about all population units is available). Units are split into subsets (strata) likely to differ in terms of what the survey aims to measure (for example, defined by age and / or gender) and separate random samples are drawn within each stratum. For example, if the population contains men and women (who can be pre-identified) and if the results for men are likely to differ from the results for women, then something is gained by listing and sampling the two sexes as separate strata, so as to predetermine the number of men and the number of women selected.  Provided the probabilities of selection are known, estimates from strata can be combined to give estimates for the total population.  Since men and women are selected randomly, this is still a probability sample. Stratification can significantly improve the precision of estimates, but its effect is, in most applications, small relative to that of sample size.
  • Multi-stage (clustered) random sampling is done in steps, using different population units at each stage. For example, in a two-stage random sampling procedure, the first stage units (for example, areas, hospitals, schools) are randomly selected from an appropriate sampling frame at stage 1; and the second (final) stage units (patients, pupils) are selected at stage 2 from within selected stage 1 units. Thus the final-stage units are said to be clustered within first stage units. Multi-stage sampling is used to reduce the amount of administrative, sample selection and data collection effort required by the survey. The results of multi-stage sampling will be unbiased, but are usually less precise (i.e. have wider confidence intervals) than those of single-stage sampling.

Calculating sample size

In research studies where the survey is embedded within a some particular study design (for example, a randomised controlled trial or other evaluative study design), the sample size will generally be based on a calculation of the statistical power required to test specified hypotheses.  If a survey is being carried out as a separate operation, sample size calculations should be based on the desired precision of the most important estimates to be made from study findings. 

Estimating required sample size for major or complex studies is fundamental and requires a knowledge of  sampling statistics, so  consult a trained statistician if you can. However, do not expect the statistician to simply produce a number, without any help from you.

For a sample survey, in which the main aim is to make parameter estimates, the statistician will want to know:-

  • What is the parameter to be estimated? (for example, ‘percentage of people who smoke’ or ‘mean number of cigarettes smoked per day’)?
  • Within what margin of error do you wish to estimate the parameter of interest (for example, for a percentage, the acceptable margin of error might be ± 5%)?
  • How confident do you want to be in these parameter estimates – what size of confidence interval do you want to have (this is usually set at 95% - 1 chance in 20 of ‘being wrong’, or 99% - 1 chance in 100 of ‘being wrong’)?
  • How variable is what you are estimating (situation at a point in time, change over time) likely to be across the population? For example:
  • for a yes/no dichotomy, what is the estimated proportion with that attribute (in the current example, proportion who smoke);
  • for a continuous variable, what is the population standard deviation of what is being measured (in the smoking example, the number of cigarettes smoked per week)
  • What is the size of the population from which the sample is to be drawn (as we have noted above, this question is most relevant when the population is relatively small).

This means that you and your adviser will have to guesstimate, in order to set a sample size, some of the very things you hoped to measure! Fortunately, it is usually possible to make the requisite guesses to a sufficient degree of precision, for example on the basis of published data, previous research, or pilot work.  For many of the variables likely to be of interest possible sources of information are the published results of the decennial Census of Population, or the annual reports and tables produced by the General Household Survey, the Health Surveys for England Department of Health), Scotland (Scottish Office) and Wales (Welsh Office) and so on. In the absence of any relevant data, and where the aim is to estimate the proportion in the population, use an estimate of 0.5 (i.e. 50%), since this produces the most conservative estimates of sample size requirement.

The formula for working out the target achieved sample size for estimating a proportion (p) to within a given margin of error (d), with 95% confidence, is given by:-

\[ N = \frac{p(1-p)z^2}{d^2} \]

where

\(p\) is the estimated proportion in the population.

\(z = 1.96\) for 95% confidence, or \(z = 2.58\) for 99% confidence.

\(d\) is the acceptable margin of error, expressed as a proportion. A common choice is \(\pm 5\%\), or \(0.05\).

For example, if we wish to estimate the proportion in the population as 0.3 (i.e. 30%), with a margin of error of ± .05 (i.e. 5%, therefore a 95% confidence interval of 25% to 35%), and 95% confidence, the target sample size is

\[ N = \frac{0.3 \times 0.7 \times 1.96^2}{0.05^2} \approx 323 \]

Bear in mind that this is the desired achieved sample size.  Since a response rate of 100% is unlikely, in excess of 323 individuals will have to be approached to yield this sample size. A further guesstimate – this time of the likely response rate – is required.  Let us say that based on previous surveys in a similar population and using the same mode of data collection we expect a response rate of 70%.  To yield an achieved sample size of 323 we therefore need to contact \(323 \div 0.7 \approx 462\) individuals.

In more sophisticated study designs, for example where the aim is to compare across two groups, calculations become more complicated.  The questions to be addressed in such a situation are:-

  • What is your null hypothesis? (for example, ‘no difference in the symptoms experienced between two groups’)?
  • How confident do you want to be in accepting or rejecting the null hypothesis? (usually 95% or 99%), or put another way, what is your desired significance level (usually 5% or 1%)?
  • How certain do you want to be of detecting for example, a real (population) difference of (say) 10% in symptom prevalence, or put another way, what is your desired power (usually 80% or 90%)?
  • How variable is what you are estimating (situation at a point in time, change over time) likely to be across the population?
  • Sample size / power calculations for these more sophisticated designs will be covered in the Medical Statistics module.

Summary of key points

  • Sampling is an efficient way for making inferences about the population of interest, offering significant savings of time and resources by comparison with taking a full census. (Where the population of interest is finite and small, a census may be feasible.)
  • Estimates based on samples are always subject to sampling variability.  The precision of sample-based estimates is influenced by population variance (beyond the control of the survey researcher) and sample size (which the survey researcher can determine).
  • For simple random samples, and most other probability-based sampling designs, the precision of the sample is proportionate to the square root of the sample size.
  • An adequate sample should be selected using a probability-based method which gives every member of the underlying population a calculable and non-zero chance of being selected, and should be sufficiently large to allow inferences to be made within a stated, tolerable margin of error.
  • Implementation of a sampling protocol requires a sampling frame – a list of all the units in the underlying population – that is accurate, up-to-date, comprehensive, contains no duplicate or ineligible units and is accessible and cleared for use in research.
  • Many of the sampling frames which might be considered for surveys of the general population are flawed in some way.
  • To avoid sampling bias, a probability-based method, involving random selection, and an accurate sampling frame are required.
  • Standard formulae for the calculation of sample size exist, involving a specification of the parameter to be estimated and the estimated variability of this variable, the margin of error that will be tolerated and the degree of confidence desired for the parameter estimates.

Further reading

Fowler FJ Junior (1993). Survey research methods (2nd edition). Newbury Park: Sage Publications. (Applied Social Research Methods Series - Volume 1) (Chapter 2).

 


 


Current guidance and methodological literature for 2026–27:

References

Angus V, Entwhistle VA, Emslie MJ, Walker KA and Andrew JA (2003). The requirement for prior consent to participate on survey response rates: a population-based survey in Grampian. BMC Health Services Research, 31, 21-30.  Downloadable from https://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-3-21

 

Moser CA and Kalton G.  Survey methods in social investigation. London: Gower, 1971.