Not all sampling methods are equally trustworthy. A random sample gives every member of a population a fair, known chance of being selected, helping results genuinely represent the wider population. Bias creeps in when a sample isn't random — like only surveying people who happen to be easy to reach, or wording a question in a way that nudges people toward a particular answer. Media reports of surveys should be read critically: checking how the data was actually collected, how large and representative the sample was, and whether the chosen way of presenting the results (which graph, which statistic) might be supporting a particular point of view rather than neutrally reporting the findings.
Example
A company surveys "customer satisfaction" only by emailing people who already opted into a loyalty program (likely to be their happiest customers) rather than a random sample of all customers — the result will be biased toward positive feedback, not because anyone lied, but because of who was asked in the first place.
Key terms
Random sample:
A sample where every member of a population has a fair, known chance of selection.
Sampling bias:
A systematic skew in results caused by an unrepresentative sampling method.
Questions
1. A random sample gives:
Every member of a population a fair chance of selection
Only convenient people a chance of selection
No one any chance of selection
Only the researcher a chance of selection
2. Sampling bias means:
A systematic skew caused by an unrepresentative method
A perfectly fair, random result
Something that never happens in real surveys
A type of statistic only
3. Surveying only people who are easy to reach can:
Create a biased, unrepresentative sample
Always produce a perfectly representative sample
Have no effect on results
Guarantee accuracy
4. A leading question is one that:
Nudges people toward a particular answer
Is completely neutral
Has no effect on responses
Is always illegal to ask
5. When reading a survey reported in the media, it helps to check:
How the data was collected and how large the sample was
Only the headline
Nothing beyond the conclusion
Only the font used
6. Choice of graph or statistic in a report can:
Support a particular point of view
Never influence how data is perceived
Only ever be completely neutral
Have no connection to how results are presented
7. A company surveying only its loyalty program members about satisfaction is likely to get:
A biased, more positive result than the full customer base
A perfectly representative result
A completely negative result
No result at all
8. Why might surveying only people who volunteer to respond (rather than a random sample) create a biased result?
People who choose to respond may have systematically different views or experiences than those who don't
Volunteers and randomly selected people always have identical views
Voluntary response sampling always produces the most representative results
Who volunteers to respond has no bearing on survey outcomes
9. Why might a question worded "Don't you agree this policy is a good idea?" produce biased results?
It leads respondents toward a particular answer rather than asking neutrally
This wording has no effect on how people respond
Leading questions always produce more accurate results
Question wording never affects survey outcomes
10. Why might a small sample size make a survey's results less trustworthy, even if the sampling method was genuinely random?
A small sample can vary considerably from the true population value just by chance
Small samples are always exactly as reliable as large ones
Sample size has no bearing on how trustworthy a result is
A small random sample always perfectly represents the population
11. Why might a news article choosing to display data as a dramatic bar graph rather than a plain table shape how a reader interprets the same numbers?
Visual presentation choices can emphasise or downplay differences, shaping the reader's perception even with identical underlying data
The way data is visually presented never has any effect on how it is perceived
A bar graph and a table always convey identical impressions to every reader
Presentation choices are always completely neutral with no influence on interpretation
12. Why might an online poll on a specific website (like a sports fan forum) not accurately represent broader public opinion?
The people who visit and respond on that particular website are unlikely to be representative of the general population
Online polls on any website are always exactly as representative as a national random sample
Website-based polls always accurately represent the entire population
Where a poll is conducted has no bearing on how representative its results are
13. Why is checking the sample size and method behind a reported statistic (like "70% of people agree") an important critical thinking step?
A statistic's reliability depends heavily on how it was collected, not just the number reported
A reported statistic is always accurate and trustworthy regardless of its source
Checking sample size and method never changes how a statistic should be interpreted
Method and sample size are irrelevant once a percentage has been calculated
14. Why might a genuinely random sample of just 30 people still be more trustworthy than a biased sample of 3,000 people?
A random sample, even if small, avoids systematic skew, while a large biased sample can be consistently wrong in the same direction
Sample size always matters more than whether the sampling method is biased
A larger sample is always automatically more trustworthy regardless of bias
Bias in a sample never actually affects the trustworthiness of results
15. Why might a company reporting survey results choose to highlight only the specific statistic that best supports their product, even if other findings from the same survey were less favourable?
Selectively reporting favourable findings can mislead an audience about the survey's complete, genuine results
Companies are legally required to report every single finding from any survey they conduct
Selective reporting of favourable results never affects how an audience perceives a product
All companies always report every finding from a survey with complete neutrality
16. Why might understanding sampling and bias be an important skill for evaluating claims made in advertising, not just formal research?
Advertisements may cite surveys or statistics that were collected using biased or unrepresentative methods to support their claims
Advertising claims never rely on any survey or statistical evidence
Sampling and bias concepts only apply to formal academic or scientific research
Advertisements are always required to use genuinely random, unbiased sampling
17. Why might a well-designed random sample of a modest size sometimes provide more useful, trustworthy information than an enormous but self-selected online survey?
Representativeness (avoiding bias) matters more for genuine accuracy than sheer sample size alone
A self-selected sample is always exactly as representative as a genuinely random sample
Sample size is always the single most important factor determining accuracy, more than method
Self-selected surveys are never affected by any form of bias
18. A survey asks "How much do you love our new product?" with answer options only ranging from "quite a lot" to "extremely." What kind of bias does this illustrate?
The answer options themselves are leading, since there is no option to express a neutral or negative view
This survey design is completely free of bias since it asks a direct question
Answer option design never introduces any bias into survey results
This wording only affects who chooses to respond, not how they respond
19. Why might a survey conducted immediately after a dramatic news event produce different results than the same survey conducted a month later?
Timing can influence how people are currently feeling or thinking about an issue, independent of any change in sampling method
The timing of a survey never has any influence on how people respond
Survey results are always completely stable regardless of when they are conducted
Only the sampling method, never timing, can ever affect survey results
20. Why might government census data (surveying the entire population) be used as a benchmark to check whether a smaller survey's sample was genuinely representative?
Comparing a sample's demographic makeup against known, complete population data can reveal whether the sample over- or under-represents certain groups
Census data can never be used to evaluate the representativeness of a smaller survey
A smaller survey's representativeness can never actually be checked against any other data
Census data and smaller survey data are always completely unrelated to each other
21. Understanding sampling, surveys and bias mainly helps you to:
Critically evaluate how data was collected and whether it genuinely represents the population being studied
Assume every reported survey result is automatically accurate and representative
Ignore how a sample was selected when interpreting results
Treat all sample sizes and methods as equally trustworthy
Answer key (parent copy)
1. Every member of a population a fair chance of selection
2. A systematic skew caused by an unrepresentative method
3. Create a biased, unrepresentative sample
4. Nudges people toward a particular answer
5. How the data was collected and how large the sample was
6. Support a particular point of view
7. A biased, more positive result than the full customer base
8. People who choose to respond may have systematically different views or experiences than those who don't
9. It leads respondents toward a particular answer rather than asking neutrally
10. A small sample can vary considerably from the true population value just by chance
11. Visual presentation choices can emphasise or downplay differences, shaping the reader's perception even with identical underlying data
12. The people who visit and respond on that particular website are unlikely to be representative of the general population
13. A statistic's reliability depends heavily on how it was collected, not just the number reported
14. A random sample, even if small, avoids systematic skew, while a large biased sample can be consistently wrong in the same direction
15. Selectively reporting favourable findings can mislead an audience about the survey's complete, genuine results
16. Advertisements may cite surveys or statistics that were collected using biased or unrepresentative methods to support their claims
17. Representativeness (avoiding bias) matters more for genuine accuracy than sheer sample size alone
18. The answer options themselves are leading, since there is no option to express a neutral or negative view
19. Timing can influence how people are currently feeling or thinking about an issue, independent of any change in sampling method
20. Comparing a sample's demographic makeup against known, complete population data can reveal whether the sample over- or under-represents certain groups
21. Critically evaluate how data was collected and whether it genuinely represents the population being studied