Lecture #4: What Not to Do: Types and Sources of Bias
STAT 200 – Introductory Statistics
December 19, 2025
Today’s Objective
After today’s lecture, you should:
- earn the basic techniques for analyzing a report.
- be reflexive with this information.
- understand the practical and ethical concerns that arise when conducting a study.
The Literary Digest vs. George Gallup
In 1936, Democratic President Franklin Delano Roosevelt was running for his second term against the Republican governor of Kansas, Alf Landon. The Literary Digest, one of the most respected and popular publications of the day, decided on an ambitious plan: one of the largest and most expensive polls ever conducted to predict the winner. They surveyed around 2.4 million people, which accounted for over 5% of the total turnout, chosen from subscriber lists, telephone directories and other available lists, and predicted that Landon would win in a landslide, 57% to Roosevelt’s 43%. Meanwhile, George Gallup ran a focused poll of only 50,000 voters.
The Literary Digest vs. George Gallup
Roosevelt went on to win 63% of the vote in what is one of the largest landslide elections in American history, a result George Gallup predicted. Roosevelt won the second highest percentage of votes in a contested election, behind only LBJ in 1968, and the second highest electoral vote total, behind only Reagan in 1984.
The Gallup Organization is now one of the most respected names in polling.
If those two polls came out about the a House race this year, how could you determine which to trust?
Things to Consider about a Scientific Study
When thinking about a scientific study reported in the mainstream press, consider the following factors:
- The source of the reporting, whether that be the population the sample was drawn from, the scientists publishing the study or the reporter mentioning it.
- The variables involved (and their meanings).
- The conclusions drawn from the data
Consider the Source
As you listen to sound bites on the radio or television spouting off statistics from studies, you should ask yourself pertinent questions:
- Who paid for the study?
- Where were the data collected?
- When was the information collected?
- Who published the study?
- How knowledgeable is the reporter on science?
Consider the Source
Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the “wet streets cause rain” stories. Paper’s full of them.
In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.
— Michael Crichton
Consider the Variables
Have you ever read a headline like “America’s 100 Best Companies to Work For” or “Top 10 Things to Make You Happy”? You should read the fine print of the study to find out by what criteria “best” is defined or whose standard of “happy” they are talking about.
Headlines like these can mean very different things to different people.
In addition to considering what the variable of the study is measuring, we also need to think about those variables not directly being measured – ones which may have an effect of the results. (Did anyone have to watch Super Size Me in school?)
A well-constructed study will make a point to identify and account for reasonable confounding variables.
Consider the Variables
Suppose I want to study the effectiveness of various methods of quitting smoking. The easier option would be to go to a smoking support group and tracking the progess of the people involved who have chosen their own methods. The more expensive and time-consuming method would be to get subjects and assign them different methods randomly myself.
What confounding variables might I introduce to my study opting for the first study over the second?
Consider the Conclusions
In the end, it is the conclusions of the study that we should be most aware of:
- Do the data support the conclusion?
- Do the results present the whole picture or just a part?
- Could there be other reasonable conclusions drawn?
- Could there be other reasons for the same conclusion drawn?
- Does the study have any practical applications?
- What is the effect size compared to the starting values?
Why Does This Matter to You?
Why does this matter to you as a scientist/scholar?
- You want your research to be published.
- You want your research to be read by others.
- You want your research to be analyzed by others.
- You want your research to be used by others.
- You need to ensure that your research is properly reported.
Study Validity
There are two main types of study validity:
External validity is the validity of applying the conclusions of a scientific study outside the context of that study. In other words, it is the extent to which the results of a study can be generalized to and across other situations, people, stimuli, and times.
Internal validity is the validity of conclusions drawn within the context of a particular study.
Study Validity
Both are important.
Internal validity concerns whether the drawn conclusions are appropriate, given the data.
External validity concerns whether these findings can be extended to other conclusions.
Because general conclusions are almost always a goal in research, external validity is an important property of any study. Mathematical analysis of external validity concerns a determination of whether generalization across heterogeneous populations is feasible, and devising statistical and computational methods that produce valid generalizations.
Study Bias
In the context of a study’s structure, bias refers to the how of unrepresentative the sample is of the population. There are several sources of bias in a study, broadly divided into researcher-based or respondent-based.
Note that no study is perfect. No sample will perfectly represent the population. Bias is only important when it is sufficiently severe to cause the researcher to draw incorrect conclusions.
Researcher-Caused Bias
Typical sources (and names) of bias include:
Selection bias occurs when the selection of individuals for analysis (or groups or data) is done in such a way that proper randomization is not achieved.
Researcher bias occurs when a researcher influences the results of a study, whether purposely or not.
Response bias occurs when a researcher’s behavior (or existence) causes a participant to alter his or her response or when a participant gives an inaccurate response.
Response Bias Example
Consider the responses to these questions, all ostensibly asking the same thing, particularly from the perspective of someone undecided on the topic, or on the edge?
- Do you approve or disapprove of former President Trump’s decision to seek reelection?
- Do you support President Trump choosing to run for President to make America great again, in spite of the deep state attacks on him, the left-wing media’s lies about him, and the Democrat’s fake prosecutions of him, despite his innocence?
- Do you oppose Donald Trump’s decision to run again after his failure of a first term, him leading a failed insurrection, his election denial and his being convicted of 34 felonies?
Participant-Caused Bias
Beyond conscious biases, there are several that do not arise from planning errors. They arise from the actions of the participants. These include:
Participation bias occurs when there is a problem with the participation — or lack thereof — of those chosen for the study.
As an example, nonresponse bias occurs when there is a lack of participation in a self-selected sample from certain segments of a population, when a person refuses to participate in a survey, or when a respondent omits questions when answering a survey.
Participant-Caused Bias
Beyond conscious biases, there are several that do not arise from planning errors. They arise from the actions of the participants. These include:
Voluntary response bias arises when you ask for participants to chose to participate in your study. Generally only those with strong opinions will chose to participate.
Another example concerns respondents who do not report the truth.
Participant-Caused Bias
Dropouts are participants who begin a study but fail to complete it.
Nonadherents are participants who remain in the study until the end but stray from the protocol (directions given).
Oops… I Ruined the Study
Finally, there are simple errors in the study. These tend to occur in the transcribing of the data into a form that the program can handle.
Processing errors are errors that occur simply from the data being processed, such as typos when data are being entered.
More Information
The National Institutes of Health (NIH) is the primary agency of the United States government responsible for biomedical and public health research. As such, they focus on proper experimental procedures. Their extensive website contains a lot of information about sampling: https://www.ncbi.nlm.nih.gov/books/NBK253184/table/background.t1
If you will be performing biomedical research, bookmarking this site might be a good idea.
Other research organs of the US government have their own websites aimed at ensuring good research is done, such as the National Institute of Standards and Technology (NIST). https://www.nist.gov/itl
