Threats to Validity

Students are introduced to the concept of validity, and a number of possible threats that might make an analysis invalid.

As good Data Scientists, the staff at the animal shelter are constantly gathering data about their animals, their volunteers, and the people who come to visit.

But just because they have data doesn’t mean the conclusions they draw from it are correct!

In the example of the cat-owning survey, the claim that “people prefer cats to dogs” is invalid because the data itself wasn’t representative of the whole population (of course cat-owners are partial to cats!).

There are several major Threats to Validity you should be on guard against:

(1) Selection bias - Data was gathered from a biased sample of the population. This is the problem with surveying cat owners to find out which animal is most loved!

(2) Bias in the study design - Data was gathered using a “loaded” question like “Since annual vet care comes to about $300 for dogs and only about half of that for cats, would you say that owning a cat is less of a burden than owning a dog?” This could easily lead to a misrepresentation of people’s true opinions.

(3) Poor choice of summary data - Even if the selection is unbiased, sometimes outliers are so extreme that they make the mean completely useless at best - and misleading at worst.

(4) Confounding variables - A study might find that cat owners are more likely to use public transportation than dog owners. But it’s not that owning a cat means you drive less: people who live in big cities are more likely to use public transportation, and also more likely to own cats.

And there are many other threats to validity out there!

Life is messy, and there are always threats to validity.

Data Science is about doing the best you can to minimize those threats, and to be up front about what they are whenever you publish a finding.

When you do your own analysis, make sure you include a discussion of the threats to validity!

Why is it important to consider potential threats to validity?

Students are asked to consider the ways in which statistics are misused in popular culture, and become critical consumers of some statistical claims. Finally, they are given the opportunity to misuse their own statistics, to better understand how someone might distort data for their own ends.

You have already seen a number of ways that statistics can be misused:

(1) Using the mean instead of the median with heavily-skewed data

(2) Using a correlation to imply causation

(3) Incorrectly explaining the r-value from Linear Regression as corresponding to something happening "some percentage of the time" instead of describing "the percentage of the variation that is explained by the explanatory variable"

There are other ways to mislead the audience as well:

(4) Intentionally using the wrong chart - suppose the census asks for data from different groups of people, and gets none from one group. That would be very suspicious! That group would show up as an empty space on bar chart, making the absence visible. A pie chart, however, would hide that absence completely - making it less likely that anyone would even notice that group had been "erased"!

(5) Changing the scale of a chart - Changing the y-axis of a scatter plot can make the slope of the regression line seem smaller: "look, that line is basically flat anyway!"

With all the news being shared through newspapers, television, radio, and social media, it’s important to be critical consumers of information!