Probability, Inference, and Sample Size

Students consider a classic randomness scenario: the probability that a coin will land on heads or tails. From a data science perspective, this can be flipped from a discussion of probability to one of inference. Specifically, "how many samples do we need, to determine whether a coin is fair or not?"

🖼Show image A stranger on the street invites you to a game of chance. They’ll flip a coin, and you’ll win money if you can predict whether it lands heads-or-tails. If you guess wrong, however, you pay THEM.

"It’s a pure game of chance", they tell you, "we each have equal odds of winning".

When we approach a strange coin, we start out assuming that it’s not biased towards heads or tails - that it will land on both "heads" and "tails" about 50% of the time. This is called the null hypothesis. A weighted coin, on the other hand, might be heavier on one side, creating a bias toward heads or tails more often! But how do we test whether a coin is fair or biased? How do we test the null hypothesis?

Have students share back their sample results, and their predictions after 5 samples and then 20 samples.

Which samples seem to support the null hypothesis? Which ones undermine it?

In Statistics and Data Science, samples like these don’t prove any claim about the coins! Instead, they either produce evidence for or against a claim. The larger the sample, the more evidence we have to support or reject the null hypothesis.

The chances of getting tails from a fair coin three times in a row are pretty good: one-in-8! Maybe it was just the luck of the draw, and the coin is still fair.

But what are the chances of flipping "head" 10 times in a row? 100 times? We might say "There’s a one in a million chance of a fair coin coming up heads 100 times. No way is this coin fair!"

But of course, there is a way. It’s just…​incredibly unlikely.

Students may think that any sample from a fair coin should have an equal number of heads and tails outcomes. That’s not true at all! A fair coin might land on "tails" three times in a row! The fact that this is possible doesn’t mean it’s likely. Landing on "tails" five times in a row? Still possible, but much less likely.

This is where arithmetic thinking and statistical thinking diverge: it’s not a question of what is possible, but rather what is probable.

Statistical inference involves looking at a sample and trying to infer something you don’t know about a larger population. This requires a sort of backwards reasoning, kind of like making a guess about a cause, based on the effect that we see.

In the coin-flip activity, we took samples of each coin and used our intuition about chance and probability to make inferences about whether the coins were fair or weighted.

In other words, we knew how the coin should behave before we even started sampling coin-flips, and then checked to see if the samples matched our expectation.

In statistics, we run the process in reverse: we take a sample and then infer something we didn’t know to begin with.

Statistical inference is used to gain information in practically every field of study you can imagine: medicine, business, politics, history; even art!

Suppose we want to estimate what percentage of all Americans plan to vote for a certain candidate. We don’t have time to ask every single person who they’re voting for, so pollsters instead take a sample of Americans, and infer how all Americans feel based on the sample.

Just like our coin-flip, we can start out with the null hypothesis: assuming that the vote is split equally. Flipping a coin 10 times isn’t enough to infer whether it’s weighted, and polling 10 people isn’t enough to prove that one candidate is in the lead. But if we survey enough people we can infer something about the whole population.

When designing a survey or collecting data, Data Scientists need to make sure they are working hard to get a good, random sample that reflects the population. Lazy surveys can result in some really bad data! But poor sampling can also happen when someone is trying to hide something, or to oppress or erase a group of people.

Can you think of other examples where biased sampling has been used - or could be used - to harm people?

The main reason for doing inference is to guess about something that’s unknown for the whole population.

A useful step along the way is to practice with situations where we happen to know what’s true for the whole population. As an exercise, we can keep taking random samples from that population and see how close they tend to get us to the truth.

The Animals Dataset we’ve been using is just one sample taken from a very large animal shelter.

How much can we infer about the whole population of hundreds of animals, by looking at just this one sample?

Let’s see what happens if we switch from smaller to larger sample sizes.

Divide the class into groups of 3-5 students.

Many people mistakenly believe that larger populations need to be represented by larger samples. In fact, the formulas that Data Scientists use to assess how good a job the sample does is only based on the sample size, not the population size.