Probability, Inference, and Sample Size

Students consider a classic randomness scenario: the probability that a coin will land on heads or tails. From a data science perspective, this can be flipped from a discussion of probability to one of inference. Specifically, "based on the number of coin flips we observed, what can we conclude about the fairness of a coin?"

A casino dealer (called a "croupier") invites you to play a game of chance. They’ll flip a coin repeatedly. On each flip, the croupier gives you a dollar if it comes up tails. If it comes up heads, you pay them a dollar.

"It’s a pure game of chance", they tell you, "we each have equal odds of winning".

A fair coin should land on "heads" about as often as it lands on "tails": half the time.

In general, we assume that in the long run, an ordinary coin will land on "heads" 50% of the time. Our assumption that there is no bias towards "heads" or "tails" is our null hypothesis. A weighted coin, on the other hand, might be heavier on one side, creating a bias toward one side! And since we lose money on heads, we’re worried about bias in favor of heads.

So how do we test the null hypothesis?

In Statistics and Data Science, samples like these don’t prove anything about the coins! Instead, they either produce enough evidence to reject the null hypothesis, or fail to do so. If the null hypothesis is actually false, larger samples give us a better chance of producing evidence to reject it.

The chances of getting "heads" from a fair coin three times in a row aren’t too small: 1-in-8! Maybe it was just the luck of the draw, and the coin is still fair.

Should we suspect a scam if the croupier’s coin flipped heads 10 times in a row?

The probability of a fair coin getting all heads for 10 flips in a row is ¹ /₂¹⁰ , or roughly 0.001.

Statisticians would explain this as follows: "If the null hypothesis were true, then the probability of getting sample results at least as extreme as the ones observed is 0.001."

But of course, there is a way. It’s just…​incredibly unlikely.

Students may think that any sample from a fair coin should have an equal number of heads and tails outcomes. That’s not true at all! A fair coin might land on "tails" three times in a row! The fact that this is possible doesn’t mean it’s likely. Landing on "tails" five times in a row? Still possible, but much less likely.

This is where arithmetic thinking and statistical thinking diverge: it’s not a question of what is possible, but rather what is probable or improbable.

Statistical inference involves looking at a sample and trying to infer something you don’t know about a larger population. This requires a sort of backwards reasoning, kind of like making a guess about a cause, based on the effect that we see.

Probability reasons forwards.

Because we know that the chance of coming up heads each time for a "population" of flips of a fair coin is 0.5, we can do probability calculations like "the probability of getting all three heads in three coin flips is 0.5 × 0.5 × 0.5 = 0.125." Likewise, we can say the probability of getting three of a kind in a randomly dealt set of five cards is 0.02.

"Based on what we know is true in the population, what’s the chance of this or that happening in a sample?" This is the kind of reasoning involved in probability.

Inference reasons backwards.

In the coin-flip activity, we took samples of coin flips and used our knowledge about chance and probability to make inferences about whether the coin was fair or weighted.

In other words, we looked at sample results and used them to decide what to believe about the population of all flips of that coin: was the overall chance of heads really 0.5?

"Based on what we saw in our sample, what do we believe is true about the population the sample came from?" This is the kind of reasoning involved in inference.

Statistical inference uses information from a sample to draw conclusions about the larger population from which the sample was taken. It is used in practically every field of study you can imagine: medicine, business, politics, history…​ even art!

Suppose we want to estimate what percentage of all Americans plan to vote for a certain candidate. We don’t have time to ask every single person who they’re voting for, so pollsters instead take a sample of Americans, and infer how all Americans feel based on the sample.

Just like our coin-flip, we can start out with the null hypothesis: assuming that the vote is split equally. Flipping a coin 10 times isn’t enough to infer whether it’s weighted, and polling 10 people isn’t enough to convince us that one candidate is in the lead. But if we survey enough people we can be fairly confident in inferring something about the whole population.

Suppose we were able to make a million phone calls to use voters…​

Bad samples can be an accident - or malice!

When designing a survey or collecting data, Data Scientists need to make sure they are working hard to get a good, random sample that reflects the population. Lazy surveys can result in some really bad data! But poor sampling can also happen when someone is trying to hide something, or to oppress or erase a group of people.

Can you think of other examples where biased sampling could result in intentionally or unintentionally misleading results?

The main reason for doing inference is to guess about something that’s unknown for the whole population.

A useful step along the way is to practice with situations where we happen to know what’s true for the whole population. As an exercise, we can keep taking random samples from that population and see how close they tend to get us to the truth.

The Animals Dataset we’ve been using is just one sample taken from a very large animal shelter.

We’re going to analyze which is better at guessing the truth about an entire population - a small sample of 10 randomly selected animals, or a large sample of 40 randomly selected animals.

The Sampler plugin features a Mixer, Spinner, and Collector. Today, we’ll be using the Collector, which chooses a specified number of cases from a dataset.

With or without "replacement"?

If we pick cards from a deck, each sample changes the outcomes of the ones that follow. There’s only one Ace of Hearts in the deck, and you can’t draw it twice! When flipping a coin, each sample has the same number of possible outcomes as the one before: heads or tails. It’s as if each one has been replaced with a copy of the same outcome.

That’s the difference between sampling with or without replacement. If it’s like rolling dice or flipping a coin, it’s sampling with replacement. If it’s like drawing cards from a deck, it’s sampling without replacement.

Many people mistakenly believe that larger populations need to be represented by larger samples. In fact, the formulas that Data Scientists use to assess how good a job the sample does is only based on the sample size, not the population size.