How can you tell if a coin is fair, or designed to cheat you? Statisticians know that a fair coin should turn up "heads" about as often as "tails", so they begin with the null hypothesis: they assume the coin is fair, and start flipping it over and over to record the results.
A coin that comes up "heads" three times in a row could still be fair! The odds are 1-in-8, so it’s totally possible that the null hypothesis is still true. But what if it comes up "heads" five times in a row? Ten times?
Eventually, the chances of the coin being fair get smaller and smaller, and a Data Scientist can say "this coin is a cheat! The chances of it being fair are one in a million!"
By sampling the flips of a coin, we can infer whether the coin itself is fair or not. Inference plays a major role in Data Science and Statistics!
-
If we survey pet owners about whether they prefer cats or dogs, the null hypothesis is that the odds of someone preferring dogs are about the same as them preferring cats. And if the first three people we ask vote for dogs (a 1-in-8 chance), the null hypothesis could still be true! But after five people? Ten?
-
If we’re looking for gender bias in hiring, we might start with the null hypothesis that no such bias exists. If the first three people hired are all men, that doesn’t necessarily mean there’s a bias! But if 30 out of 35 hires are male, this is evidence that undermines the null hypothesis and suggests a real problem.
-
If we poll voters for the next election, the null hypothesis is that the odds of voting for one candidate are the same as voting for the other. But if 80 out of 100 people say they’ll vote for the same candidate, we might reject the null hypothesis and infer that the population as a whole is biased towards that candidate!
Sample size matters! The more bias there is, the smaller the sample we need to detect it. Major biases might need only a small sample, but subtle ones might need a huge sample to be found. However, choosing a good sample can be tricky!
Random Samples are a subset of a population in which each member of the subset has an equal chance of being chosen. A random sample is intended to be a representative subset of the population. The larger the random sample, the more closely it will represent the population and the better our inferences about the population will tend to be.
Grouped Samples are a subset of a population in which each member of the subset was chosen for a specific reason. For example, we might want to look at the difference in trends between two groups ("Is the age of a dog a bigger factor in adoption time v. the age of a cat?"). This would require making grouped samples of just the dogs and just the cats.
These materials were developed partly through support of the National Science Foundation, (awards 1042210, 1535276, 1648684, 1738598, 2031479, and 1501927). 
Bootstrap by the Bootstrap Community is licensed under a Creative Commons 4.0 Unported License. This license does not grant permission to run training or professional development. Offering training or professional development with materials substantially derived from Bootstrap must be approved in writing by a Bootstrap Director. Permissions beyond the scope of this license, such as to run training, may be available by contacting contact@BootstrapWorld.org.