Threats to Validity can undermine a conclusion, even if the analysis was done correctly.
Some examples of threats are:
-
Selection bias - identifying the favorite food of the rabbits won’t tell us anything reliable about what all the animals eat.
-
Study bias - If someone is supposed to assess how much cat food is eaten each day on average, but they only measure how much cat food is put in the bowls (instead of how much is actually consumed), they’ll end up with an over-estimate.
-
Poor choice of summary - Suppose a different shelter that had 10 animals recorded adoption times (in weeks) as 1, 1, 1, 7, 7, 8, 8, 9, 9, 10. Using the mode (1) to report what’s typical would make it seem like the animals were adopted much quicker than they really were, since 7 out of 10 animals took at least 7 weeks to be adopted.
-
Confounding variables - Some shelter workers might prefer cats, and steer people towards cats as a result. This would make it appear that “cats are more popular with people”, when the real variable dominating the sample is what workers at the shelter prefer.
These materials were developed partly through support of the National Science Foundation, (awards 1042210, 1535276, 1648684, 1738598, 2031479, and 1501927). 
Bootstrap by the Bootstrap Community is licensed under a Creative Commons 4.0 Unported License. This license does not grant permission to run training or professional development. Offering training or professional development with materials substantially derived from Bootstrap must be approved in writing by a Bootstrap Director. Permissions beyond the scope of this license, such as to run training, may be available by contacting contact@BootstrapWorld.org.