Referenced from lesson Threats to Validity

The issue

It seems like “false facts” are on the rise today. People share conclusions in the media – including social media – that may be a misrepresentation of the data. Graphic representations of data can often be misleading. To understand the factors that undermine our confidence in conclusions, some important questions to ask are:

  1. How was the sample that was surveyed selected?

  2. How many people were surveyed?

  3. Did the sample selected cover all represented in the population we want information on?

  4. What other factors could be at play in the situation that could impact the conclusion made?

Objective

Students will demonstrate understanding of the various types of threats to validity and their impact on conclusions made.

Procedure

With a partner, decide on a research question from a topic of your choosing. Check the topic with your teacher. Create a claim based on faulty sampling techniques which incorporate:

  • Selection bias

  • Sample size problems

  • Sampling errors

  • Confounding variables

Produce

Create a presentation (could be Google Slides, poster, or be creative) that addresses:

  1. Your research topic & question

  2. Your faulty plan for selecting individuals for the study

  3. Be sure to incorporate all four threats listed in the procedure section

  4. Explain what the problems will be with the validity of your conclusion based on the threats you allowed in your sampling

  5. Explain how you could change your study to minimize these threats

Have fun – pretend to be terrible data scientists – costumes encouraged!

Rubric

Got it! Mostly Getting There Somewhat Nope

Statistical Question: Clearly outlines the research topic and statistical question. The question lends itself to many threats needing to be addressed.

15

13

11

4

0

Threats: Developed to clearly be a threat to validity. Includes a threat from each of the four types listed in assignment.

35

27

21

10

0

Conclusion: Explained how the threats to validity cause problems to the conclusion. Explained what changes should be made to minimize the threats.

40

31

24

12

0

Neatness of Product: Project is neat, organized. Clear that effort was put in.

10

9

6

3

0

(Project by Joy Straub)

These materials were developed partly through support of the National Science Foundation, (awards 1042210, 1535276, 1648684, and 1738598). CCbadge Bootstrap:Data Science by Emmanuel Schanzer, Nancy Pfenning, Emma Youndtsmith, Jennifer Poole, Shriram Krishnamurthi, Joe Politz, Ben Lerner, Flannery Denny, and Dorai Sitaram with help from Eric Allatta and Joy Straub is licensed under a Creative Commons 4.0 Unported License. Based on a work at www.BootstrapWorld.org. Permissions beyond the scope of this license may be available by contacting schanzer@BootstrapWorld.org.