Threats to Validity Rubric

	Wow!	Getting There	Needs Improvement
Selection bias	The research plan lays out clearly and specifically how information will be gathered from a non-representative sample of the population.	The research plan lays out how information will be gathered from a non-representative sample of the population, but is lacking in detail and specificity.	The research plan does not address how information will be gathered from a non-representative sample of the population.
Bias in the study design	The study is clearly biased, with “loaded” questions that lead to misrepresentation of true opinions.	The study includes some bias, but it is unclear how the plan will lead to misrepresentation of true opinions.	The study design does not include bias.
Poor choice of summary data	The research plan includes extreme outliers that will shift the results of the analysis in ways that do not represent the population as a whole.	The research plan includes some outliers, but they are perhaps not adequately extreme to shift the results and misrepresent the population.	The research plan does not include extreme outliers.
Confounding variables	The study overlooks factors that clearly influence a relationship, leading to invalid claims.	The study overlooks factors that might influence a relationship, perhaps leading to invalid claims.	The study does not include any confounding variables.
Discuss Conclusions	The concluding discussion clearly articulates how all of the threats to validity cause problems to the study’s conclusions.	The concluding discussion only articulates how some of the threats caused problems or is lacking in detail and evidence / analysis.	The concluding discussion is lacking in detail and evidence. Understanding of how threats to validity influence a study is not demonstrated.
Good Data Scientists would…	Clear explanations are provided of what changes could be made to minimize the threats.	Some explanation is provided of what changes could be made to minimize threats, but more details are needed.	Understanding of how to revise the study to minimize threats is not demonstrated.

Wow!

Getting There

Needs Improvement

Selection bias

The research plan lays out clearly and specifically how information will be gathered from a non-representative sample of the population.

The research plan lays out how information will be gathered from a non-representative sample of the population, but is lacking in detail and specificity.

The research plan does not address how information will be gathered from a non-representative sample of the population.

Bias in the study design

The study is clearly biased, with “loaded” questions that lead to misrepresentation of true opinions.

The study includes some bias, but it is unclear how the plan will lead to misrepresentation of true opinions.

The study design does not include bias.

Poor choice of summary data

The research plan includes extreme outliers that will shift the results of the analysis in ways that do not represent the population as a whole.

The research plan includes some outliers, but they are perhaps not adequately extreme to shift the results and misrepresent the population.

The research plan does not include extreme outliers.

Confounding variables

The study overlooks factors that clearly influence a relationship, leading to invalid claims.

The study overlooks factors that might influence a relationship, perhaps leading to invalid claims.

The study does not include any confounding variables.

Discuss Conclusions

The concluding discussion clearly articulates how all of the threats to validity cause problems to the study’s conclusions.

The concluding discussion only articulates how some of the threats caused problems or is lacking in detail and evidence / analysis.

The concluding discussion is lacking in detail and evidence. Understanding of how threats to validity influence a study is not demonstrated.

Good Data Scientists would…

Clear explanations are provided of what changes could be made to minimize the threats.

Some explanation is provided of what changes could be made to minimize threats, but more details are needed.

Understanding of how to revise the study to minimize threats is not demonstrated.

These materials were developed partly through support of the National Science Foundation, (awards 1042210, 1535276, 1648684, 1738598, 2031479, and 1501927). Bootstrap by the Bootstrap Community is licensed under a Creative Commons 4.0 Unported License. This license does not grant permission to run training or professional development. Offering training or professional development with materials substantially derived from Bootstrap must be approved in writing by a Bootstrap Director. Permissions beyond the scope of this license, such as to run training, may be available by contacting contact@BootstrapWorld.org.