Beyond Rigor: Improving Evaluations With Diverse Populations

Home About Accurate Data Appropriate Analysis Right Data Context Funder Tools Resources PDFs

Science

The culture of science, its mores and folkways, career ladders and methods, is the primary context for the evaluation of Science, Technology, Engineering, and Mathematics (STEM) projects/programs. Without that knowledge and understanding, evaluators will be unable to formulate the right questions and understand the answers. For example, without knowing about the academic career ladder in science, it would be very difficult to do a quality evaluation of efforts to increase the number of women science faculty. Without understanding the kinds of math skills that are needed for science careers from nutrition to nuclear physics, evaluations of projects/programs to increase undergraduate retention in the sciences may miss an important bottleneck. Knowing the funding mechanisms for science graduate students, which is often quite different from that for other graduate students, is an important piece of context for evaluations of projects to increase STEM graduate student diversity. And of course, these funding mechanisms may be quite different for different funders. There is a culture of science but that culture varies based on the field. For example, there are similarities in the culture of biology and that of physics but there are also differences.

Knowledge of the cultures of science is very important to do a quality evaluation of STEM projects/programs. Also needed are skills, knowledge, and expertise in evaluation. Yet few evaluators have the background, training, and experience in both science and evaluation. STEM evaluators primarily fall into one of two groups: those from non-STEM backgrounds with formal evaluation training or those from STEM backgrounds often now in administrative positions, with little formal evaluation training.

In theory, the easiest solution is for people from both groups to work together as a team where there is equal power and each learns from the other. In practice, this can be difficult. Evaluators without science backgrounds often feel that evaluation skills are context-free and can be applied to any project and program. But they can't. Scientists who do evaluations often feel that knowing the science discipline, the research design, and statistics is all that is necessary to do a quality evaluation. But it's not. In both cases, the skill sets are necessary but not sufficient. While advisors and informants are useful and important in these cases, they are not enough.

Evaluators without science backgrounds need to be able to understand implications for evaluation of such areas as the “hierarchical status” of different science disciplines and the changing definitions of what is “real” science and of the concept of scientist as “who you are” more than “what you do”. Scientists who do evaluations need to be aware of the role context plays in evaluation, the issues of cultural competence in evaluation, and the potential social, educational, and even political implications of STEM evaluations.

For more information:

• Hrabowski, F. (2012). We need to change the culture of science teaching and learning. TEDxMidAtlantic
• Science Magazine

For more about the role of context in evaluation, click here