Beyond Rigor: Improving Evaluations With Diverse Populations

Home About Accurate Data Appropriate Analysis Right Data Context Funder Tools Resources PDFs

Context

While there are many definitions of context, for these purposes, context might best be described as the interrelated conditions in which something exists or occurs.¹ For evaluators, context covers that which they need to consider in addition to the Science, Technology, Engineering, and Mathematics (STEM) project/program being evaluated. Contextual factors include individual factors such as age, race and ethnicity and group factors such as the local economy, available resources, location and changes in project/program leadership.

Depending on the project/program being evaluated and the participants themselves, different contextual factors may be more or less important. For example, in an evaluation of a program to reduce the impact of stereotype threat on students' continuation in STEM; gender, disability, race and ethnicity become more important factors than perhaps college type. College type may become a more important contextual factor in an evaluation of the impact of different remedial math programs on students' continuation in STEM. Participant involvement in other projects/programs whether STEM related or not, could be another important contextual factor.

How participants identify themselves or how they are identified by others is one way to help evaluators understand the individual contextual factors that may be most salient for the evaluation. Learning about what is currently going on in an institution and the surrounding community can also help to identify other important factors. When contextual factors are salient, attention may need to be paid to them in evaluation design, implementation, analysis and interpretation of results.

Across contextual factors, evaluators need to:

• have knowledge and understanding of the contextual factors that are important to the evaluation. If that knowledge and understanding is lacking, then people with the needed experience and expertise need to be hired or another team should do the evaluation.
• be aware of their own world view and the assumptions they make especially for data analysis and interpretation. They also need to be aware of the world views and assumptions of participants and stakeholders. Having members of targeted groups involved in conceptualization of the evaluation and having a mechanism for checking interpretations with participants can correct inaccurate assumptions.
• build relationships and trust with project/program staff and participants. Spending time on the campuses or sites to get to know staff and participants and to have them get to know the evaluators can make a difference. Having someone who is well respected and trusted by staff and participants vouch for the evaluators can help build trust as well. This can be particularly important for evaluators who are new to a particular area.

The eleven factors included here do not begin to encompass the comprehensiveness and complexity of context. They are, however, factors that are often important in STEM workforce development efforts and may need to be considered in the construction, implementation and interpretation of evaluations. For each factor there is an overview of implications for evaluation as well as sources of more information about the factor.