Tip: Remind the potential investigators of the comments and questions the reviewers had about the outline of the proposed evaluation and ask them to respond to those comments and questions.

Tip: Prior to their writing the more complete evaluation plan, encourage possible evaluators to review such resources as:

Tip: Have the more complete evaluation plans reviewed by a reviewer or program officer with expertise in evaluation and provide possible questions to ask about the evaluation plan. Sample questions include:

• Will the evaluation plan collect appropriate data to measure the degree to which the proposed project's objectives are achieved? Are there appropriate comparison groups? If the project's long term goals will not be achieved during the project period, does the evaluation focus on the achievement of shorter term goals that are explicitly tied to longer term goals? Along with addressing project objectives, does the evaluation adequately reflect the goals of the funding program (e.g. if a goal of the program is institutional transformation is that included in the evaluation)?
• Is there an advisory board that will provide oversight to the implementation of the evaluation? What populations and areas of expertise are represented in the advisory board?
• Have the instruments and interview protocols been validated for the targeted populations? If not, are there any plans to do so?
• Is the data collection plan feasible? Are the data accessible for both participants and comparison group members? Are there strategies included for engaging and building relationships with target populations, prior to the collection of data?
• By what demographic characteristics will the data be disaggregated and the results reported? How will multi racial participants be categorized? What categories of disability will be included?
• What will be done with the results of the evaluation? Will they be shared with different stakeholders, including participants, in ways that can be understood and used?

Rationale: In the initial review of a proposal, it is most important to assess the quality and value of that which is being proposed. An excellent proposal with a poor evaluation plan could be competitive for funding but a poor proposal with an excellent evaluation plan could not. Once a proposal has been judged as competitive, more information about the evaluation should be provided. Having someone with expertise in evaluation review the more complete evaluation plan and provide feedback gives proposed evaluators an opportunity to refine their evaluation plans and hopefully conduct better evaluations. With the need to diversify the STEM workforce, it is important that evaluation plans are set up in ways that allow for the collection of accurate data over diverse populations and that the analyses are planned in ways to determine possible differential impacts across subgroups.

It's important to consider cell size. For statistical reasons and often for confidentiality, some aggregation of data needs to be done even though information will be lost in each aggregation.