Fish swimming up a fish latter with steps labeled 'request for proposal', 'proposal review', etc.

Grant Award

Tip: Require more complete evaluation plans for proposals that are considered “best and final” or competitive for funding. The more complete evaluation plan should include responses to reviewer comments about the original evaluation plan in the panel summary. Along with providing more detail on the design, data collection procedures, measures, and analysis, the more complete evaluation plan should include information on the target populations and their relationship to the evaluation in such areas as:
  • validity of measures for the target populations;
  • strategies for building trust with target populations;
  • feasibility of data collection strategies;
  • appropriateness of the proposed data analysis plan; and
  • demographic characteristics by which the data will be disaggregated and the results reported.
Tip: Remind the potential investigators of the comments and questions the reviewers had about the outline of the proposed evaluation and ask them to respond to those comments and questions.
Tip: Prior to their writing the more complete evaluation plan, encourage possible evaluators to review such resources as:
Beyond Rigor
Framework For Evaluating Impacts Of Broadening Participation Projects
Measuring Diversity: An Evaluation Guide for STEM Graduate Program Leaders
NSF The 2010 User-Friendly Handbook for Project Evaluation
The Universal Design for Evaluation Checklist
Tip: Have the more complete evaluation plans reviewed by a reviewer or program officer with expertise in evaluation and provide possible questions to ask about the evaluation plan. Sample questions include:
Rationale: In the initial review of a proposal, it is most important to assess the quality and value of that which is being proposed. An excellent proposal with a poor evaluation plan could be competitive for funding but a poor proposal with an excellent evaluation plan could not. Once a proposal has been judged as competitive, more information about the evaluation should be provided. Having someone with expertise in evaluation review the more complete evaluation plan and provide feedback gives proposed evaluators an opportunity to refine their evaluation plans and hopefully conduct better evaluations. With the need to diversify the STEM workforce, it is important that evaluation plans are set up in ways that allow for the collection of accurate data over diverse populations and that the analyses are planned in ways to determine possible differential impacts across subgroups.

It's important to consider cell size. For statistical reasons and often for confidentiality, some aggregation of data needs to be done even though information will be lost in each aggregation.