Comparison Groups
Tip: Make comparisons across more and less effective projects/programs. Factors common across effective programs may also be common across ineffective projects/programs.
Rationale: If only exemplary projects/programs are included, it is not possible to determine if “effective” characteristics identified by the research are indeed unique to these positive outliers. Comparison groups of other projects/programs are needed to see what is unique to effective projects/programs. For example, one study comparing effective schools to typical schools found both were using similar curriculum.1
Tip: When using existing data sets for comparison groups, determine the process that was used to categorize participant race/ethnicity and disability status and then use a similar process to categorize your participants to ensure data are comparable.
Rationale: If data sets use different methods to categorize race/ethnicity and disability status, cross data set comparisons will not be valid. For example, in the Survey of Earned Doctorates (SED) a person can define themselves as non-Hispanic, Mexican/Chicano, Puerto Rican, Cuban, or Other Hispanic as well as by race.2 In the Integrated Postsecondary Education Data System (IPEDS), one can identify as non-Hispanic or Hispanic and by race. However, aggregate data reported to IPEDS has Hispanics of any race as one category and only non-Hispanics are reported by race.3 Also, SED4 asks about five broader areas of disability while IPEDS5 reports on 11 areas of disability.
1 Clewell, B. C. & Campbell, P. B., with Perlman, L. (2007). Good schools in poor neighborhoods: Defying demographics, achieving success. Washington, DC: Urban Institute Press
2 NORC. (2009). SED (Survey of Earned Doctorates)[survey instrument].
3 Institute of Education Sciences (IES)/National Center for Education Statistics. (2007). Memo: Changes to race/ethnicity reporting to the integrated postsecondary education data system (IPEDS) as mandated by the U.S. Department of Education.
4 NORC. (2009). SED (Survey of Earned Doctorates)[survey instrument].
5 Raue, L., Lewis, L., & Coopersmith, J. (2011). NCES 2011-018, students with disabilities at degree-granting postsecondary institutions: First look [report and data].