Beyond Rigor: Improving Evaluations With Diverse Populations

Home About Accurate Data Appropriate Analysis Right Data Context Funder Tools Resources PDFs

Disability

As an underrepresented group in the Science, Technology, Engineering and Mathematics (STEM) workforce, people with disabilities are frequently targeted in STEM workforce development efforts. Regardless of whether people with disabilities are targeted, since they are 10% of the US population between the ages of 18 to 64,¹ they are part of most STEM workforce development projects/programs.

The US Census categorizes disabilities into three different domains: communicative, physical and mental.² Because many disabilities, especially those in the communicative and mental domains are not obvious, evaluators may not know if there are persons with disabilities involved in their evaluations. Developing data collection methods that are accessible to as broad a group as possible is one of the best ways to collect better data across participants. . The Universal Design for Evaluation Checklist³ can be a valuable resource to do this. Under universal design things are designed to be usable by all people without the need for adaptations.

As indicated in “Beyond Rigor: Accurate Data”, data collection sites need to be accessible to participants with mobility impairment. Web sites and online measures need to be constructed so they can be read by text readers (which convert text to speech for visually impaired participants) and in fonts and colors that low vision participants can read. Directions should be short and clear and all instruments and protocols should be pilot tested with people who are similar to the participants. Sign language interpreters should be available as needed and attention should be paid to information that might be biased toward hearing people such as providing auditory information on websites without captions and asking questions about sounds or things that are heard.

Asking and answering questions about disability-related issues can be very sensitive, especially in groups that include people with and without disabilities. Explanations as to why the information is being asked, who will have access to it, and how it will be used can help people feel more comfortable disclosing their status.

Although the disability rights movement has been with us for 40 years, some still view people with disabilities as weak or in need of “fixing”.⁴ Today, while there is a greater emphasis on removing environmental, social, and attitudinal barriers and on associating disability with political rights, identity, and independence, many of the older ideas are still with us. Evaluators need to reflect on their feelings about interacting with people with disabilities. If they are not comfortable with people with disabilities or have little or no experience with people with disabilities, they should not be involved in the data collection.

Having, or not having, a disability is not the only defining variable of an individual or a group. As appropriate, the analysis needs to include other types of diversity such as race and ethnicity, income and education level.

For more information:

• ADA National Network. Information, guidance and training on the Americans with Disabilities Act.
• Kiernan, N. E., & Matason, R. A. (2001). Do participants have disabilities? Tipsheet #9, University Park, PA: Penn State Cooperative Extension.
• Sullivan-Sulewski, J. & Gothberg, J. (2012). Universal design for evaluation checklist (3rd ed.).

For more about the role of context in evaluation, click here