Author(s):
In the information age, numerical skills such as proportional reasoning, graph reading, estimation, and arithmetic are essential life skills. They are critical components of health, financial, and scientific literacy, and are thus an essential component of a liberal education. Particularly among math-avoidant or math-anxious students, numeracy is often developed in general education (gen-ed) science classrooms. However, research-validated strategies for improving numeracy in these environments are currently lacking. To inform the numeracy landscape and identify effective classroom, curricular, and pedagogical strategies for improving it, we created and validated an assessment instrument that measures students’ numerical skills and affect. We utilized the assessment to answer the following questions: (1) Do students’ numeracy scores change significantly over the course of a semester of gen-ed science instruction? (2) How do affective factors impact students’ numeracy scores and the size of apparent “achievement gaps”? And (3) Are the “real world” contexts and phrasings of the questions on the assessment equally accessible to students from all demographic groups? Answering these questions helps us understand how gen-ed science classes, and students’ numerical affect upon entering them, impact numeracy development.They afford us insight into the complex interplay of student affect and apparent numeracy level. After validating our instrument, we found that significant, class-level changes in students’ numerical assessment scores over a semester are rare. However, changes in affect are more common. Furthermore, compensating for differences in affective domain variables such as level of math anxiety and numerical self-efficacy substantially reduced, and in some cases entirely removed, demographic correlations with score (so-called “achievement gaps”, though this is a misnomer). Measuring numerical skills accurately requires that our assessment be equally accessible to every student. To assess the equitability of our assessment, we identified questions where students’ perception of the question’s relevance to daily life showed small but significant differences across demographic groups and, in most cases, also statistical differential item functioning (DIF). We conducted focus groups to understand these differences, and made revisions to the questions to improve their accessibility. Nearly all of the items on our assessment are now free of DIF between Hispanic, Black, and White students. Our analysis also indicates that students generally perceive the assessment items as relevant to their own lives, and this was confirmed by participants in seven focus groups. Bias in how assessments are designed, created, administered, and interpreted can preclude them from accurately measuring the performance of groups who are historically underrepresented in STEM domains. Thus, we conducted our assessment development, revisions, and evaluation with an eye toward ensuring that the contexts of our questions were inclusive of a diverse range of personal experiences, and that no items systematically favored or punished participants for differential lived experiences. Because the results of our quantitative and qualitative analyses align and indicate that our assessment is equitable, accessible, and relevant, it shows promise as a tool for appropriately evaluating numeracy and identifying effective classroom and curricular practices for helping gen-ed science students develop it.
Coauthors
Vonna Hemmler, Amherst College; Isaac Rosenthal, Amherst College
