Leveraging Institutional Data to Advance Equity in STEM Courses

Caption: Course staff discuss their Course Equity Report during the Foundational Course Initiative’s Course Design Institute in 2019. (Image credit: Kairos Marquardt)

How do faculty learn about the historical equity landscape in their courses when they have neither the time nor the data access to perform analyses themselves?

Many institutions face this challenge which is not easy to address. If instructors and administrators are to understand where their efforts to advance student learning and equity in their courses are succeeding, they need to measure it. That approach needs to be adaptable to specific questions, and also scalable to cover different types of courses at the university.

This problem turned into the underlying motivation for creating what we call course equity reports. Building on our expertise as researchers and instructors, we’ve developed a standardized set of analyses to help instructors understand what is happening in their courses for different groups of students and reflect on whether the outcomes are aligned with their equity goals. By developing these reports centrally rather than requiring each instructor to obtain data access and conduct the analysis independently, we have lowered the barriers to engaging with this information.

The ability to produce these reports at scale enables departmental leaders and administrators to look across courses to identify priority areas for diversity, equity, inclusion, and justice (DEIJ) efforts and, by periodically requesting new reports, observe progress over time.

In this post, we describe how we have developed our course equity reports and the lessons we’ve learned along the way. Our intentions are twofold. First, we aim to show a possible path forward for departments and institutions wishing to leverage their existing data to support student success but don’t have a strong sense of where to start. Second, we aim to inspire institutions at an earlier stage of their institutional data wrangling efforts with an example to motivate the difficult work of creating a robust learning analytics environment.

Laying the Groundwork & Developing Our Course Equity Reports

Several pieces were in place that allowed this effort to emerge. The University of Michigan recognized the importance of digital student records early, creating and maintaining a comprehensive data warehouse containing more than two decades of student data. The University prioritized active use of this data for research purposes by constructing a simplified dataset from this warehouse that contains the most frequently requested elements, is well documented, and has a clear review process to request and gain access. In parallel, central units within the University, including both the Center for Research on Learning and Teaching and the Center for Academic Innovation, invested both funding and time to foster learning analytics communities and leverage this dataset for both institutional and publishable research efforts.

Supporting Equity-Focused Analytics Takes Infrastructure, Resources, and Time

Figure 1: Timeline of institutional events that allowed the Assessment Toolkit to happen.

One important early finding was that a number of important ‘foundational’ classes–large introductory classes that students must succeed in to proceed into a given field of study–had high proportions of students receiving lower grades than in their other courses (aka receiving a grade penalty).^1-3 Large classes in STEM disciplines feature disproportionately in this group.

Digging deeper revealed troubling trends. In most of these courses, historically minoritized students received larger grade penalties than their peers, with potentially critical impacts on persistence in the field. These insights spurred faculty-led efforts at the university to understand these patterns and design interventions to address these inequities.

Recognizing the urgency of these challenges, the University of Michigan supported several efforts to increase student success in these foundational courses. Most prominent was the Foundational Course Initiative (FCI), which partners expert consultants with instructional teams from foundational courses on multi-year course redesign efforts to improve equity and pedagogical practices in these courses.

Upon joining the FCI as Head of Learning Analytics and Assistant Director, Heather Rypkema was tasked with leveraging the University’s data infrastructure to inform evidence-based course design decisions and promote more equitable outcomes for historically marginalized students. She scoured the database for information about students–their identities, backgrounds, and educational trajectories–to create visualizations and analyses that could inform pedagogical improvements. While investigating grade outcome disparities for minoritized students, she saw that intersectionality (the impact of having multiple historically marginalized identities) almost universally magnified those disparities. This work culminated in a “Course Equity Report” intended to help faculty better understand their students and expose the ways in which their course might be perpetuating historical inequities in higher education through grade outcomes. An example visualization from the report, which demonstrates that students tend to experience larger outcome disparities when they hold multiple marginalized identities, can be seen in the figure below. The key metric in this figure is the “barrier index^a”, which is a way of unpacking the implications of intersectionality, while masking specific identity characteristics that might trigger observer bias.

Summary of Student Performance in ANON101 Based on Barrier Index

Figure 2: Sample plot from the course equity report for an anonymized course. Here, we plot each student based on the difference between the grade they earned in this course and their average grade in all other courses they’ve taken (GPAO) and the relative amount of systematic barriers they face in higher education (the barrier index). For an equitable course, we would expect there to be no differences in the distributions based on the barrier index. A negative Grade – GPAO signifies that the student earned a lower grade in this course than their average grade in other courses (grade penalty). The box plots show the first and third quartiles as well as the median difference between students’ grade in the course and average grades in other courses. Each data point represents an individual student’s outcome.

Starting in May 2018, faculty who partnered with FCI in a time and resource-intensive 3-year course redesign process (5-6 new courses embark on this process every year) received these reports. These reports informed and focused redesign goals as well as prompted questions for further analysis in the course. In the summer of 2019, Rypkema connected with Eric Bell, a faculty member who was proposing a centralized effort, the Assessment Toolkit, to provide faculty and programs with resources to utilize institutional data to inform pedagogical and curricular decisions. They agreed to collaborate and launched their effort with funding from CAI in winter 2019, with the Course Equity Report as one component prioritized for scaling.

Work on the Course Equity Report^b accelerated in the pandemic at the urging of the College of Literature, Arts and Sciences Associate Dean for Undergraduate Education, and previous report recipient, Tim McKay. Concerned that remote learning would further limit instructors’ abilities to observe and understand their students’ experiences and that the pivot to virtual classes and remote learning environment was likely to widen the outcome gap for many marginalized students, McKay urged the scale-up and distribution of reports to the instructors of large introductory courses where the historical data showed gaps between the outcomes of minoritized students and their majoritized peers. The Assessment Toolkit team worked to generalize the reports, standardizing the text with input from behavioral scientists to be broadly applicable and include guiding questions to help readers interpret the reports and think about next steps. These reports also provided a list of university resources to help faculty mitigate the impact of the pandemic on minoritized students.

Today, FCI courses continue to receive customized consultations on the reports to inform their course design efforts, and The Center for Research on Learning and Teaching is currently developing the infrastructure to provide consultations around these reports to the broader Teaching and Learning community at University of Michigan.⁴

What We’ve Learned and Next Steps

Reflecting on our experience developing these Course Equity Reports and working with faculty, we’ve learned three key lessons that have been essential to the project’s success.

First, launching such an effort requires substantial university investment over a sustained period. Our project was enabled by a pre-existing University commitment to making learning analytics data available to researchers, the existence of a core group of driven senior leaders (administrators and faculty), and the availability of institutional and external resources to fund both the initial research and the scale-up required to make an important impact.^c Without all three of these co-occurring, this project would have remained only an idea.

Second, having dedicated personnel work on the project was essential to its success. Learning to work with institutional data is a time-consuming process and framing the questions appropriately requires both expertise in the problem and the available data. As such work is typically undervalued in the departmental evaluations of faculty members, a dedicated team of researchers whose focus is on developing these analytical tools and learning the nuances of the institutional data is important. For us, it was also essential because our university does not have a centralized institutional research office to assist with projects like these. For researchers working at institutions in similar situations, this point cannot be understated.

Finally, we’ve learned that there is no one-size-fits all approach to sharing these analyses. The findings from these reports are complex, and it can be emotional for faculty to engage with the findings we’re surfacing. Some faculty members appreciated receiving the reports a few weeks before the start of their course while others lamented that a few weeks would not be enough to redesign their courses in light of the new information. Some faculty expressed wanting to explore the data themselves so that they could better understand what was happening in their course and particularly whether previous efforts had positively impacted student success. Some faculty liked being able to directly see the data for their courses; others thought that only department administrators should have the reports and then discuss them with individual faculty members as needed. With all of this in mind, we have committed to exploring a variety of different mechanisms for promoting and accessing the reports including intensive and structured models of cohort-based curricular change (like the Foundational Course Initiative), individual course-specific conversations (like the equity report consultations being developed by CRLT), and collaborating with Diversity, Equity, Inclusion, and Justice leads within each department to identify the right approach within a particular domain. We’ll need to pay close attention to feedback from our faculty and administrative partners, and be willing to iterate both the product and our approach to community engagement, to ensure our efforts are effective.

Reflections on Pursuing Work at Other Institutions

Institutions are in vastly different places with respect to data infrastructure, access and policies, and not all schools will immediately be in a position to implement this type of centralized data reporting project. In recognition of this, we want to close with a general overview of the steps necessary to leverage institutional data to promote equity at a large scale.

1. Establish a data architecture specifically designed for powering equity-focused educational research. This involves subsetting the full institutional database to a collection of salient variables for assessing equity outcomes in a course or departmental context. These data should include anonymized ID numbers so that personally identifiable information (PII) are not automatically associated with the data set.
2. Make a data stewardship plan. Ideally this involves establishing a single unit to have authority over maintaining the dataset and approving access. At the University of Michigan, this is the Office of Enrollment Management which houses the Registrar’s Office.
3. Determine the policies around data access. Who is eligible to request access, and what criteria do they have to satisfy before access is granted? Provide training around responsible data handling and ethical best practices.
4. Facilitate data access to the identified personnel, whether it is limited to institutional research researchers or broadened to approved faculty and staff.
5. Promote communities of practice or collaborative research/reporting projects among users of the dataset. Connections across units and individuals help researchers share knowledge/ideas, thereby enriching the equity-focused learning analytics landscape. This phase should not be considered the end-point, but rather an ongoing effort to engage available data to better the institution.

If you would like to see this kind of work happen at your institution: determine which phase of development your institution is in and begin planning for how to progress. Whose approval or resources are required to advance through this sequence? What can individuals do to help their institution get there? Ultimately, large-scale changes can be triggered by a single conversation.

While such an effort might be challenging and time consuming to establish, we believe the effort is worthwhile. In addition to its use for understanding and promoting equity in STEM courses, simplified access to institutional data can enable a broader set of analyses around student success in STEM. If higher education institutions are to fulfill their missions, regular progress checks at the course and departmental level in addition to the institutional level are essential. Providing straightforward access to institutional data and results to faculty and instructors are key to ensuring that can happen.

Footnotes

^a The barrier index is a combination of three binary identity barriers: Underrepresented/Marginalized racial/ethnic (URM) status, First Generation (FG) status, and low-socioeconomic status (SES). Each student is assigned one point for each identity they hold that is historically marginalized in higher education, creating a range of 0-3 barrier points. A barrier 3 student is URM, FG, and low-SES, while a barrier 0 student has none of these marginalized attributes.

^b  Our reports are produced through an R Markdown (RMD) document, with the code for generating figures and statistics wrapped into code chunks with suppressed display, and the main body of the text written in LaTeX. This enables us to batch-generate reports by feeding the course name/number in as a variable when we render the RMD.

^c For readers interested in launching such an effort at their own institution, we refer readers to Lonn & Koester 2019,⁵ which documents how the University of Michigan developed their learning analytics infrastructure. Unlike other institutions, the University of Michigan does not have a centralized institutional research office to help with these types of projects.

Acknowledgements

We would like to acknowledge Caitlin Hayward for her work in co-leading the Assessment Toolkit project and feedback on this article. We would like to also recognize the many contributions of the Assessment Toolkit team, W. Carson Byrd, Susan Cheng, Nate Cradit, Holly Derry, Rashonda Flint, Ben Koester, Steve Lonn, Rebecca L. Matz, Mark Mills, and Kyle Schulz to the ideas in this blog post and the work described within. This effort would not have been possible without their support.

We would also like to thank Tim McKay for championing the Course Equity Reports and pushing for them to have a broader distribution outside of the Foundational Course Initiative.