Author(s):
Need: Interdisciplinary environmental programs (EPs) are increasingly popular in U.S. universities. However, EPs lack disciplinary consensus about core concepts and learning outcomes. Currently there is no concept inventory for EP content and instructors and administrators of EPs struggle to assess student learning outcomes due to these gaps. To address this need, we are developing a tool for assessing environmental students’ foundational knowledge and their ability to use complex systems-level concepts. Specifically, we will apply an established method of developing and evaluating constructed response (CR) questions to create a Next Generation Concept Inventory (NGCI). The NGCI requires students to construct their own answers, which can better reveal student thinking about complex topics and phenomena. Once developed, machine learning models can automatically evaluate students’ text responses. This project focuses on Food-Energy-Water (FEW) nexus as an initial context.
Guiding Question: Initially, we performed content analysis on collected EP course materials from a range institutions to identify shared objectives across programs and courses. Once identified, we began to investigate student thinking on these topics. We asked: What preconceptions of EP topics are held by undergraduates in EP courses? How do undergraduates engage in systems thinking about complex EP problems?
Outcomes: We conducted ~100 semi-structured interviews with EP students across a diverse range of higher education institutions, which we analyzed thematically. We found most students understood connections between food and water systems but struggled to connect energy systems to food and/or water resources. From our course materials and interview analyses, we identified four key topics for assessment prompts: Explaining connections between FEW, Identifying sources of FEW, Cause & Effect of FEW usage and Tradeoffs. Using prompts targeted to these 4 activities, we developed 3 sets of CR items using different environmental phenomena as contexts. To address content validity, we surveyed 8 EP instructors about the items and collected responses from 306 EP students across 7 institutions. We are currently developing coding rubrics to identify students’ scientific and informal ideas in CRs. Our initial findings from qualitative analysis suggest students understand environmental injustices committed against historically marginalized peoples to acquire FEW resources. We also found students often conflate forms of energy needed to produce food with nutritional forms of energy. When completed, we will have a free, publicly available NGCI instrument for key topics in EP, which includes machine-learning models to automatically evaluate student text responses to the instrument.
Broader Impacts: We have engaged a range of institutions in the project so far. Institutions were chosen as a representative sample of EPs across the US, including baccalaureate (4), master’s (3), and doctoral (3) colleges or universities. This is critical to ensure that findings and outcomes are applicable to undergraduates across the US. Further, we anticipate our findings from reviewing EP course and curricula materials will inform those making curricular and staffing decisions regarding college environmental science. Finally, development of a NGCI that produces CR items focused on complex systems will provide EP faculty with a valid and reliable instrument for evaluating student learning.
Coauthors
Chelsie Romulo, University of Northern Colorado, Greeley, CO
