Characterizing Student Perspectives on Departmental Culture: The Student DELTA Survey

Traditional Student Experience Metrics

National reports have revealed that although undergraduate education has undergone significant reform, students are not graduating at rates that mirror the broader national demographics.^1,6 This suggests that not all student populations are being supported in ways that help them reach graduation. Traditional metrics used to evaluate student success occur at or near the end of a student’s journey, and typically focus on retention and graduation rates. Using metrics to evaluate success at the conclusion of a student’s academic career means that systemic improvements will not impact the graduating cohort or be tracked in real time. Methods for collecting actionable data about the current student experience will help institutions be more nimble in their approach to making changes that support students.

The Relationship Between Departmental Culture and Student Experience

Organizational change researchers have suggested that institutional culture can strongly impact student experience and should be considered when designing and implementing change efforts.^3-5,10 Culture includes the underlying values, beliefs, and assumptions that guide the more visible practices, policies, and behaviors.¹³ Culture shapes a student’s experience, from implicitly or explicitly communicating what is valued in a discipline and institution to impacting the policies that govern student behavior.

While some cultural features are consistent across a higher education institution, academic departments are thought to have a more coherent and distinctive culture.^10-11 Departments typically establish their own policies, and departmental values and assumptions are often influenced by their broader discipline. Since departmental culture is likely more coherent and bounded, more recent change efforts have focused on departments as a locus for change (e.g., Departmental Action Team Project, American Physical Society Inclusion, Diversity, and Equity Alliance, Effective Practices for Physics Programs). Thus, understanding a department’s culture provides information about what students are experiencing at a local level and provides a baseline for identifying areas and potential mechanisms for change.

Measuring Departmental Culture: The Student DELTA Survey

Since culture is often invisible and unspoken, it is challenging to investigate. Existing student surveys typically include items related to climate (e.g., psychological safety, sense of belonging), but no survey has focused on capturing a holistic perspective of departmental culture from the student point of view. The student DELTA survey (s-DELTA) presents a way for collecting quantitative data around departmental culture from students that can be used to advance diversity, equity, inclusion, and accessibility efforts.

Articulating Departmental Culture: Core Principles

We chose the Departmental Action Team’s (DAT) core principles⁹ as an articulation of ideal departmental culture. Although this list is not exhaustive, it is rooted in education reform literature. The core principles were developed to guide change efforts in undergraduate education and consist of the following:

1. Students are partners in the educational process.
2. Work focuses on achieving collective positive outcomes.
3. Data collection, analysis, and interpretation inform decision-making.
4. Collaboration among group members is enjoyable, productive, and rewarding.
5. Continuous improvement is an upheld practice.
6. Work is grounded in a commitment to equity, inclusion, and social justice.

The DAT core principles have also been used in several research studies to examine existing culture and the impacts of change efforts.^7,9,12

s-DELTA Structure

Each core principle is represented by 5 to 15 items that explore students’ experiences related to departmental culture. The items focus on either departmental practices (what the department is doing) or perceived departmental values. Students are asked to respond to each item based on their experiences within their current department and what they desire in an ideal department. We used a Likert scale with six options: strongly disagree, disagree, somewhat disagree, somewhat agree, agree, strongly agree. There was no neutral option, although “I don’t know” was an option for responses to the context of the current department only. An example of one item and its structure as seen through the Qualtrics platform is provided below.

s-DELTA Development Process

The development of the s-DELTA survey involved several phases and shares some structural features with our existing departmental DELTA survey.⁷ We drafted sets of items that related to the core principles and collected student feedback via cognitive interviews. The structure of the survey was also informed by Chan’s² work on composition models, which articulates the assumptions made when transitioning from individual survey responses to aggregated results as well as the development process used in the DCAT survey, a departmental climate survey that makes similar assumptions.¹⁴ We are writing up our complete process for developing and refining the student DELTA survey and will publish it on our website once complete.

Using the s-DELTA Survey for Equity

Data collected using the s-DELTA survey can be utilized in a variety of ways to provide insights regarding departmental culture. The s-DELTA is complex, so for this post we highlight one potential lens that can be used to understand student perspectives. We focused on items related to core principle 6 (“Work is grounded in a commitment to equity, inclusion, and social justice”) and examined the difference in response distribution across pairs of current and ideal department items (which featured the same item stems). For all P6 item pairs, there was a significant difference in the distribution of responses given for the current item of the pair and the ideal item of the pair (Wilcoxon Signed Rank tests, all p<.001). For every P6 item, respondents indicated that their ideal department would be more equitable in its practices and values, than their current department.

Disaggregating data by demographics can highlight differences in experiences that are lost when the data are combined. When disaggregated by gender, we noticed that women differed significantly from men in the distribution of their responses for 17 out of 50 P6 items (Mann-Whitney U tests, all p<.05). Within this set of 17 items, there are three pairs of current and ideal items that differ significantly between women and men (Figure 1). For these pairs of items, women tended to respond with a lower value on the scale (expressing less agreement) than men with respect to their current department. Women also tended to respond with a higher value (expressing more agreement) than men on these items, with respect to their ideal department. Across the remainder of the dataset, women’s responses differed from men for only one item of a pair (11 items) or did not differ from men (33 items).

Figure 1. Distribution of significantly different P6 item pairs for women vs men*
The pink circle and pink diamond represent the mean response to the item from the context of the current department, while the blue circle and blue diamond represent the mean response to the same item but from the context of the ideal department. *While we included other gender options (e.g., transgender and nonbinary), we did not receive enough responses for some of the demographic groups to be able to represent them meaningfully in this visualization.

With this type of visualization, one can quickly see the differences in what students say they are currently experiencing within a department as opposed to what they would ideally like to experience. For example, for the item “My department is transparent about how often harassment and discrimination within the department affect its members (students, faculty, and staff)” the visualization shows that there is a large difference between mean responses to the current department (indicated in pink) vs. ideal department (indicated in blue). This suggests an area where a department could focus its efforts on changes that reflect what students consider to be an ideal implementation of this value.

Implications

It is important to note that although we performed several rounds of refinement, the psychometric testing of the s-DELTA survey pilot data did not produce well-defined constructs and likely required a greater sample size to improve resolution. This means that the s-DELTA is currently more valuable as a way to explore local student perspectives on departmental culture, than as a quantitative research instrument. Researchers can also learn from the structures and item content we developed.

While the s-DELTA survey is focused at the level of the department, positive changes made to STEM departmental culture have the potential to impact students beyond the classroom and undergraduate experience. For example, responses to the student DELTA may reveal that students do not feel empowered in their undergraduate education experience (core principle 1). Changes made to help students take ownership and feel a sense of agency over their education can lead to a program that will attract and support a greater diversity of students, thus positively impacting the greater STEM enterprise.

The s-DELTA offers departments a method for tracking student perspectives over time, giving them a way to visually assess changes they are making to policies or structures within a program. With continued validation, the s-DELTA survey will be ready for use by department chairs looking to effect change in their unit, organizational change researchers who want to design ways to measure impact of interventions or change efforts, and higher administration who want to better understand students’ experiences across their campus.

In conclusion, departmental culture can have a significant impact on a student’s undergraduate experience. The s-DELTA survey represents an early milestone in supporting departments in gathering student perspectives on culture. We encourage institutions and departments to use emerging tools like the s-DELTA survey, along with existing surveys, exit interviews, and other tools, to integrate student input into departmental policy development and decision-making.

Acknowledgements

We would like to acknowledge and thank the students who participated in the development of the student DELTA survey and JoJo Yang and Jess Keating for invaluable conversations around survey development and validation. This work was supported by the National Science Foundation award 2021110.