How one university created a departmental dashboard with actionable metrics for student success

Common reasons for untimely completion

Faculty discovered that untimely degree completion primarily stemmed from three root causes: major changes, overcredentialing, and underenrollment. Non-academic reasons for not enrolling in certain courses included student employment, lack of awareness that a full course load entails enrollment in 15 rather than 12 credit hours, and poor planning around course availability.

UW Eau Claire also found that 25% of students changed their major at least once and 1/3 of students were over-credentialing by enrolling in unnecessary courses. The vast majority of students under-enrolled and failed to complete 30 credits during their first year and 60 credits by the end of their second year.

Resulting policy and curricular changes created one-time fixes for untimely completion

The degree audit and transcript analysis revealed a number of opportunities to address timely degree completion through quick policy and curricular changes. By allowing faculty to directly interact with the data and view the names of their students, contacts report that faculty were much more invested in the change management and reforms that resulted from the exercise.

Public Accountability Matrix (PAM) reports data on departmental performance

The state of Wisconsin mandates that all institutions publicize institutional performance on state-dictated metrics as part of a statewide initiative to improve graduation rates. Therefore, UW Eau Claire developed the Public Accountability Matrix (PAM), a spreadsheet that benchmarks performance at the department level.

Each department’s performance on student engagement, student success, faculty productivity, and economic development is publicly available for viewing. However, since PAM is reported to the public, and often by third parties, UW Eau Claire has very little control over how metrics like retention and completion are measured.

PAM raises faculty concerns regarding metric measurement and student connection

While PAM is effective in reporting department-level metrics, the metrics themselves do not directly speak to the student experience and do not provide an outlet for faculty and staff to improve those metrics. When PAM was first implemented, faculty voiced these concerns, claiming that PAM is not personalized and lacks a direct connection to the student experience.

In addition, faculty reacted negatively to the fact that PAM metrics are typically calculated by the state of Wisconsin and the Department of Education, leaving UW Eau Claire very little say in determining how metrics like retention and completion are measured. For example, the above metrics for retention and completion apply only to first-time, full-time freshman, which leaves transfers and other non-traditional students out of the equation.

Finally, the metrics in PAM are not directly actionable: while most faculty buy into the notion that the institution must improve first-year retention, it is very difficult for faculty to understand how they can help move the dial on a day-to-day basis.

SAM developed to create personalized and actionable metrics for individual faculty

While the degree audit and transcript analysis successfully instituted curricular and policy reforms to facilitate timely degree completion, the exercise did not solve the need for more actionable metrics for deans, department chairs, individual faculty, and advisors to act upon daily. The implementation of PAM paved the way for assessing more granular data on student success at the department-level, but the metrics were too broad to generate real behavioral changes.

To address faculty concerns and incent departments to directly participate in student success initiatives, the university implemented a more targeted version of PAM called the Strategic Accountability Matrix or SAM. The university began to roll out SAM in 2009 and fully implemented the matrix in 2011. SAM benchmarks departments on specific metrics that can be inflected by individuals within the department.

Instead of telling a department chair to improve 1st year retention, SAM measures departments on metrics including participation in mid-term grade reporting and student involvement in high-impact practices. The goal was to create metrics that are student-centric, shorter-term, and grassroots-level, while still focusing on initiatives that amalgamate to improve retention, progression, and completion. However, general cost, tuition, retention, and completion metrics were removed from SAM because the data is not actionable for individual faculty members.

Weighting of metrics signals departmental priorities and allows greater flexibility

While all departments are benchmarked on the same 18 metrics, each department weights the metrics differently depending on their strategic goals and student population. Department chairs, deans, and the provost negotiate to allocate a weighting of 0, 1, or 2 to each metric. These negotiations aim to ensure that each department allocates the weight that naturally fits the demographics, priorities, and specific characteristics of the program.

The weightings of metrics signal to faculty members which metrics are most important to the department during any given academic year. For example, lower enrollment majors like Philosophy and Religious Studies can weight the metric for “New Freshman Majors” at 0 while Nursing may weight the same metric at 2. The weightings display the priority level of each metric and identify areas where faculty can have a particularly high impact. Contacts affirm that allowing departments to weight each metric helped relieve faculty skepticism related to the impact that faculty and departments may have on any given metric.

Strategic Accountability Matrix metrics

Actual to expected performance ratio calculated to determine funding allocation

To calculate overall performance, UW Eau Claire created a ratio of actual to expected performance for each individual metric. The ratio is multiplied by the specific weight that the department allocated for the metric. The weighted ratios are then added to create a weighted sum or “score” for overall performance on all 18 metrics. Departments receive a portion of the $400,000 central fund, which is calculated by dividing the department’s score by the total score for all departments. Initially, the funding was reallocated from other areas within the division of Academic Affairs and has since become a permanent bonus pool.

The funding is allocated 50 percent based on the department’s FTE and 50 percent based on the department’s performance on SAM, to ensure that the funding does not disproportionately benefit smaller departments. On average, departments receive about $10,000 each year. The funding has no strings attached; department chairs and deans convene to determine the best way to invest the additional funding in order to continue to improve on the departmental metrics. In the wake of state budget cuts and very little flexible spending at the department level, department chairs welcome the additional funding to finance smaller projects and necessities.

Departments quick to react to visible performance gaps

SAM is updated annually and sent to department and dean’s officers as a PDF. In addition, faculty and staff can request to view their department’s performance at any point. The increased visibility that SAM provides to departments incents chairs to make the necessary short-term steps and investments to improve student academic and engagement outcomes.

Contacts report that SAM has increased collaboration between departments and heightened faculty awareness of student success outcomes. The Institutional Research department is currently developing an interactive dashboard version of SAM that can be updated each semester and is easily accessible online.

Iterative development of SAM encourages reforms based on institutional priorities

Each year, deans, department chairs, and the provost review SAM to ensure that the metrics are still actionable for faculty and pertinent to institutional goals. The metrics have changed as flaws in the model have been discovered or as performance goals for the institution are prioritized.

For example, when the metric for compliance with midterm grade reporting was added to SAM, faculty participation rose from 20 percent to 80 percent across the institution. The addition of this metric signaled the importance of mid-term grade reporting as a retention strategy, thus encouraging faculty participation.

Departmental dashboards foster lasting cultural change at the university

In addition to the local curricular reforms and investments in student support, the implementation of SAM has successfully created an accountability mechanism for departments to invest in student outcomes. The departmental dashboards clarify each unit’s role in contributing to institutional performance goals, which creates awareness of how all of the individual actions within each department add up to ultimate improvement or decline in student success.

Deans and department chairs have begun to include goals for these metrics in their annual budgeting, planning, and reporting processes to add an additional level of accountability and ensure that the metrics stay top-of-mind for faculty.

Results: Departmental dashboards prepare institutions for like move to performance-based funding

As many states have recently moved to performance-based funding models, institutions have scrambled to acclimate to the new budget reality. SAM has allowed UW Eau Claire to adequately prepare for a likely move to PBF at the state level. The departmental dashboards force faculty, staff, and unit leaders to adjust to a culture of evaluation and continuous improvement, preempting a potential top-down system dictate.