How to Manage Student Success Data

[music]

0:00:11.6 Speaker 1: Hello and welcome to Office Hours with EAB. Today’s episode centers on the technologies that are giving universities more efficient access to the institutional data they need to manage student success efforts. Our guests do their level best to explain how data warehouses and related technologies work, why data governance matters, and other pretty technical concepts in a way that even non-technologists can understand and appreciate. So give these folks a listen, and enjoy.

[music]

0:00:46.5 Devin Jones: Hello, and welcome to Office Hours with EAB. My name is Devin Jones, and I’m a strategic leader with EAB’s data and analytics team. One of the coolest parts of my job is that I get to see how the leading colleges and universities in the nation are using technology and data in really innovative ways to transform their institutions and improve the way they serve their students. Using technology and data wisely is important to every business, and no matter what industry you look at, individuals and enterprises tend to be distributed along the spectrum with early adopters on one end, and late adopters on the other. Colleges and universities are often accused of falling largely into the late adopters camp, but that’s not always true.

0:01:25.2 DJ: Today, we’re gonna talk about the technologies and best practices used by innovative institutions to improve students’ retention and help more students earn their degree in a timely manner. And we’re going to try to do it in a way that’s not going to put anybody to sleep, and doesn’t require listeners to be as big of a data nerd as I am. Joining me in the conversation today is someone doing really effective work in this area. Matt, would you mind introducing yourself and telling us a bit about your institution and your role there?

0:01:51.1 Matt Nickodemus: Sure. Thanks for that very generous introduction, Devin. So my name is Matt Nickodemus. I run the Institutional Effectiveness Shop at Utah Tech University. Utah Tech University is an open access polytechnic located in Southern Utah, and we’re about an hour and a half north of Vegas, and about 45 minutes south of Zion, and so we’re in a really gorgeous part of the country. We’ve got a lot going on here, great weather. This is my plug to try and recruit anybody who’s looking to attend university. You should come down to Utah Tech. We have a lot of really exciting stuff going on. We’ve been using Edify and Navigate since around 2019. We’ve had just a tremendous amount of growth as a university over the past 10 years. I think we’ve nearly doubled our student body in that period of time.

0:02:49.3 DJ: Wow.

0:02:49.4 MN: We’ve grown our programs by about 200, so just a really great place to be, great weather, lots of exciting stuff happening at the university. My role at the university is largely around data, data governance, and strategic planning. Our Institutional Effectiveness Office sits in the office of the president, and we help facilitate his vision of what the strategic plan is university-wide. And then we also run the majority of the data processes. So data governance runs through our institutional effectiveness office. We run the data warehouse, and then we handle a lot of the just strategic organization around data, which is… That’s a lot. That’s a large portfolio for our little office to tackle.

0:03:36.0 DJ: Right. But welcome to the podcast, Matt. So excited to have you on. But you mentioned it before that you’ve been working with EAB with a Navigate and Edify product, and I know you actually led a session at our Connected24 conference last month to explore the ways that Utah Tech is using technology and data to measure and evaluate your student success efforts. In fact, you actually won the award for the Technology Pioneering award at Connected. What would you say was the essential premise of your Connected presentation, and what did you hope the audience might take away from that session?

0:04:08.1 MN: I think the biggest topic for our session that we really wanted people to walk away with was that student success really requires university-wide effort and really effective data processes. The marriage between really effective advising and really effective data processes is really potent at improving student success outcomes. We’ve seen that at Utah Tech, we’ve increased our fall to fall retention rate by about five percentage points over the past five years, and that’s been a lot of things. We’ve introduced a lot of courses aimed at students when they’re first coming in in order to acclimate them to the university and to give them kind of an idea of what’s involved in the University Education. We’ve got a lot of students who come to Utah Tech who are… They don’t come from a background where their parents attended college, so this is all new to them, so we wanted to acclimate them.

0:05:09.8 MN: The other piece that I think we really tried to get across in that talk is that marrying your data team and your student success team is a fantastic idea. As an institution, if those two groups are working together really effectively, you can expect really great outcomes. And then tying together the technology that those two teams utilize is another really, really good idea. For us, that’s Navigate360 and Edify, we’ve had some really, really good successes in tying those two products together. It hasn’t always been the smoothest road, but we’ve had some early successes and we’ve got big plans for what we wanna do in the future.

0:05:57.9 DJ: That’s awesome to hear. You were saying you had some kind of early bumps in the roads. Really, I think it’s fair to say that Utah Tech’s been pretty much an early adopter of marrying, as you said, Navigate360 and Edify. Would you mind just kind of going through and maybe spilling the beans a bit on those big plans that you might have for putting those two products together? How does Utah Tech plan to inform these strategic decisions by combining these products around student success?

0:06:25.8 MN: So, our plan moving forward is integration and automation. We utilize Navigate360 as kind of our central communication tool. We have communication between faculty and students. We have communication between advisor and students, communication between faculty and advisors, that all occurs within Navigate360. We have Edify as well, and Edify serves as our data warehouse for the campus. So we try and organize all of our information within Edify and then communicate that information to Navigate, and then use Navigate to communicate information on an operational basis within Navigate. We also utilize a ton of tools to facilitate communication just around data. Communication around data is difficult for, I think… Almost every university in the country is just, they’re struggling with the amount of data that they now have access to, and how do they get everybody on the same page, and how do they effectively communicate the expectations around that. Our data warehouse does a great job of that.

0:07:39.4 MN: We’ve also worked really closely with the team at EAB to automate some of the communication that occurs between Edify and Navigate. I think that’s one of the pieces that we’re most excited about in the future, is starting to open that revolving door between those two data systems so that we can collect data and analyze data within Edify, communicate that data to Navigate, automate communication with students through Navigate, collect the feedback from the student actions based on that communication within Edify, communicate that back to Navigate so that we’re constantly trying to close that loop, so that we can most effectively communicate with our students.

0:08:17.9 DJ: Right. That’s a big workflow you just described as well.

0:08:21.5 MN: Yeah. It’s big and like I said, it’s still a work in progress. I like to imagine what we’re doing as reading a gigantic novel, so something like War and Peace. And we’re about two or three chapters in. We’ve seen this is gonna be a massive undertaking. There are a lot of characters in this thing. We’ve got a lot of work left to do, but we’ve been very happy with our results so far. Like I said, it’s work in progress, but we’ve made some pretty good strides, I think.

0:08:56.7 DJ: Yeah. And I think one piece that we tend to focus on when speaking about the data is the technology side and gathering the requirements, getting the reporting correct. But I think an unspoken piece is really about getting the buy-in from that cast of characters as you said. As you just mentioned this, this data goes through and is affecting a lot of different stakeholders at the institution, and it sounds like there needs to be quite a bit of buy-in to go ahead and make this vision come to reality.

0:09:23.7 MN: Yeah. Yeah. You gotta get everybody on the same page. I think one of the points that I’m continually making around campus is if you’ve got good people on your team, and you’ve got good products that your team is using, then good things are gonna happen. That’s our formula for creating a pit of success. We really try to work as hard as we can to make success the default. We’d like it if people are just stumbling around trying to do something, when we want them to stumble into success. And getting that buy-in is difficult on campuses. Everybody has what they’re trying to do, and large universities like ours have a wide group of people who are trying to do a lot of things, and getting them all on the same page can be really difficult.

0:10:10.9 MN: Getting them all on the same page with regards to data is also pretty difficult. So those are very challenging topics. Getting the culture of the university right is hard. That really requires effective messaging from your senior leadership. If they’ve got a really consistent vision of where they want to go with data, and they’re very consistent in who they feel should be on the team to lead that for the university, that makes all the difference in the world. We’re really entering kind of our second phase of that. We’ve had a run at that for the past three years, and like I said, we’ve had quite a bit of success. We’ve also seen a lot of things that we did wrong. So, in the second go around, we’re trying to fix some of that, and communication is at the top of everybody’s list of frustrations. So the communication piece is one of the most challenging.

0:11:02.2 DJ: Right. Thank you for that. And you mentioned this kind of in your answer just then as well, but also previously, the importance of having not only a large host of data at your disposal, but also the data governance and data lineage tools to go ahead and make sure that data is going downstream correctly, making sure people are getting reports they need. What is the value to your institution or really you think any institution in creating a data warehouse, and how do you go about managing the data governance issues that go along with all of that?

0:11:31.7 MN: Yeah, so I’ve built a data system at a campus that didn’t have a data warehouse. I’ve built one at a campus that did have a data warehouse. And then I’ve built a data warehouse at a statewide level. And working without a data warehouse is an incredibly frustrating experience. I think the biggest benefit for a campus for bringing a data warehouse onto campus is that you can automate so much of your workflow. When I first took over this position, one of the things that was completely apparent to me was that we had a huge workload that was all dedicated to state and federal compliance reporting, and we weren’t gonna make any strategic progress at all until we were able to clear that up. And I think like every other university in the country right now, nobody was throwing positions our way, so we had to figure out how to utilize technology to do this.

0:12:33.3 MN: So we spent one year going through our state and federal compliance workload, and we automated that. When we were automating that, we used that time to build out our tables in our warehouse in order to make it so that we could report as quickly as possible, and to minimize the maintenance that would be required. So we put in a lot of structure around the tables that we report on, so that logically, we wouldn’t have to be thinking about all the calculations that we needed to do. Edify was fantastic at mapping out all of the build steps and things like that so that we could follow that logic. It’s also a fantastic tool. We have just one person on our campus who manages our entire data warehouse flow, so that we have just that single vision of what should be happening there, and then everybody reports off of that.

0:13:24.9 MN: We were then able to translate those gains that we made in automation to move our team into a much more strategic position. So we were focused more on strategic reporting, and we were able to take all of the automation tricks that we learned from the compliance reporting and apply those to our strategic reporting. So right now, we’re finishing up about a year of planning out how we report all of our metrics, how we define all of our metrics, how we communicate all of our metrics to campus. And we’ve gotten to the point now where we’ve got a pretty good inventory of all of our metrics on campus. They’re all enumerated. We’re assigning those two specific strategies that our campus is undertaking, and we’re building out the definitions for all of those metrics to communicate them to everybody. With the gains that we make in the automation there, we’re now gonna move into some predictive capabilities that we’ve been building up for a while. And I think this is the real benefit to a campus of having a data warehouse, is it allows you to automate and then take the time savings from that automation and apply them in a direction that you’d like to go.

0:14:30.7 DJ: Right. Save some time on kind of the digital manual labor, so to speak.

0:14:34.7 MN: Yeah. The drawback of that is, once you start automating all of that, you really are looking at the health of all of your data systems around campus, because you’re ingesting data from all over the place, from your LMS, from your student information system. We ingest data from everywhere. All of our student clubs go into our data warehouse. And when you start to do that, you can start to see where your data systems are unhealthy. We also do a ton of snapshotting and daily reporting. So we’ve gotten to the point now where we can have somebody who’s involved in a data process, who doesn’t know where that data point that they’re creating is going the next day in front of our senior leadership.

0:15:15.0 MN: And then we get a report in front of senior leadership, and they call me and they say, this number moved, why is that? And then the next thing, I’m on the phone with somebody who’s like, what, why are you calling me? So that raises all kinds of data governance issues around process change and who’s inputting the data. And if somebody decides how they’re gonna record data, what effect does that have. And that kind of communication, I think is… Because kind of foreign to a lot of universities, just communicating a lot about the minutia of how your data processes work pushes you into data governance in some ways that are difficult. It causes you to have difficult conversations.

0:15:56.3 DJ: Sure.

0:15:57.8 MN: The benefit of having a data warehouse though, is you can comb back through all of your reporting procedures and you can see exactly where those data points are entering in your report. And you can see all the downstream reports that are running off of it. So it’s a really effective tool because it allows you to monitor the health of your data systems at a really granular level, but it pushes you into data governance, and some of those conversations are difficult.

0:16:25.7 DJ: Sure. Yeah. Data governance conversations are largely uncomfortable, probably whatever industry you’re in. And so it’s almost more of a philosophical question, right? Is opening pandora’s box more or less uncomfortable than maybe frequently reporting incorrect data, so to speak. But I like what you said there is that. There’s a few things I really picked up from your responses then. One is that it pushes you to go into the data governance. If you’re able to go ahead and see this data centrally stored in a data warehouse and you see things repeatedly coming in incorrect, it makes you have those maybe tough conversations that otherwise one you wouldn’t, well, really, you wouldn’t even be aware of, I guess if you had not been able to see that.

0:17:09.1 DJ: But two, it also, it sounds like what you said is maybe allows senior leaders to go ahead and focus more on the next strategy, the future of the institution or kind of next project we wanna do, rather than just kind of constantly playing catch up. And I think a lot of conversations that happen in the student success space recently have been trying to shift away from being reactive to being proactive. You mentioned some predictive modeling in the future. How has, I guess, working with this governed data warehouse allowed your institution, especially with Navigate as well, to go ahead and begin being proactive on the student success front?

0:17:48.0 MN: Well, I would say that’s where we’re getting to now. We’ve built out a lot of things. We’ve built out a lot of our capabilities. We can analyze a lot of our data processes, and we’ve exposed a lot of issues. Those issues have gone up to cabinet, cabinet has had conversations around those issues, and then they’ve come back and said, we need these fixed. How should we do that? So I feel like where Utah Tech is right now is we’re making that turn from mostly reactive reporting and trying to identify problems to, we’re now starting to predict where we’re going to be. And Cabinet has really nice controls over how we can integrate data and strategies in order to achieve future strategic outcomes. The way that we’re tackling that is we have a lot of student data, and so now, we wanna start building out predictive models and then saying, can we get return on investment from those predictive models?

0:19:00.4 MN: We’re also standardizing, like I said, all of our metrics, getting everybody on the same page, and building that measurement process into our strategic planning. We need to go through a couple of cycles of closing the loop on that. But we’re, I think we’re right on the cusp of having a unified data process around our campus where each specific group is working on a very specific part of an overall strategy that Cabinet has set. And then we can collect and aggregate all of that data from each of those individual strategies. And we can start saying, this has worked really well.

0:19:38.0 MN: It impacted over here. Two years from now, we’re gonna be at this point, and these two decisions that we’re gonna make today are really going to have an impact in two years, and then we’ll be able to see that impact. So we’ve kind of built out that planning tool for our senior leadership and now we’re starting to implement it.

0:19:56.4 DJ: So really moving from… If I were to, from what I’m hearing, putting these phases, Phase one is actually doing the dirty work of getting the data warehouse there, cleaning up the data as best we can, phase two of really understanding what processes have been and have not been working and how we can kind of improve upon those. And now what I’m hearing what you’re saying is this kind of next turn in this phase three is going to be… Now that we have these baseline things in place, what can we do to go ahead and try to expand upon the future of student success in Utah tech?

0:20:26.1 MN: Yeah, I think a really concrete and a really good example of that is the work that we’ve done around the student plans within Navigate. So we recently exported all of the student plan activity from Navigate into Edify and we’re able to analyze, what are the specific course requirements that are tripping up our students? How many students are planning to take certain classes in the next year? We’re going to combine all of the student activity data that we’ve collected to start making predictions about which students are gonna pass which classes. So within the next probably two years, we’ll have a pretty concrete process where we can say, “We are predicting that two years from now, we’re going to have a course enrollment in this course of this many students, and we should start planning for that now.” We don’t like to get to a position where we can start looking at our incoming class and saying, “For our incoming class, we’re predicting that we’re going to have a lot of students who are going to struggle with writing, so we really need to populate the Writing Center more than we have, in order to prep for this incoming class.” Obviously we have to test all that and make sure that we’re making good predictions, but those are the kind of things that we’re to get to.

0:21:42.9 DJ: And that project in particular will go on and serve faculty advisors, serve a series outside of academic departments as well, like you said, the writing center, the tutoring center, things like that as well. You’ve talked about freeing up opportunities for senior leadership to think more on the strategic side. You’ve talked about your team moving towards an area where we’re getting things more automated and just making sure things are running correctly and smoothly and building upon that process, but no longer… I guess the running around and trying to decide, “Is this the right way to do things as much… ” Still some sounds a little bit like that a little bit, but for the most part, we’re getting things more solidified. What would you say is the really kind of strategic impact from the different [0:22:33.4] ____ of the institution, by being able to do that, I mean, how would this go into effect the day-to-day of an academic advisor or a faculty member teaching the Intro to Biology course, for example.

0:22:45.1 MN: So on that front, Navigate360 is where they’re probably going to see that on a day day basis, so we have were… We’re rolling out a lot more robust version of our early alerts, and then we wanna tie that with some predictions that we can make off of the robust data set that we have within Edify. So one of the things that we’ll be doing is taking that early alert process tying that in with the predictions that we’re going to be able to make off of the data we’re collecting through Edify and then communicating that back to an advisor. So the advisors can start to see, “Hey, we’re predicting the students to have some difficulty they might need you to reach out to them.” We want to do this in the least intrusive way for everybody, and we wanna make sure that we respect the privacy of the students, so that we’re not communicating sensitive data to people that don’t need it. But at the same time, we are really laser-focused on student success and we want to utilize every tool that we have at our disposal in order to ensure that our students are successful as they can possibly be. So those I think will start to impact our advisors and our faculty on a day-to-day basis.

0:24:05.1 MN: I think the other piece that our faculty will start to see is they’ll just have much more robust tools when it comes to planning, they’ll have a much better idea of… This is where we think we’re gonna be…

0:24:16.6 DJ: For course planning?

0:24:19.3 MN: Yeah. Course planning and things like that.

0:24:22.1 DJ: So in terms of your team then, we kind of touched on some other people, institution, but what would you say is next on the horizon for your team and your team’s ambition retrospect to how you’re using data technology at Utah Tech?

0:24:34.9 MN: So we’ve been really happy with this integrate and automate process, we’ve found a ton just savings in terms of human resources needed to run a process. So if we can standardize the data ingestion in Edify and then automate all of the work that people do largely through Excel files now. So if we can sit down with somebody and look at, this is a three-month process that I do and it’s all Excel-based. We’re really confident we can automate almost all of that, and when we do that, we save that person three months of time, and then that person is able to move into much more strategic activities so that they’re not just doing the same thing every single year, year after year. So integrate and automate, that’s a direction that we really wanna go. The other thing that our office is really focused on is getting ready for AI. The large language models that are coming out now are really impressive, and getting our institution prepped so that when those tools take off and become ready to go into production mode, we want to make sure that we’ve got all the data collected so that we do the prompt engineering, so that when we are in a position to train our own models, those models are trained using data that we’ve had to interact with students.

0:26:00.3 MN: So for instance, if we were to have AI advising, then we wanna make sure that we can feed the AI advising, the data that we’ve collected from Navigate on our interactions with our advisors with our students. So that we have a really custom interaction with the students. So we’re collecting a lot of that data moving forward, we’re also working with EAB in order to be at the forefront for report generation. We’d really like to get to the point where our campus constituents can sit down and say, I need a report that does this and we want to have that report delivered to that person, we want it to be exactly what they want, but we also want it to be correct. And that’s the thing that I think has us the most worried is when we start to generate dashboards through AI and things like that, we want the data that’s presented to be correct, and we want all of that to work seamlessly. So, we’re prepping for that. We’re also working a lot on developing packages. So packages are kind of a technical term, this is an idea from open source software development. So you develop specific functions and the specific functions do things that you would take a lot of steps.

0:27:25.2 MN: So imagine, you think if you’ve got an Excel file and you have to point and click and move things around and it takes you about a half an hour to do this, but you do it every single day, we can build a function that will do that for you, and that we want to put that together into a package so that everybody across the campus can use that package. We’re doing a lot of training for our staff about how do you use the packages that the IE office develops so that you can run data processes and you can automate things yourself. Then we’d like to share those packages with other universities, so other universities can get the benefit of the work that we’ve done. And we really are hoping that we can connect with other universities so that they start to develop their own packages that we can then utilize so that universities can start to leverage each other’s work. That’s a direction that we’re really excited about going. And then just kind of standardizing the language that our campus uses around data. When we automate things, a lot of us are doing the exact same thing, so if we can centralize that in one place, standardize a function and then have everybody use it.

0:28:28.1 MN: The benefit of that is, if everybody’s using it, then you can do cross training so that you’re using best practices across your campus. And if everybody’s using it, you can standardize the usage so that there’s… So don’t replicate work. So those are the kinds of things that we’re really hoping take off over the next four to five years.

0:28:52.1 DJ: I wish we had another half hour to go into any number of those topics that you just described, AI as you said, I know if you’re listening to this podcast, Matt is trying to recruit you to both your Utah Tech and the Open Source community as well. So please keep that in mind. But as I said, these are complex topics and we really only scratched the surface. But, before we go, what advice would you offer to others who may be starting down this road, maybe who are still at kind of that phase one, like we said. What do you think is the most important thing that they need to get right directly from the get go?

0:29:25.3 MN: I think the biggest thing is your data environment is a culture, and culture is hard to change, but it is also the most powerful thing at the university. There’s a famous quote, I don’t know who said it, but it’s, culture eats process for lunch. So if you establish a good culture, your good culture will guide all of your other processes. So when you’re getting started on the data path, I think the best thing that you can do is have some hard conversations with the senior leadership at the university and ask them, “What is it exactly that you want to happen? Where is it that you would like to go? And who do you want to be on the team to take the university in that direction?” Those are really hard things to do, but if you’re seeing your leadership can establish that early on, it takes away a lot of the conflict. Then once that team has their marching orders setting out to develop your data governance processes and your data culture as a campus, that really minimizes a ton of conflict. At the same time that you’re doing that, having a data warehouse is absolutely essential. So if you don’t have a data warehouse, I would highly recommend that you invest in one, and then build out the tools from the data warehouse, the automate and then save time, I think is a really good work plan to follow, but focus on culture.

0:30:56.1 MN: Culture is the hardest thing, if you get the culture right, then almost everything else will fall into the place. So that would be my advice is focus on the culture, get good data governance and get the right people on the team. Once you do that, then the pit of success really opens up and you’ll just find yourself falling into success after success.

0:31:17.4 DJ: I like that term as well, and that might be a good place to end it, falling on success after success. Matt, always enjoy talking to you, but I want to be respectful of your time, thank you so much for coming on the program today and sharing your thoughts.

0:31:28.5 MN: Thanks Evans, it’s always good talking to you, and let me know if you ever want me to come back.

0:31:33.0 DJ: Sounds good. Thank You.