How to Build a Better Academic Program Review

[music]

0:00:11.6 Speaker 1: Hello and welcome to Office Hours with EAB. Today, we hear how leaders at the University of Colorado Denver built a smarter, more inclusive approach to reviewing academic programs. An approach that centers faculty voices, uses data to guide decisions, and turns feedback into action. So give these folks a listen and enjoy.

0:00:38.9 Paul Gunther: Hello and welcome to Office Hours with EAB. My name is Paul Gunther and I’m a Managing Director with Strategic Advisory Services. I do a lot of listening in my role and I tend to hear a lot of the same struggles with similar issues. That isn’t to suggest that the solutions are cookie-cutter because, after all, each university operates within a unique set of circumstances with different budget constraints and with different set of mandates from their stakeholders. An evergreen topic receiving renewed focus is the process by which universities understand and make decisions surrounding academic programs to meet the needs of students, staff, and the surrounding community. University of Colorado Denver are with me today to talk about their process that they developed and iterated on over the past few years and are seeing great results. I’m excited to welcome Katie Linder, Joanne Addison, and Beth Myers to Office Hours with EAB. Thank you for being with us. Let’s start with some quick introductions. Would you mind introducing yourselves and sharing briefly what is your role with CU Denver?

0:01:43.3 Katie Linder: Thanks, Paul. Glad to be here. I’m Katie Linder and I serve as the chief of academic strategies under the provost’s office and support various large initiatives on campus, including change management initiatives.

0:01:56.1 Joanne Addison: Hi, Paul. My name is Joanne Addison. I am a longtime faculty member at the University of Colorado Denver. I’m in the English Department and also a longtime member of Faculty Assembly, our main shared governance body, which is important to the work that we’re doing here and was co-mentor on this particular project.

0:02:16.0 Beth Myers: Hi, Paul. Beth Myers. I am currently serving as associate vice chancellor for technological innovation and chief information officer. However, in my previous role, I oversaw academic planning and institutional effectiveness.

0:02:31.9 Paul Gunther: Great to be with all of you. Thank you again for being with us. Well, let’s jump in. Before we get too deep, there’s a lot of overlapping terminology and goals and outcomes with this topic. So, Beth, could you tell us about how CU Denver defines the difference between an annual program check-in versus a full program review, and then how those might differ from what most might consider to be a more traditional academic program review?

0:02:58.4 Beth Myers: Yeah, sure. So at the University of Colorado, at the system level, required by regent policy, we’ve had a requirement for program review that is required for every degree program at least every seven years. That’s actually been around, started in about the 1980s. And so that is something that we knew we would have to continue to have a seven-year review process for all of our degree programs, really with a goal of having high quality. And those are considered kind of a comprehensive review. A lot of times those are aligned with either professional or specialized accreditations as well. So many of our degree programs work with specialized accreditors, and those cycles are generally six, seven, or sometimes 10 years, depending on which specialized accreditor. So that’s what we would say is our program review process and is required by our regent policy. But we acknowledged that a lot can happen in seven years.

0:04:02.2 Beth Myers: And so we were asked, as part of an academic transformation initiative, to bring together a working group to investigate what it would look like to have a more frequent review of program viability and curricular innovation. And so from that initiative, we created an annual program check-in process, and we’ll talk more about that here today. So really what we’re thinking is the annual program check-in can augment a seven-year, more comprehensive review, but it’s something that helps us directionally along the way.

0:04:40.9 Paul Gunther: Anyway you define it, it’s a big job. Could you walk us through the goals and major elements that comprise an annual program check-in at University of Colorado Denver?

0:04:51.0 Beth Myers: Sure. So what I’ll say in terms of the big picture goals, the group really wanted to have an inclusive, systematic process to assess program viability and foster that curricular innovation I was just mentioning. So we wanted to have something that would allow continuous improvement that included existing data, policies, and some existing processes, although we’ve created some new process as well. Really, the group wanted to focus on a no-surprises and a transparency approach so that it was, kind of like I was saying, this inclusive process that included a lot of different constituents from across campus. The goal really is to position programs for long-term strength and the ability to adapt and change based on workforce, industry, and demand coming from our potential learners, and to make sure that our campus portfolio of curricular offerings was strong.

0:05:55.6 Beth Myers: In terms of some of the goals really within the process, we are hoping that programs get the opportunity to look at their strengths, their areas for improvement. They actually create goals and plans, so we’ll talk through some of that as well. In terms of some of the components, so our annual program check-in process includes some learning ahead of time. So we had a whole slate of workshops to help our campus community understand the data that was already available to them that could be used as part of this process. Then each program went through and created a short self-study. So that’s the other difference between a seven-year program review and the annual is that the annual is expected to be much shorter.

0:06:44.4 Beth Myers: So each program faculty and administrators work together to create a self-study. After the self-study, there were program action steps. Really, what would a program do in the next academic year or beyond to work on improvements or work on growth, those types of things? And then after the action step process, those were provided to the deans, and the deans would look at their whole portfolio within their college or school. And then the provost also had the opportunity to look at this at a campus-level portfolio review as well. So I would say those are kind of some of the key components of our academic annual program check-in.

0:07:28.9 Paul Gunther: Beth, that’s so helpful to hear, and I love how intentional you all were to think about the scaffolding that folks were going to need. Not to just say, “Here’s the data,” but to think about what was the upfront education that you did? How did you bring folks together to teach to use the data that was available? So you guys had a really thoughtful sequence to not just what data are we going to make available, but how are folks going to use it? Again, I know that this type of complex work doesn’t happen in a vacuum. It’s a lot of different folks working together. Joanne, I wonder if you could tell us a little bit about the governance structure and maybe share some of the formal and informal mechanisms you all used to solicit input as you built that process.

0:08:10.3 Joanne Addison: Sure. Paul, I have to say, it feels like we talked to every single person on campus by the end of this process. We really did a lot of engagement. We had two faculty leads and two administrator leads. And the faculty leads came from our faculty assembly, which I think is really important to let faculty know that shared governance is really an important part of this process. The members of the group either volunteered, an all-call went out to campus, they either volunteered or if we found a unit that wasn’t represented through that volunteer process, then people were nominated so that we made sure we had representation from all of the various academic disciplines, not every discipline, of course, but the various schools and colleges on campus and programs that might be outside of schools and colleges as well.

0:08:57.8 Joanne Addison: And at various points in the process, we met with stakeholder groups. So you meet with all of the deans, you meet with, we have a council of chairs and directors, just all those sorts of groups to try to solicit input. And this was all what we were doing as part of building the initial report. But really the input process is a very, has been, continues to be, a very iterative process. So even after people submitted their initial check-in reports, feedback was collected to find out what needs to change for the next time around. And so those are really the formal structures. A lot of formal structures were built in. Informally, and this is again where I would emphasize the importance of having shared governance folks in the room, informally, shared governance group folks heard a lot. And we would bring it back and we would say, “We find this legitimate concern over here, over here, over here,” and we would bring it back to the group as a whole and work through it. So that was one of the primary means through which we received informal input.

0:10:07.3 Paul Gunther: There’s just so much that comes back in a process like that, both on the metrics itself, some of the decisions that happen there, but as you mentioned, also on the process and that becoming iterative. Katie, let me bring you in a little bit. As you’re gathering all of that information, how do you start to synthesize all of that feedback, especially when some of the feedback could be contradictory? So, in other words, how do you ensure that the feedback is heard and respected while ensuring that it aligns to broader institutional strategy?

0:10:34.4 Katie Linder: I think that one of the things we really thought about, Paul, was how do we create feedback loops that are ongoing? Kind of like Joanne was saying, it never ends in terms of collecting the feedback. And having just a constant dialogue, I think, also helped us to reassure our community, and particularly our faculty community, that this was an iterative process, that it was in a bit of a pilot stage, especially in the first year. We were trying to learn what was going to help, what was going to be more challenging, especially as we were rolling out the data dashboards. Some people had different experiences with those because of being outlier programs or there was some kind of factor that wasn’t making sense, and collecting that feedback. But we also did a very intentional listening and learning process that Joanne mentioned at the end of the first year, and we released a public report with quotes from that process, synthesis of themes.

0:11:24.6 Katie Linder: It was shared with our provost to really try to articulate what we heard from the community throughout the entirety of the operationalizing of the recommendations from our original report. And now that’s being broken out in a few ways. One is really thinking about operational feedback. So there were maybe certain things in the process that it’s like a timing issue or we’re trying to understand operationally how something might change. And then there’s some things that we heard in the feedback that need additional discussion on our campus. It’s not really a decision that one person can make. It’s more we need to have some dialogue, we need to hear faculty voice, we need to have consultation with different groups to move that forward. And that will continue as we head into the second year of working through the process. But that recommendation report, we heard a lot of very good response to that, that people felt like it was representative, that it wasn’t biased in its, we weren’t just pointing out what worked, we were pointing out the things that people really had challenges with.

0:12:21.5 Katie Linder: And I think that there is an ongoing understanding that this is a process that we will continue to collect feedback on over time. So it’s not like one discrete moment of feedback collection and you’ve missed your opportunity. There’s a lot of access to leaders in this space to make sure that you’re getting your feedback to the right person who can actually hear it and potentially implement some of the things that you are offering feedback on. The last piece I would say is, of course, the feedback is contradictory. We had in the report, and that was actually part of what we wanted to point out to our community, is some people over here want this, and some people over there want the opposite of that. And it’s important for the community to see that this is complicated. It is not a simple process. All the inputs and outputs are very complex as we think about an entire portfolio of programs. And I think that that has also offered an opportunity for everybody to have a little bit of empathy as we’re trying to figure this out as a campus community, that it is not simple. And we’re certainly not trying to hide that, and we’re trying to surface that so that everybody can be engaged in a really deep way.

0:13:29.8 Paul Gunther: Katie, it’s so helpful to hear. And again, that was a place where you all were so intentional around it. It’s also one of the missteps I see other universities make is trying to come up with a single narrative that can explain that complexity. And instead, I think what you all did of making that complexity real and visible really helped to build engagement into the process rather than trying to work around it. Now, you use the word process and I use the word process all the time. We’ve worked together for a while, you know I’m a bit of a process nerd. So can you take us one step deeper? Joanne, maybe you can help to make this real for us. Tell us about the volume and the cadence of meetings that you have to hold with various stakeholders. Just from a pure… How do you manage the logistics of it to ensure that the meetings are productive rather than feeling like an intrusion? Can you spell that out for us a bit?

0:14:19.7 Joanne Addison: Yeah. The first thing I would say is that if you are really committed to campus-wide buy-in for this process, then be committed to the work, because it was a lot of work and you should know that going in. We held, just the committee alone held 27, 90-minute meetings over the course of about a year. And that was just the meeting time. That wasn’t reading and preparation for the meetings, and that was the committee as a whole. In addition to that, the two faculty leads and administrator leads met on the off weeks to try to make sure we knew what we were doing and where we were going. And this was all just to get to a point where, and there was a lot of behind-the-scenes work too. For example, Katie was doing a lot to help with drafting after we were meeting, and Beth was doing a lot to gather feedback and data after we were meeting. So there was a lot of support work behind the scenes that the committee members didn’t even see, even though they were meeting a lot.

0:15:20.3 Joanne Addison: And then of course, as we’ve said, this led to meetings with deans and other stakeholder groups and a pretty intense engagement phase, which was very well planned out from the beginning in terms of all the various steps of… It wasn’t just, “How are we going to get this generalized input?” It was, “Okay, we’re going to have this workshop on how do you work with the dashboards,” and then we’re going to get input on how that went, and then, “How do you fill out the template,” and we’ll get input about how that went. It was very purposeful. And the thing I would say about in this kind of process, it’s really easy to waste people’s time or to just be intrusive, as you were saying, Paul. And what I would say is that we were very, very goal-oriented. So I feel that every workshop, every meeting had a goal and it had some kind of outcome we were working towards. And if you don’t know that ahead, not that you can’t shift if you realize you’ve set yourself up for the wrong goal, but if you don’t know that going into it, what’s your goal, what’s your outcome for almost every single meeting, every single workshop, then you will just start wasting people’s time and nobody wants that.

0:16:31.1 Paul Gunther: Right. Saying that, “Hey, in this meeting, we want your input,” or, “In this meeting, we need to make a decision,” letting folks know what that meeting is going to focus on and what their role will be as part of it is really important. Let me take us to another component of this. As I think about University of Colorado Denver’s story and the work you did here, you were doing this amidst a lot of executive transition as well, right? New chancellors, new provosts. We know that some turnover among senior leaders and staff has just become more prevalent across higher ed in the past few years. In fact, EAB, in our institutional agility work last year, talked about executive resilience. How do you safeguard the really critical priorities for an institution to make progress on amidst some of that turnover? So, Katie, maybe you could tell us, how did you all think about adjusting to minimize disruptions from some of those executive transitions that occurred midstream throughout your process?

0:17:26.8 Katie Linder: Sure. I think we all know that one of the quickest ways to lose the trust of faculty is to collect all this information and feedback from them and then never implement it and have a process be disrupted. We’ve all experienced that on our campuses in various ways where we’ve put a lot of work into something and then nothing ever happens with it. And this was something that I think the three of us on this call and several other people who were involved in this initiative, we were very committed to not having that happen in this case because of the amount of work that went into this process and the need for us to actually implement it and make sure that we were serving our learners well. And so we did have a chancellor turnover, which led to the turnover of about half of our executive cabinet, including a provost. So we had our original provost who approved the recommendations. We had an interim provost who started to kind of launch the operationalizing of the recommendations. And then we have our current provost who has been overseeing the first year of this pilot of the annual program check-ins.

0:18:27.1 Katie Linder: And one of the things that helped us the most to minimize the disruption was a large amount of documentation of the work itself. So we had the recommendation report, of course, but we also had a bunch of other artifacts, and many of these are available on our campus’s website. This has been a very transparent, public process for us, and we wanted to make sure everybody knew what the steps were, what the workshops were, the process that we were going through. And so when we had various leadership transitions, it was relatively, I won’t say everything was easy, but we were able to pull together documentation relatively quickly and walk people through kind of the phases and the steps of what we had done. I also think there was a really large emphasis in this process from the very beginning on faculty voice. And that is very hard to argue against when you’re talking about curricular and innovation transitions that you’re trying to make on your campus.

0:19:19.8 Katie Linder: And I think that all of our provosts that transitioned during this period of time were incredibly grateful to the faculty who had put in the time and effort to really think about this intentionally and wanted to make sure, they were committed also to not having this be disrupted as we were moving it forward. Also just want to mention that we had a team of a project manager, a change manager, some operational folks who were kind of helping from the side. And this is a big component of our campus initiatives. We really try to lean into support for process orientation and make sure that we have those steps available for people. And so having that team sitting alongside the work was also kind of ensuring that we had that continuity and knowledge transfer as we were working through these leadership transitions.

0:20:03.7 Paul Gunther: It’s an incredible amount of work that goes into it. And I think that project management piece to support the overall effort is such a secret ingredient. Too often that happens as a side-of-desk job, but that’s really, especially when you think about the momentum you all were able to sustain, a lot of that is because of how you hardwired the process and the people around it. So I think that’s key. Fast forward a little bit. You now have, you’ve had heavy engagement with campus, you’ve documented that incredibly well, there’s been a set of recommendations that have moved along through various stages and various leaders. So then once those recommendations have been offered and decisions have been made, how do you then start to operationalize and manage the changes that get made in order to minimize potential negative impacts as well?

0:20:50.7 Katie Linder: So I think we’ve actually thought on our campus pretty deeply about the difference between having a working group making recommendations and then changing the phase of that to an operational strategy where you’re actually implementing recommendations and kind of signaling a change in that phase. And part of what we have done alongside that transition is to create an actual change management strategy around this major initiative. So I think both Beth and Joanne mentioned that we had a bunch of training workshops, and that was the first step of helping our campus understand that these recommendations are now being operationalized and we’re not going to force you immediately into the work, we’re going to show you the process that you’re going to go through in about six months, and we’re going to walk you through every step of that, get initial feedback on that, and really just kind of socialize and orient people to the actual steps of the process. We all know that not everybody is going to sit down and read a 40-page report in detail. So how do we start to translate that work into something that feels more operational?

0:21:52.8 Katie Linder: The other thing we did that I think made a really big difference in this particular recommendation report is we embedded a work plan into the appendix. So we actually did the work of saying, “These are the steps we think this will take and this is the timeline in which we plan to have it happen.” Now, it didn’t exactly match everything that happened, but it was pretty close. And I think that also signaled to the campus community, like, this is what you can expect to see. There’s not really a secret plan here. Here’s exactly the steps and tasks and the people involved and the timeline for that.

0:22:25.0 Katie Linder: And so anybody who was really paying attention was able to kind of see some of that signaled from the very beginning. I think it also allowed us to hit the ground running immediately when it came time to operationalize because we’d already done the work plan and embedded it into the appendix. So I do think that as we have gone through this process, there are some people who clearly have concerns. You’re never going to have a full 100% buy-in where everybody is just ready to come along with the major initiative change. But I do think that all of the change management components and the feedback loops that we built into the process have helped to minimize that as much as possible and to really give people opportunities and vehicles for their voice if they do have concerns and they want to raise that, and really to know that that is not going unheard. It can be built into the process and it absolutely impacts changes that we may make in the future.

0:23:17.8 Paul Gunther: I’ve used the terms… Oh, please, Beth.

0:23:20.3 Beth Myers: Yeah, I was gonna say, if I might add one thing which I think Katie mentioned, but I wanted to say I thought was really important, was a workshop specifically around the roles and responsibilities and really making sure that the various different people across campus understood who were making which decisions at what point. And I will say some of that even changed a little bit as we had some of those leadership transitions, but I think that was helpful to provide back to some of that transparency we talked about, making sure people understood how were they to engage with this process and what was their role in engaging with it and which leaders would be making which decisions.

0:24:00.0 Paul Gunther: Beth, that’s such a good point. And I’ve used the phrase “secret sauce” a couple of times, but I want to bring out something that I heard there. I think a lot of times I see institutions kind of stop with developing a rubric. In other words, they’ll say, “Okay, here’s how the information is going to be used,” but they don’t take that next step to say, “And here’s what that work is going to look like over the next few months.” And I think that’s one of those secret sauce elements that has made this successful for you all, is being able to say not just here’s how it’ll be used, but then also here’s how we are going to work with you beyond that. So I know that some of your team have participated in working sessions facilitated by EAB and that helped to serve as an input to some of this work. Beth, maybe you could speak a little bit to what do you feel like you got out of those sessions that helped you do the work better or more effectively?

0:24:49.3 Beth Myers: Yeah, for sure. So I would argue that CU Denver has had a little bit of a culture of being data informed. Prior to doing this initiative, we already had what we called our decision support toolkit that was a bunch of dashboards that brought a lot of program information together. That being said, there was never a process associated with the expectations for how people would use that information. And so one of the things that I found really helpful, I know Katie and I got to participate in multiple EAB sessions around this, was how we would tie things like the value drivers. Using that framework of value drivers that we heard from the EAB sessions, that really framed how we chose the various metrics that would be used to be part of the quantitative metrics that we used on our dashboard. And we already had all of that data in our decision support toolkit, but tying the data and the analytics back to a process was really helpful. And I would say that was something that helped frame the way that we did that. I think another thing that was really useful was just having the network of other people that were thinking about this and having a place to talk about it, to get insights from other places. I think that was really beneficial as well because there were some of the conversations that we were having within our working group and we could go to the broader group of people at various different institutions and get their insights and see what worked or didn’t work where they were at as well.

0:26:36.0 Katie Linder: We also took the opportunity to bring EAB in to work directly with our working group. And having that neutral third party present to the working group was also a really helpful level set for everyone I think as we were getting started. And just to give a little bit of context, when we first started this work, there had been a very public situation at another institution that had closed significant amounts of programs, and there was a general understanding that that had not really included a faculty process. And so it was very, I think, anxiety-provoking for a lot of campuses. And when EAB came in to present, they were basically like, “That may not be the way to do this.” And I think that really calmed down our working group to say, “We’re not trying to push a particular way of doing this,” and to give some neutral ways of thinking about how these kinds of processes might be built. And so I think that that was a really important partnership during this process as well.

0:27:34.7 Paul Gunther: That’s super helpful. I know we’ve only scratched the surface in terms of what goes into this work that you do, and I want to be respectful of your time. I’ve heard a couple of kind of secret sauce elements. One, there was very deep engagement, very intentional engagement with campus that led to strong documentation. And that documentation and the process that you put around it helped to build some of that executive resilience. But it didn’t happen by accident. You put really strong processes and change management structures, including project managers, around it to safeguard that work. And then you didn’t just stop with saying, “Here’s how the information will be evaluated.” You really taught folks around how the information was going to be used and what they could expect over the coming months. So I think those are three elements that really stand out to me as being unique to CU Denver, to the work that you all did, that I think would be helpful for anyone who’s engaging this work. But those were kind of my learnings. I want to make sure that we’re not missing anything. So before we go, what would you say are some of the most important things that you’d recommend that institutional leaders elsewhere might want to keep in mind as they undertake similar efforts? Katie, let’s start with you.

0:28:41.4 Katie Linder: I think there’s a couple of things here that I would point out. One is that it’s important and it’s okay to take your time. This is an initiative that will have, hopefully, a very long-term impact on our campus. And if it continues to have this kind of annual check-in, it’s actually changing the culture of what we’re doing at CU Denver around data, around programs, around curriculum, and really thinking about that carefully. And I think that taking that kind of long-term view, it’s you have to take the time to build that. As Joanne said, we had 27, 90-minute meetings. It took a lot of time and effort to kind of bring that out. And then secondarily, you cannot do this work without faculty. You have to partner with your faculty leaders and with faculty across your campus and ensure that faculty voice is centered in the process.

0:29:27.2 Katie Linder: And I think that trying to find all the different ways that that could happen, having partners like Joanne and our other faculty co-facilitators of this working group to give very specific feedback throughout the entire process has been an incredible asset to our administrative team and to our executive leadership to understand and to be able to tap into that feedback throughout. And the generosity of our faculty during this process to share what they’re learning and also what’s working and what’s not, we could not be more grateful. It has shaped the process in ways that have made it successful, have made it what it is today.

0:30:03.1 Paul Gunther: Katie, thank you. Joanne, what would you add?

0:30:06.2 Joanne Addison: You know, one of the things that I would add is that we weren’t content with just having faculty leaders being the voice. Too often faculty get put on these committees, but still somehow the large majority of faculty don’t know what’s going on and feel like faculty didn’t have a voice in the process. And so one of the things that we were really insistent on was that faculty at all levels, we find ways for faculty at all levels to own this process, right? And so one of the things that we did was in the template that department chairs or unit heads were filling out, one of the questions was, “Explain to us how you engaged all faculty members of your unit in the report that you wrote.” Like, we actually tried to hold people accountable for that level of engagement because we didn’t want it just to end up being overwhelmed department chair writes a quick report as a bureaucratic exercise because they don’t want to bother their faculty because they know their faculty are working too hard. And we all get that. But at the same time, this was such a big initiative that that wasn’t acceptable to us. And we really wanted those department or unit heads to explain to us how you engaged everybody to have some accountability there.

0:31:30.2 Paul Gunther: I’m just sitting and listening to that and thinking about the space that you all created for those voices to emerge. I think it’s easy to, when we try to get to an outcome quickly, to kind of miss that. And you all created some really intentional space for the community around it. Beth, what about you? Take us home. What would you tell us or what would you tell other leaders?

0:31:50.1 Beth Myers: Yeah, building on what Katie and Joanne have said, I would say as it relates to bringing people into this process, don’t only bring your champions in. Bring your naysayers in too. I will say we had a few, actually many naysayers on this working group. And what was amazing was when we really grappled with these issues, we didn’t get everybody all on the exact same page, but at least they understood the importance of what we were doing and they could go back and they could communicate that to their school, their college, their department. And I think that’s really important. So don’t just bring in your champions or the people, I think Joanne was alluding to, the people that are always the ones that get tapped for this work. Bring in a varied group of constituents. I would say that’s one point.

0:32:42.8 Beth Myers: Another one, and at the beginning I had mentioned that we were required by our regent policy to do seven-year program review. At the time, there was no requirement for us to do this annual process. If you have the opportunity, I would argue do this before it’s required. We are now at the point where now we have a regent policy that says that every one of the campuses within CU is required to do something. I would say our process is more robust than what is required from the regent policy now. But if you have the opportunity to do this before you’re being mandated or required to do it from your board or from whoever, I would say that’s helpful because now we feel like we got to create the process on our timeline. Like Katie was saying, take your time to do this on our timeline, engaging people as much as we felt like we needed to because we were not rushed.

0:33:40.1 Beth Myers: And the final thing that I would say is, again, I can’t not talk a little bit about data and analytics, but be data informed, but also don’t let people get stuck in the analysis paralysis. Because we got a lot of concerns across our campus around, “Well, that data doesn’t make sense for our program for this very specific reason.” And Katie alluded to some of this when she was talking about some of those ways that people were pushing back. And that can be true at the same time that you create something that allows a systematic review. And you can acknowledge that, “You’re right, this particular metric doesn’t make sense for your particular department, but we’re going to look at a grouping of metrics and across the board, those are the types of things that might make sense.” So be data informed, don’t get stuck in that analysis paralysis that means that you can’t move forward.

0:34:38.3 Paul Gunther: That’s fantastic advice. I know we’re coming to the end of our time, and so I want to thank you, Katie, Beth, and Joanne, for sharing your time and sharing your expertise with us today. As you mentioned, this is an iterative process. You set this up so it wouldn’t be a one-time thing, and so I hope that you will keep us close to some of the outcomes that you continue to achieve. It’s really thoughtful work that you all have built. I know that we at EAB are privileged to be a part of that story and are excited to continue to help celebrate the outcomes that you all achieve together. So thank you for your time and for your great work on this.

0:35:13.6 Katie Linder: Thanks, Paul. We enjoyed the conversation.

0:35:15.5 Beth Myers: Yeah, thank you for having us.

[music]