Equity in Data: A Framework for What Counts in Schools

[00:01] Announcer:

Welcome to Principal Center Radio, helping you build capacity for instructional leadership. Here's your host, Director of the Principal Center, Dr. Justin Bader. Welcome, everyone, to Principal Center Radio.

[00:13] SPEAKER_01:

I'm your host, Justin Bader, and I'm honored to welcome to the program Andrew Knips and Michael Savoy. Andrew has more than a decade of experience teaching students, leading teams, and coaching leaders in Philadelphia's public, alternative, and charter schools. He is an education leadership coach, executive coach, data consultant, and racial literacy trainer. Previously, he was a high school English teacher and school administrator who has presented at conferences such as AERA and NCTE and has published articles on blogs such as Edutopia and Education Post. Michael Savoy has 25 years of educational experience, including teaching mathematics at the middle school, high school, and college levels, working with community organizations on school policy, advocacy, and involvement, and working with K-12 teachers, teacher leaders, and administrators to improve the equitable education experiences and opportunities for all their students. He's the author of several journal articles and book chapters on educational change.

[01:02]

And we're here today to talk about their new book, Equity in Data, a framework for what counts in schools.

[01:10] Announcer:

And now, our feature presentation.

[01:13] SPEAKER_01:

Michael and Andrew, welcome to Principal Center Radio.

[01:15] SPEAKER_02:

Thanks for having us.

[01:16] SPEAKER_01:

We're very excited to talk about data because this is one of those topics that's been with us forever, but often our thinking about data changes, our sources of data change, legislation changes, pendulums swing as they always do. Why this book at this time? What did you see happening in the profession that prompted you to write this book, Equity in Data?

[01:36] SPEAKER_02:

What I noticed is a overwhelming focus on data and a simultaneous overwhelming focus on equity and those two things being completely detached. And I want to sort of bring those two pieces together because I saw opportunities for equity to be baked into our data practices. I think the book outlines sort of each facet of the school community from hiring teachers to professional learning communities to just simply when kids walk in the front door and how we can implement data practices in that space in a more equity focused way.

[02:05] SPEAKER_00:

Yeah, agreed. I'd say what I've seen over the years is a need to really focus on other pieces of data rather than those assessment scores that get people riled up so much and are so controversial. There are other measures within school that I think are more important and more to the well-being of our students and teachers and everyone within the school community that just don't show up in typical hard data. While we're thinking about how people are feeling accepted in schools and a part of that school community, that does not show up in data and that does not show up in how we think about our students and our staff. And so our ability to really think about equity in that data in a different way was something I think was very important at this time.

[02:48] SPEAKER_01:

I'm excited to get into that with you a little bit. As we've seen the importance of data rise over time, as we've seen the recognition in the profession that data is something we need to be fluent with, to be looking at, I've almost seen a rise in this belief that data has some sort of magical power, right? That if we just collect data and make pretty graphs out of it, then change will happen. And I can't help but wonder whether we've always thought about what we're going to do with the data that we collect. And of course, sometimes we collect data because other people want us to. There's a district survey or a state survey or a state assessment.

[03:21]

What do you see as some of the end games that can really play out effectively when we follow the approaches that you're talking about in your book? What do we do with this data? Because we know if we just collect it, nothing's going to happen automatically.

[03:35] SPEAKER_00:

Exactly. I've seen more schools have sort of those data meetings periodically and they sift through a ton of data and then don't know what to do with it. We've already moved on from the lesson or we just don't have the time to slow down to really understand what that data means and what to do with it. And so, yes, I think it's very important for us to think about data in a different way, to really be able to dig into what's under it. Those numbers are great, but what does that really mean for how a kid is experiencing school? What does that really mean for how teachers are experiencing their jobs and their work with students?

[04:07]

And so the ability to see that data in a different way is very important. For me, qualitative data is about a system that I don't understand, right? I think in order to count something, you have to really understand it. So in this education system that we're in now, we don't understand a lot, right? So I can't just put numbers on something. I really have to dig in and understand it at a level I have not tried before as an educator.

[04:30]

And I think that's what we're looking at here is what are the different things that you should be looking at that aren't just numbers, that aren't just phenomena that I can count, but actual human experience and the way they feel and how they come to us and experience the learning. All those things are important that we just often don't look at, but those are the things that we can change actually on a continuous basis. But when a kid got on a math score, I can't go back and change that.

[04:56] SPEAKER_02:

Yeah, the obsession with color charts, something I've seen a few too many times where you hear the conversation of, okay, so we got it to the yellow in this area. So we're doing okay. We got to move that to the green. All right, that's our next step. Kind of this meaningless commitment to continuous improvement that obviously is missing a lot of clarity and specificity. And so opportunities to get into actual error analysis when looking at student work, And looking at survey data, like employee engagement survey data, for instance, there's just so many opportunities to actually drill a lot further down into actual information that can drive next steps.

[05:32]

And so that could look like following up with subgroups within the school community. That could certainly look like individual follow-up. And I think ultimately it's this question of like, why do we listen? Why are we listening? to do something with that, right? Like we're taking in all this information so that we can follow up and provide the support and the responses that folks are looking for.

[05:49] SPEAKER_01:

Absolutely. Michael, I wanted to go back to something you said that I think is just critically important, that we can't really measure something or count something that we don't really understand. And sometimes we have these quantitative measures. We have district surveys, state tests, things like that, that were designed for some purpose that might not be quite the same purpose that we have in mind today. Talk to us about how qualitative and other approaches to understanding the reality that we have can help us ask better questions. And then ultimately, if we need to collect quantitative data, do a more focused job of that.

[06:22]

Because I agree completely with what you said there.

[06:24] SPEAKER_00:

If we're going to count it, we have to all agree on what the thing looks like. And I don't think anyone in education, if you say, what does student belonging look like? What does someone who's sitting in class really engaging in classroom practice and learning really look like and be able to have everyone see the same thing and count the same thing? That happens so often when we do walkthroughs, administrators do walkthroughs, things like that, that they don't sort of norm on that or think about what biases may come up as they are looking at a certain situation or how kids are engaged. And so I think those are really complex areas in which we want to say a teacher is successful in their classroom. And yet everyone who is looking at it can't agree on what that looks like.

[07:09]

And so you have to have deeper conversations, which takes time to slow down, right? And you have to have that norming, that understanding of how your identity walks into that room, how your biases walk into that room with you, how you're looking at everything within the system and even the numbers that you're looking at. Being able to slow down and walk back and go, okay, so what am I looking at? Is this really where we want to go? And what does that really look like when we get there? What process are we going through?

[07:36]

How far have we gotten? What do we need to do? What shifts do we need to make in order to get to that point? That does not often come from our analysis of the current data we have. And so we have to figure out, so what other evidence should we be using for this? And that takes time to think about what evidence is there.

[07:51]

What do we already collect? What don't we collect? What other ways in which we can observe the success of our students? And that takes time. And so we have to give people the time, resources, and space to really examine themselves, where they come from, and what they're going to see.

[08:05] SPEAKER_02:

One thing I'll just add is I think the source of the data is a really important question to explore. Oftentimes from a disciplinary stance, it's very top-down driven, right? These are the behaviors we are defining, looking for from students or looking to avoid from students. And so we're kind of monitoring and making the judgment call of what data we're going to gather as opposed to, Flipping that to students, students actually being the ones to tell us when there's been harm in the school, when they've experienced something or when they've noticed something or been a witness to issues in the school community. Similarly with teachers and coaching, how much of the coaching priorities are coming from a district mandate or a school priority that's been stated, as opposed to from the teachers themselves, you know, who are professionals and have the ability to name and are going to be way more invested if they have the opportunity to say, this is what I want to work on. This is an area that we can maybe collectively agree is a priority.

[08:53] SPEAKER_01:

Let's talk a little bit more, if we could, about how we can choose a type of data to collect, an area to focus on, and ultimately, hopefully, end up with something that we can influence. Because if we look strictly at outcome data, often outcome data doesn't tell us a whole lot that we didn't already know. In many cases, we have long-term trends, and we sometimes despair of finding things that we can actually influence. We look at systemic issues, we look at factors that are outside of our control, outside of families' control. talk to us about some of what schools are doing to find things that are going to have leverage that's actually accessible to them, right? Because, you know, we have all these big picture things we want to see change in our society, but at the end of the day, we've got to focus on things that we can actually move the needle on.

[09:36]

So what are schools doing data-wise to find those opportunities?

[09:40] SPEAKER_02:

Yeah, I mean, one thing that comes up is employee engagement surveys, which can be as complicated or as simple as you want. Anywhere from a couple of quick questions of like what's going well, what could be improved. And at one school in Philadelphia regularly does employee engagement surveys. Now, the last few years, they collect the data. They sit down with the entire leadership team, which includes a bunch of teachers and teacher leaders in that space, and they collectively analyze the data together. I saw a principal go from sort of that close the door, like, I'm the boss, I'm making all the decisions on my own.

[10:13]

That approach totally shifted when they started using employee engagement surveys because all of a sudden again we're listening to people right we're actually responding to these needs and we're coding and we're analyzing trends and we're looking at this through a data lens but we're also able to pull out individuals within those data sets of saying oh that person needs some follow-up or that's an issue that nobody else mentioned but that we hadn't thought of either and we can follow up on that there's some immediacy there and there's some localization as well to those the issues that the school can follow up on

[10:40] SPEAKER_00:

Yeah, I would agree. I think there's nothing more powerful for a teacher than to hear the honesty from a student in their classroom. We assume quite a bit across the board, whether it is a teacher and a student or an administrator and their staff. I think we assume quite a bit the symptom of whatever it is, as opposed to the actual reality. cause or the challenge that's happening. And so actually hearing from people, and I know like student voices is big these days, right?

[11:07]

But I've seen the power of a teacher who feels like they're doing well. And then they talk to their students in a very honest and open way. And it's like, I didn't even realize that that was something that was detrimental to you or something that was still a challenge for you or something that I really could do differently tomorrow if I just knew that that was the challenge. But we, again, look at certain pieces of data and just make assumptions off of that, often negative assumptions, not asset-based assumptions. but often start in the negative and not really. So what is the challenge?

[11:38]

What are they experiencing as a student, as a teacher in the school? How are they experiencing it? And if there's any one little shift that I could make that could change the whole experience for someone, we don't often ask that question. We don't get that straight from them. And so finding those other opportunities to do that, again, that takes more time. But finding those opportunities to really engage people at a different level really can lead to some concrete next steps that actually do get to the solution.

[12:07]

Oftentimes, our measures for things are lagging indicators instead of leading indicators, right? We're looking at assessment scores. We're looking at things down the road. When I could change something tomorrow, if I just found what is the actual leading indicator that's going to now get me to that lagging indicator down the road. But if I don't look at that, then I don't know what to change immediately. And that's kind of where we need to get to is what are small things that I can do that are really going to get me larger wins down the road.

[12:34] SPEAKER_01:

Let's talk more about listening to students. To what extent do students know what they need and know how to advocate for what they need? Or how do we kind of tease that out? If students have a particular take on maybe something they don't like, what's been your experience as far as how much you get from students about what to change, what to do differently?

[12:53] SPEAKER_00:

I think kids at any level can, any age can express a challenge or something that is frustrating them. And then we can do something about that. I mean, obviously the older the kids get, they have very powerful voices and can really name the thing. But there are ways in which you can, again, have conversations with students. at all levels to really get to whether or not I'm just sitting in the wrong place or I need a little more time on this or my mind starts to wander when I'm sitting here for 10 minutes. So if I had a five minute break every once in a while, I could do much better.

[13:25]

There's things that come up that if you just ask a student what's going on with them, And I say this because it happens with my own children. And I'm thinking about how do I help support them? And I've had to really go, okay, let me just ask them, what's going on here? I have been amazed by the things that they say as what is their challenge? And then like, oh, well, that's not a hard one for me to do a little something differently. Now, if I was trying to do that all over the place and just guessing on what that may be, that's different.

[13:50]

But just asking, you're right, you have to sit a student down and build that trust and then say, so what's going on? And it may not be their teacher. It may be, hey, another teacher has to do that. Someone they trust within the school, have them sit down with them and ask these questions about what's going on with them. In that relationship, you can start to tease out what are their challenges and what are the things that, again, would completely turn things around for a student that we may not ask.

[14:15] SPEAKER_02:

A lot of the data work, especially on the qualitative side, is about trusting the source. And even if that's somebody who you wouldn't normally trust, right, or you don't share identities with, or you think is too young to know, right? I think so much of our obligation when we think about education leaders as data gatherers and data practitioners is about trusting kids and believing that they know what is best for themselves. One of the things we talk about in the book is the difference between a deficit lens and assume the best lens and a gifted lens. And that deficit lens, certainly we probably all know what that means. And that's kind of thinking the worst of kids, but the assume the best lens is sort of this like weird ambivalent kind of like, okay, maybe they know versus the gifted lens is all about like, let's go find out what makes every single kid amazing and what incredible gifts they're going to bring.

[15:05]

And I think with that lens of data gathering and data conversations with kids, we unearth a whole new set of potential possibilities for our engagement with them.

[15:15] SPEAKER_01:

I'm thinking about the idea of students who are eager to share or who have some, you know, something that's obviously not working for them versus the students that we might tend to overlook because we don't hear from them. They're not showing up on any presentations. early warning systems, you know, they're just kind of quietly there. What can school leaders and teachers and counselors do to make sure that we're hearing from students who might otherwise become invisible? We can look at kids who are getting suspended. We can look at kids who have failing grades, but often we overlook kind of everybody in the middle, right?

[15:48]

Everybody who's just kind of showing up, doing their work, not really saying very much. How do we make sure that we hear from a cross section of students who might have different perspectives on their education?

[15:58] SPEAKER_02:

I think a big task in this work is prioritization and thinking about who you're gathering that data from. And so, you know, we would certainly argue that like the data that's going to matter most in your school community is the data that's coming from minoritized groups. So part of the work I think is actually almost limiting the scope. Like the goal isn't to like figure out how every single kid is doing every single day necessarily, although we're always kind of working towards that, but it's more of a focus on like how do I actually reprioritize and shift some of my data gathering energy, if you will? So maybe that's a lunch bunch with students or a lunch with a principal. Maybe that's a targeted survey.

[16:32]

So instead of saying, hey, every teacher, you need to give a survey to every single one of your students. We all know that you're probably not going to have time to actually review that data. Give that survey out to your ELLs. Give that out to the students with IEPs. And then I think also just putting the power back in students' control when it comes to like a critical incidence tracker where students can report harm that's happened in the school community. If that's an accessible structure and process for students, kids will bring the data to us instead of us having to kind of extract it from them.

[16:59] SPEAKER_00:

I think it's a different, again, as we've said before, a different understanding of what evidence should be collected. If we're just collecting the same traditional evidence then the same traditional people will respond to that or give us what you said nothing new i think we absolutely have to open up in different ways and engage in different ways that kid who is spends a lot of time in the gym either the gym teacher or a custodian may be the one person that they connect with the most and are we actually engaging all of our staff in understanding all of our kids. And so you have to be looking for those in different places. Are you engaged in the cafeteria worker who every day knows what this one quiet kid wants and gives them the apple or what have you. And that's the person that can sit down with them and go, Hey, what's going on?

[17:50]

How are you doing? How are things with school? If we don't engage all of those individuals in caring for our students and caring for each other in the school system, I think we're missing an opportunity. Those are the sources that we do not tap into. We give a student survey. We give we have assessments.

[18:08]

We have these parent nights. And then why don't parents showing up? This one parent I can never get a hold of. And we're using traditional ways. Right. We haven't gone to their church yet.

[18:17]

and said, hey, how are things going? We haven't gone to different spaces that they're in and connected with them in different ways and had other people who, again, traditionally aren't the ones that we say are the ones we want to talk to. There's someone who engages with all of them. And we just have to figure out who they are and be open to finding that person. and inviting them into understanding and helping us be a better educational community. We have to define that in a different way.

[18:43]

And I think we're very strict with how we define the educational community. And we need to open that up in order to find those other sources of evidence and data that we're looking for.

[18:51] SPEAKER_01:

So I want to ask next about the kind of systems that schools can put in place to look at this. So this seems kind of like the type of thing that we could do casually, just as a way to increase our awareness, whether we're administrators, counselors, other education professionals. A lot of this sounds like on the front end, just being open to listening and to noticing where we might need to devote some more attention to listening. On the other end of the spectrum, it sounds like there are systems that we can put in place. There are processes that we can use to be a little bit more intentional and deliberate about collecting this type of evidence. What are schools doing on the more organized end?

[19:28]

What systems and protocols are schools putting in place?

[19:30] SPEAKER_00:

I would say one area we talk about is hiring practices and the traditional hiring practices that we have. Do we open the hiring practice up to a team of teachers or, again, community members who bring in different viewpoints, different experiences, different ways of seeing what is important and necessary for the school community. And so putting in a structure in which hiring you go through, you really think about your own identity and your biases, and you really think about, so what is it that we want to be as a school community? And what does that look like in an educator? And how do we find those, recruit for those in a different way and select? our teachers in a different way so that they are, you know, we hate the word culture fit, but they bring in, they expand our culture.

[20:16]

They add to who we are as a school community in a way that makes us better. That's often not the case, right? We have a checklist of, did you go to X school? Did you get this many credits? Did you take these classes? As opposed to who are you as a human and how you relate to each other, your colleagues and how you relate to students.

[20:35]

So that's one way of doing that. On the other hand, we have evaluation systems. that are very punitive as opposed to supportive? And how do we build teachers who may not have the skills that we think are necessary, but have other skills that are very pertinent to us building a great school community? How do we build those up and recognize those in our evaluation systems? They're just not there right now.

[20:59]

And so we have to sort of get away from the traditional hard line way in which we evaluate teachers and start to look at the other skills that are just as important again as a classroom educator and build on that what are their assets that i can that are they're contributing to our school community in a way that if i just need to help them with building a lesson plan that's a little easier than i need them to help them be a kinder gentler teacher those are some things that are harder to measure but are almost i would say even more important than some of the harder skills that we tend to focus on

[21:31] SPEAKER_01:

I remember 10 or 15 years ago, the Gates Foundation did a huge research project called the Measures of Effective Teaching Study, and they stole my advisor. So I'm a little bitter about that. But one of the things they found was of all the ways we have of measuring teacher performance, whether that's Principal ratings, whether that's test scores, value-added scores, all of that. One of the most reliable sources of evidence about teacher performance is simply student input. Students are incredibly good at telling us how their teachers are doing, and yet our process hardly ever involves hearing from students. The only time we hear from students with regard to their teachers is when we're in the classroom.

[22:07]

Hey, what are you working on today? And we're not going to get much of a perspective just right there in the middle of class. So I think that's very interesting to think about just the range of things that students can tell us if we're willing to listen.

[22:18] SPEAKER_02:

something Michael and I will probably resonate, especially with us, given that our work is grounded in a lot of teacher leadership development. So we've seen many, many, many schools lean into teacher leadership as a primary avenue of creating a culture of shared leadership in the school. And also when we think about kind of the source, right, if teachers are the source of conversations about students, as opposed to admin kind of facilitating or kind of Setting the conditions for teachers to be doing that work, it has more of a ground up feel. So thinking about what happens when you put teachers on your hiring committee, when teachers are joining on instructional rounds and classroom visits, when teachers are on the leadership team, when teachers are facilitating PLCs, all of a sudden you're blurring a bit of those lines of hierarchy in the organization. Right now, our education system is really over-evaluating and over-emphasizing evaluation when we're really struggling to get teachers to stay

[23:14]

and to come to the profession. And so we have like a pretty simple little math equation that we throw in there, which is just the pool of applicants that you have minus teacher attrition equals your intensity of evaluation. And so if you are struggling to get folks to come work in your school, and you're also losing a bunch of teachers, why are you evaluating teachers so closely? And I hate to put it that way, right? Because we want to have high expectations for the adults that are serving our kids. And at the same time, we know what turnover mid-year, what kind of an impact that can have.

[23:45]

And we know how long it can take for even a really talented teacher who is new to a school to sort of be fully integrated into that school community and feel like they can perform at their best. So it's a bit of a caution around that common source of data collection and how I think oftentimes that evaluation system is actually pushing folks out because of how rigorous it is.

[24:04] SPEAKER_00:

And we're evaluating without support oftentimes, right? I think a recent study came out from Brown and Harvard that said that one of the biggest factors for teacher success is coaching. Do they have the actual ability to grow? against their evaluation, right? A lot of times evaluations are used to push people out. We don't have that luxury anymore in the education system to push many people out, right?

[24:29]

We do need to really focus on the development and the learning of our educators in many spaces in many different ways, right? Including teacher leadership, including learning from each other, have the opportunity to engage in learning with and from each other. as the basis of your evaluation system, not the other way around. It should not be punished. Learning should not be the punishment for an evaluation. The evaluation should help you understand what it is you need to learn and do better with, and that's the way we look at it.

[24:59]

That's how we look at all that data. It should not be as summative as we make a lot of data pieces out to be. That is an opportunity to understand where gaps are, where challenges are, and how to do better from students to our faculty to administrators. That's how we want to see data is what are the opportunities coming from the data? What are the punishments that come out of that data?

[25:20] SPEAKER_01:

That's a really great point because we certainly are in a different hiring market. The profession is just in a different place and often it's a matter of developing the people we have rather than trying to quickly turn over the people that we're not thrilled with and just replace them with readily available people who of course are not there at all. We've really got to take seriously that challenge of inducting people into the profession and recognizing that that is not a one-week process that happens in the summer, but a long-term process over a period of years to really get somebody up to speed and keep them long-term. So the book is Equity in Data, a Framework for What Counts in Schools. Andrew Knips and Michael Savoy, thank you so much for joining me on Principal Center Radio. It's been a pleasure.

[25:59] SPEAKER_00:

Thank you so much.

[26:00] Announcer:

Thanks for listening to Principal Center Radio. For more great episodes, subscribe on our website at principalcenter.com slash radio.