Letting Data Lead: How to Design, Analyze, and Respond to Classroom Assessment

[00:01] SPEAKER_01:

Welcome to Principal Center Radio, bringing you the best in professional practice.

[00:06] Announcer:

Here's your host, Director of the Principal Center and Champion of High Performance Instructional Leadership, Justin Bader. Welcome everyone to Principal Center Radio.

[00:15] SPEAKER_02:

I'm your host, Justin Bader, and I'm honored to be joined today by Dr. Eileen Depka. Dr. Depka is an avid educator who's committed to student achievement and teacher growth, an internationally known assessment expert. She's the author of eight books, including Letting Data Lead, how to design, analyze, and respond to classroom assessment.

[00:36] Announcer:

And now, our feature presentation.

[00:38] SPEAKER_02:

Eileen, welcome to Principal Center Radio.

[00:40] SPEAKER_00:

Glad to be here, Justin.

[00:41] SPEAKER_02:

So, classroom assessment is, you know, as a principal, I believe, one of the key ways that we ensure that students are learning. And as you work with teachers, as you work with schools, what are some of the things that you see happening in schools, or what were some of the things you saw happening in schools that prompted you to write Letting Data Lead?

[01:02] SPEAKER_00:

I was being asked questions a lot about, okay, now that we have the data, what do we do about it? And teachers not quite sure once the assessment was written and once they had the data, where do we go next? What do we do? How do we do it? How do we find time? How do we involve students?

[01:19]

Those types of questions kept surfacing, which led me to want to do something with not only assessment itself, but the entire gamut from starting the assessment all the way to planning in advance what do we do about the data so that when we have it and we get there, we can respond immediately. It's not a matter of now what do I do, but there's a path directly leading from the standards to the assessment to the lessons to the what do we do about the data so that we can instantly get students back on the right track if they're not there by the assessment results that are found.

[01:56] SPEAKER_02:

And we should clarify, the kind of assessment that we're talking about here, we're talking about teacher-created assessments, is that right?

[02:03] SPEAKER_00:

Yes, for the most part, yes, we are talking about teacher-created assessments. Also, beyond the assessment world, the book also includes information about gathering attitudinal data from students and what to do about that, gathering information about college and career readiness, and how students are able to work in collaborative groups, getting information about that to respond to that kind of data as well in order to get students in a frame of mind that they can perform as best as possible within that classroom.

[02:40] SPEAKER_02:

So Eileen, I think we've become comfortable with this idea of data over the past, say, 20 years, and just the sheer number of assessments that are available to us commercially, the number of benchmark assessments and interim assessments and curriculum-based assessments. I often feel like we're kind of drowning in data But so much of that data comes from assessments that were designed by people outside of our schools, outside of our district, people who maybe were designing a curriculum or people who were maybe designing a standardized test. But in letting data lead, you're talking primarily about assessments that are designed by the teachers who will be teaching the students who are taking the assessment. What's the value in making assessment design part of teachers' professional work and not just kind of outsourcing that to people who maybe do that professionally or maybe vendors, companies that only do assessments.

[03:37]

Why is this part of teachers' work in your view?

[03:40] SPEAKER_00:

Well, there's a variety of reasons. I think the one that comes to mind first is that as teachers are designing assessments, they have to become very familiar with the standards that they're teaching. in order to design those assessments, the lessons that are directly related to the assessments that they've built. So there's a direct match between standards and the target they choose to achieve, but also the response time between that assessment and what a teacher can do about it is within a day, within an hour, within minutes, if they choose to. So the standardized tests, for example, The lag time between the time you get back data and the time you can do something about it is so broad that it might no longer be even practical to do that type of response, but perhaps into the next year. Where teacher-created assessments can be responded to immediately, providing students with feedback, providing students with clear and accurate targets about what's being done in the classroom and how they're expected to perform.

[04:44]

And then a response should students not understand. And that can be so immediate that it truly makes a difference in the lives of the students. So when teachers build their own assessments, that direct relationship between the assessment and the response is immediate and effective. Not that it can't be with standardized tests, but it is the lag time slows down the process.

[05:10] SPEAKER_02:

Well, I'm interested in this idea of data response because I remember when my school was implementing NWEA's MAP, which I think stands for Measures of Academic Progress, which is a computer-based standardized test designed to measure growth, but it does so in a norm-referenced way rather than a standards-based way or a curriculum-referenced way. So teachers don't know exactly what's on the test, they can't teach to it in any way, and of course we have lots of negative things to say about teaching to the test, but there's this sense that it's just kind of a black box, and if our scores go up we're happy, and if our scores don't go up we're not happy, but our ability to respond Because we don't know what's on the test and we're not specifically teaching what is on the test, or rather the test is not designed around our standards and what we're teaching, there's a limited ability to respond.

[06:02]

So take us into that idea of data response. What does that really mean to respond to data?

[06:09] SPEAKER_00:

In Letting Data Lead, I talk about the response to data really starting with the standards. It's starting at the very beginning. It's not starting at the end. So when standards are clearly identified and teachers are able to build assessments, whether formative or summative, around the standards that they're teaching, they're able to have that clear picture and understanding of exactly what they want students to do. So at that point, then, lessons are developed, obviously prior to implementing those standards, to help the students better understand what they need to in order to adequately perform the standards and content related to it. So the assessments are then given, but at that point, during lesson development, the idea in Data to Lead is at that point, while we're researching what we want to teach the students and how we're going to do it, what strategies are we going to use?

[07:04]

What materials are we going to use? What videos are we going to use in order to teach the content? At that same point, we can also be saying, but if they don't get it, what are we going to do then? So that response to data is built within the lesson planning framework. And then when the data occur and we need to know what do we do next, these students are either floundering or they're stellar in their performance, what do we do now? Well, we already have a roadmap planned because of our lesson design process.

[07:38]

Letting data lead also talks about when do we respond and when don't we respond. There are times that if the students, for example, understanding the difference between a mistake and a misunderstanding. Knowing that a mistake can be corrected by a student, a misunderstanding is something that we need to take a look at and say, if the student simply made a mistake and can do something five times, but that sixth time they had an error, they can correct that themselves. A misunderstanding, however, is something that if that is important to that content at that time, we need to correct misunderstandings. So letting data lead not only talks about how to respond, when to respond, but when not to respond as well.

[08:21] SPEAKER_02:

I wonder if we could get into some applications to specific subject areas, because when I think about assessment, I naturally tend to think about the subjects I taught. I was a science teacher. I taught sixth and seventh grade science, did my student teaching in chemistry, and conceptually, There were things that I knew students needed to understand in every chapter or in every unit before we moved on to the next one. What are some of the variations in how this process shows up in different subject areas? Because we think of assessment in terms of math, in terms of science concepts. tend to be a little bit more cut and dried.

[09:01]

How does data show up in some of the schools that you've worked with in subjects where it's not simply a matter of understanding how to do something correctly or understanding conceptions? excuse me, I said that in a clunky way, where it's not necessarily just a matter of misconceptions or knowing how to solve a certain type of problem correctly, but say language arts or social studies or other subjects where we do have very clear standards that we want students to master, but it's not just a matter of, you know, You know what I'm saying? Where we have to think through, you know, the skills a little bit in a little bit different way than you would in, say, a math class.

[09:46] SPEAKER_00:

I think that there's an overlap between all subjects. I believe with assessments that in some subjects that we sometimes consider cut and drag, we sometimes think that there's a right and a wrong answer. But that often is fact-based where process is important as well. If we're talking about something like science or social studies or actually any subject at all, if we go back to the standards, the standards really supply a really nice direct connection to what students are being asked to do. You will see verbs in the standards like do they know, you know, very fact related. Can they analyze something?

[10:23]

Can they create something? And those verbs lead us to the types of assessments that we need to build. So within any assessment, within any subject area, I believe that it's important that the assessment itself be set up in such a way that the data provides us with the information we need. By that, I mean that are there foundational skills that students need to do in order to perform what they're being asked to perform in the subject or know what they're being asked to discuss in the subject? So those foundational pieces might be more fact-based. They might be kind of lower level thinking skills.

[11:03]

But we want those in there so that the data tells us, for example, if they can't do something higher level, why is it that they can't do it? Why can't they apply the information that we're asking them to apply? What are their deficits? Because if we ask them simply the higher level questions and they do something incorrectly, they can't do it. We don't have the data supplied to us that says this is why we believe they can't do it. So, for example, in language arts, if we're talking about theme, for example, of a certain piece of literature, we might want to back up and ask some factual questions that guarantee us that students understand that piece of literature.

[11:44]

Do they know the characters? Do they know the setting? Do they know what happened in that piece of literature? So that they can go to directly say, okay, so what is the theme? Explain the theme. Tell us about then next, what is your basis of understanding?

[12:01]

What evidence do you have that that's the theme? And we might even then go to, if we were to add a new character to the story, how might that change the theme? What's the personality of that character? So the point is going from lower level thinking or questions to that higher level application will give us the data that we need so that, for example, if the student can't identify the theme, but they did know all of those factual-based questions about the characters in the story, the setting, the main events that took place, and they can't even identify the theme, that tells us something different than if they can't answer correctly those foundational questions. So I believe in any subject, by going through that type of process and going from what do the students need to have the background in, what do they need to know in order to perform well on this assessment, going all the way to the application understanding creation type questions that we have in any subject, it will help us get the data that we need in order to analyze and then respond to it.

[13:06] SPEAKER_02:

Beautifully said. And I think this is such an important point. You've kind of activated a Justin Bader soapbox here when it comes to lower order and higher order questions, like a Jeopardy Daily Double you just kind of stumble across. I see an epidemic in our profession of bias against lower order questions because someone somewhere is telling everybody that higher order questions are good and lower order questions are bad and not thinking about the necessary relationship between them. So I think exactly what you said is happening. We have students who are unable to do these higher order tasks in, say, analyzing a piece of literature, but if we don't know whether they're struggling with that because they didn't read the literature or they didn't understand some of the vocabulary in it, or if they're actually struggling with the higher order task itself, the way we respond to that instructionally is going to be very different.

[13:59]

And I think that's such a key point there that we have to understand the nature of the gap in students' understanding. And if we're going to respond to it, we've got to clearly understand what it is that we actually need to teach. And that sounds so basic when I say it that way. that we need to understand our students' misconceptions better. But I love the kind of foundational role that those lower order standards or lower order questions play in kind of diagnosing where students are. And Eileen, I wonder if you see this kind of development of a little bit of kind of a learned cynicism as we go to the higher grade levels and I think certainly there are lots of teachers at the secondary level who do fully believe that everything they teach is teachable.

[14:51]

But I think about kindergartners and kindergarten teachers. You get a new group of kindergartners every year, and kindergarten teachers truly believe that everything is teachable. They can teach you how to sit on the carpet and not touch other people. They can teach you how to write your name. Kindergarten teachers have an infinite belief in students' ability to learn and be taught what they need to know to succeed. But I feel like somewhere along the way we start to lose that and we start to think, well, I'm gonna throw some spaghetti at the wall instructionally and kind of hope it sticks, see what sticks, but that's gonna be up to the individual kid.

[15:28]

As I hear you talk about beginning with assessment design, beginning with thinking about how we're going to respond instructionally and what our students are, it gives me hope that everything can be teachable all the way through, that we're not going to be stuck at the 11th or the 12th grade level with the situation where students don't have the skills or the knowledge that they need, and we just think, oh, well, if they don't have it by now, they're not going to get it. It gives me hope that we can see everything as teachable if we approach it purposefully.

[16:07] SPEAKER_00:

Oh, I totally agree. And I think that when we look at data, when we think about analyzing student data, sometimes we think about just what they did on this assessment. So we're looking at the questions that they got right, things that they misunderstood and finding out why they misunderstood. But I think we also need to make sure that the types of questions we're asking when we analyze assessments includes what strategies did I use? What went well with those strategies? And perhaps bravely asking the students the same thing, saying, you know, during this lesson, these are the types of strategies that I'm going to use with you.

[16:43]

These are the types of ways that I'm going to help you understand the information that that is important to the standards and content that we're teaching, after you're assessed or while you're assessed or before you're assessed, I'm going to be asking you how those strategies worked for you. What went well? What didn't go well? So that it's not only about the student performance, but it's about my performance as teacher as well. What can I do differently? What worked this year?

[17:09]

What might I want to make sure that it's in my bag of tricks consistently? And what else might I want to try next time? or actually during any kind of reteaching moments, if these certain strategies that I used worked really well, I can continue to use them. If they didn't work well with a handful of students, what am I going to do differently to teach this content so that students understand? Because it's important, I think, for all of us as teachers to believe that if we're teaching content to students that they can and will understand it. The timeframe might not be the same for all students, but the possibility is the same.

[17:47] SPEAKER_02:

Well, I'm hearing hints of student ownership and metacognition, being aware of your own needs as a learner. And in the book, you actually talk about encouraging students to respond to data and kind of monitor their own progress, right?

[18:03] SPEAKER_00:

Absolutely. They are the key ingredient in the whole data analysis process, as well as the reassessment or reteaching process. They have to understand, first of all, I think that they have to understand that when they are being assessed, it's for us as teachers to really understand what they know and don't know and what they need to do in order to achieve at the next level of success within that course. within other courses that have the same type of content, the things that are important for them to succeed as not only students, but as people in the real world as well, they need to have that kind of buy-in. They're the key players. So if the data supports some kind of a reteaching event, how is the student involved?

[18:51]

What are their responsibilities? How are they going to, between the assessment and the reassessment, learn what they need to in order to be successful the next time. Because it's not about just retaking a test, redoing something. It's about the learning that needs to take place to help them be successful. So they are partners in that learning. How can they be engaged?

[19:13]

What videos might be available for them? How can they use the teacher? How can they use their colleagues? How can they use the materials at hand to help learn those key essential things that they have not yet had the ability to perform well on? So engaging the students is a key factor in student success.

[19:34] SPEAKER_02:

Well, and it seems to me that it's one of the most important transferable skills that we could possibly help students develop. Like this idea that I don't know something. you know, if I don't know how to do something, like we live in a world where no one has to persist in not knowing something for more than about five minutes, right? Seeing what kids can learn on YouTube, what you can look up on Wikipedia, what you can learn from a more in-depth online course. I mean, the ability to figure out, okay, what do I know? What do I not know?

[20:07]

How can I close the gap between where I am and where I want to be? I mean, that to me seems like the master skill for the 21st century. So I think that ownership is just so critical, even of things that as educators we traditionally have owned, right? Like we've seen the standards as ours, but you're talking about giving the standards over to the students and giving some of that opportunity to self-assess and respond over to students as well.

[20:34] SPEAKER_00:

Absolutely. I think it's teaching them a life skill, as you mentioned, It's something that we're only with them typically a year, maybe a little bit more. When they leave us, we don't want them thinking, well, I can't possibly know this because the teacher is not in the room. I have skills. I have ways to find out. I have access to information beyond belief, and I can do this myself.

[20:58]

I can be a self-starter. I can engage, and I can be proud of what I'm doing because I am learning. Certainly, With the core content, teacher guidance is essential. It's definitely important, but that student ownership piece takes them beyond the classroom into the real world and gives them skills that will be important to them for the rest of their lives.

[21:19] SPEAKER_02:

And speaking of beyond the classroom, but keeping our focus within the school, let's talk about the collaborative aspect or the common aspect, because I know a lot of what we've been talking about so far parallels closely the PLC questions. You know, how do we respond when students already know it? How do we respond when students Don't learn what we want them to the first time. What do you see as the chief value of developing some of these assessments collaboratively? What do you see teachers doing collaboratively with their data that you have seen to be effective? And what do you recommend that schools think about as they are going through your book and implementing some of the recommendations you have?

[22:01] SPEAKER_00:

Collaborative teams are so important. Two heads are better than one, three heads are better than two, and so on. Being able to rely on each other's expertise, being able to have the conversations about, so what do we do next? What went well? What should we change? Crucial, I think, to student success and so important when we can share as opposed to being in an isolated classroom.

[22:24]

One of the things that letting data lead promotes is, again, looking at data. When we're building common assessments, common doesn't just refer to the assessment itself, the things that the students are actually being asked to do. But one of the things or several of the things that I suggest in there are making sure that common is common with implementation so that our data is such that it's reliable and that we can actually compare it with our colleagues. So, for example, one of the things I mean by that is have we talked about the directions of the common assessment? Are they written? Am I going to read them to the students?

[23:02]

Are the students reading them themselves? If I'm going to read them and you're not because, not because you don't want to, but because we just didn't see it, we didn't talk about it as a collaborative team. My students have the advantage possibly because we have auditory learners. We have some students who might not read the directions. We want to make that common. Another thing that we want to talk about ahead of time are any kinds of tools that the students are being allowed to use.

[23:29]

Are they in your room going to be allowed to use computers or calculators in certain subjects? Or are they going to be allowed to use word processing programs? If they are in your room, we need to talk about that ahead of time so that it's open for all students. And otherwise, the data will not be the same. Do we have special education students who are used to using specific tools within the classroom? If those tools are available for my students, they need to be available for yours.

[24:03]

If not, the data will be skewed. Is there a time limit in your classroom and not in mine? Is mine open-ended? I mean, there are several things in letting data lead, but these are the types of questions that if we talk about them as a collaborative team ahead of time, perhaps even with the first assessments we give and then carry that on throughout the year, our data will be more comparable. Because if our situations are not comparable, our data will be skewed and we won't really be able to rely on it as much as we could have otherwise.

[24:34] SPEAKER_02:

Well, I think that raises a really critical issue here around accountability or around teacher evaluation, because anytime we have assessments that directly allow us to compare from one classroom to another, there is at least the possibility of using that data for accountability as part of the teacher evaluation process. What are your thoughts on that for our audience of administrators?

[24:59] SPEAKER_00:

I believe that if we want collaborative teams to be successful, there has to be a really strong trust factor in that community of learners, in that group of teachers. So for example, if we're going through assessment data and I look at yours and your students are so stellar on one of the questions that we've asked them to perform and mine have done terribly, I need to be able to say to you, what did you do? How did you do this? I need to be able to kind of bare my soul and look for help from my colleagues. And that likely will not happen as much. People will hide that openness if it's going to be that they're going to be criticized for it, that they're going to be critiqued.

[25:51]

Unless the theory behind accountability is if I see a weakness, if I see something that needs to be done, I do everything that I can to help that teacher perform at a higher level. So if it's about help, if it's about getting teachers to be the best that they can possibly be, then being able to use data to compare teachers I think is a good thing so that you can buddy teachers up and you can have teachers share each other's strengths and share each other's techniques that help students be successful, but it's not to condemn or downplay or tear down teachers. I think the trust factor is so important too, so that if I do bare my soul with my collaborative team and I say, I don't get this, I don't think I did the best job teaching this, I do need to do some reteaching, what do you guys suggest?

[26:48]

How can you help me with this? This is no longer than a, it's not gonna be a parking lot conversation. I trust my colleagues to know that what I say in that room with them stays in that room with them and that I'm truly asking for help and they're truly there for me so that I can become a better teacher.

[27:05] SPEAKER_02:

I recall sitting in on many kind of collaborative scoring meetings. We had several different types of assessments that teachers gave and scored collaboratively at different points throughout the year. And one of the things that was already in place when I arrived, so I can't take credit for this as a principal, but is this idea of blind scoring that everybody brings their papers and, you know, either we fold back the page with the name on it or we put a sticky note over it. And we shuffle them around and we score them and then talk about them without knowing whose student it was or even which student's work it was so that we can just look at the work and learn from it. And I know we're getting into some, that's getting into a topic that's a little bit outside the scope of the book as far as kind of looking at student work and learning from student work. But I think as leaders, we have to be really careful about what we're holding teachers accountable for.

[27:59]

And sitting in those meetings, seeing that teachers were accountable to their colleagues, teachers were accountable to the process, teachers were accountable for their teaching, it never really... occurred to me or it never really seemed like it would be helpful to me to say you're accountable for your scores I want to see your scores going up because what would happen instead if that was my focus and if that was the pressure that I was applying to teachers is we would see big differences from classroom to classroom in the kinds of things that you noted you know one classroom gets really really good clear directions the other class you know maybe doesn't get those those great directions and what we want this to be about is the teaching, right? We want the teaching to get students to where they need to be. And I just think we have to be very careful with accountability that we're not putting pressure on teachers to focus on something else.

[28:53]

So I wonder if you see that accountability pressure distorting this process in any way.

[28:58] SPEAKER_00:

I think one of the things that I like to say to teachers, in fact, I do it all the time, is that data are. They are what they are. It's what we do about them that's important. So when they happen, it's a fact. It's a current status of where students are. The next step is what we do about them.

[29:20]

The accountability piece would come in if we do nothing about them. If I say, you know what, they are what they are. I'm going to move on. I don't care what's happening. That's a little bit different. But I think that as long as we continue to say, okay, This is what we see.

[29:37]

They are what they are. There's a weakness area here in my classroom. This is what I'm going to do about it. And this is the help I need from you. That's where we want to go because it's not about criticizing or condemning. It really is about growth.

[29:50]

It's not only growth in the students, but growth in teachers as well. You know, I taught for many, many years and I never stopped growing. I'm still growing in my knowledge and understanding. And I think that that's the key to looking at data, the growth that we can experience by looking at not only what our students have done, but what we have done to get them to where they need to be and looking at our strengths and looking at the areas that we can grow and improve in and working on those areas with the support of our collaborative teams. and with the support of our administrators. You had mentioned earlier, in talking about the blind scoring piece, I've done that in many occasions, and it really is beneficial.

[30:33]

It really puts the honesty in scoring because we're not feeling sorry for somebody who might have more difficulty or more struggles. But you had mentioned before, too, about the accountability piece and teaching to the test. And I think what happens if we if we pound on accountability, I believe in teaching to the test, teaching to the targets, we would automatically do that, but we don't want to teach the test. And I believe when accountability becomes such a strong point in any district that, that we set up an atmosphere where teachers want to begin to teach the test to help students be stronger at doing this test that they're doing rather than teaching what we normally would have taught in, in, um, and trusting with the fact that the strategies that we're using will be effective in helping the students understand.

[31:28]

But we're not teaching that, oh, on this test, you're going to see two plus three. So you have to memorize that that is five. Do you understand what I'm saying?

[31:37] SPEAKER_02:

Absolutely. Absolutely. And I think that, you know, that kind of gets back to the whole idea of standards-based instruction and assessment for learning that, you know, we need to not be afraid of teaching with the test in mind or even doing what I think probably some parents would think of as teaching to the test. If the test is good, we should be teaching to the test, right? Yeah.

[31:59] SPEAKER_00:

Absolutely. And if we have that wide variety of questions in there, teaching that content, teaching those targets is not going to, it's not teaching questions on the test. Because they're such, if the students have to apply their knowledge and skills to new and unique questions, they have to use the information that they have in their head at that time. It's not about, it's not about memorization at that point. It's really about the application of knowledge and skills.

[32:27] SPEAKER_02:

Well, Eileen, thank you so much for joining me on Principal Center Radio. I hope people will check out the book, Letting Data Lead, How to Design, Analyze, and Respond to Classroom Assessment. Eileen, thanks once again.

[32:41] SPEAKER_00:

Thanks a lot, Justin. I appreciated the time today.

[32:44] Announcer:

Thanks for listening to Principal Center Radio. For more great episodes, subscribe on our website at principalcenter.com slash radio.