MINDWORKS

Mini: Measure twice, cut once (Shawn Weil, Courtney Dean, and Evan Oster)

April 15, 2021 Daniel Serfaty
MINDWORKS
Mini: Measure twice, cut once (Shawn Weil, Courtney Dean, and Evan Oster)
Show Notes Transcript

Whether with methods or tools or just intuition sometimes to measure humans to improve their performance in doing their job, we track measurements. Have you ever thought to yourself, “why is measurement important”?  MINDWORKS host, Daniel Serfaty, sits down with Dr. Shawn Weil, Courtney Dean, and Evan Oster of Aptima to learn more!

 

Listen to the entire interview in Mission-Critical Environments: Can we improve human performance? With Shawn Weil, Courtney Dean, and Evan Oster.

Daniel Serfaty: So Shawn, looking back, why do you think it's important to be able to measure? We go back to that notion of capturing something, of measuring, whether with methods or tools or just intuition sometimes to measure humans in order to improve their performance in doing their job. Why is measurement important?

Shawn Weil: I've thought about that a lot. Courtney was just talking about his children, and I have two children who are school age. And I think about how they're measured and where measurement is worthwhile and where measurement actually might be a distraction. The reason why you need to measure is because humans have bias. Humans as they are going about their work and trying to accomplish their goals have only so much vision, they have only so much aperture. Especially in training situations, traditionally in the military at least, what they've done is they don't have professional instructors, per se, they have operators who are expert in those domains. And they watch performance, and they cue into the most salient problems or the most salient successes. And they build their training around those narrow bands, that narrow view effects.

So when you do a more comprehensive performance measurement scheme, when you're measuring things from different perspectives and different angles, especially when you're measuring aligned to objectives of what the group is trying to accomplish, what you're doing is you're enabling instructors to overcome their biases, to think more holistically about what it is they're trying to give to their trainees, to their students. And to do it in a way that is more empirically grounded and more grounded in the action they're trying to perfect or improve.

Daniel Serfaty: So Evan, you're listening to what Shawn is saying now. Can you think of instances when you've seen that actually a particular measurement changed the way people talk or people educated? How can we link right now for our audience our ability to measure in sometime precise detail certain aspect of cognition or behavior or decision-making and eventually turn that into an opportunity to train and therefore to improve learning and eventually to improve people performance on their job?

Evan Oster: Yeah. So on measuring human performance, one of the things that I think is critical is being able to challenge the bias, as Shawn was talking through. The expectations that might be placed on students maybe by one instructor and another, you're lacking consistency. And there are nuances to the training that we're trying to get a more objective measure of. So when we're looking at how to measure human performance, being able to get concrete, specific, and objectives is really critical in getting the training to be well aligned. One thing that I've seen through some of our efforts is we've had a particular training that was being trained in groups and lacked the ability for instructors to know if a mistake was made, was that by one student over another? Was it due to the team dynamics that were there, was it due to a lack of knowledge? And when they were training in that group environment, they weren't measuring in a level where they could distinguish between those nuances and those differences.

When we in and started measuring at a more granular level, we were able to then help them disentangle what was happening, when it was happening, and with whom it was happening. And that way, then the instructors were able to tailor what they were doing to specific students at a specific point, in a specific way.

Daniel Serfaty: So you're in fact making the point that this so-called objectivity, and that's a topic probably for another podcast about why the objectivity of measurement is not here to tell the teacher or the instructor that they are wrong, but more to augment what they do naturally as teachers by giving them basically a rationale for intervention, for particular instruction, for focusing on a particular competency of the student. One of my early mentors was General Jack Cushman who passed a couple of years ago. He has an old-fashioned, crusty, three-star general in the army who after he retired was actually training other generals to conduct complex operations and giving them feedback.

And he was always exasperated when people say, "Oh, we did better this time." And he was always asking them, how do you know? How do you know you did better other than just a good feeling? And he came up with this quote that I really respect a lot, I use it a lot, which means you can't improve what you don't measure. How do you know? It's such a simple statement, you don't improve what you can't measure. But very profound in the way it's changing not only our military training but also education at all levels.

Courtney Dean: I stand on that a little bit, Daniel.

Daniel Serfaty: Of course, Courtney, yes.

Courtney Dean: So three thoughts that I don't have written down, so I'll try to channel them together coherently. So number one is I know what right looks like, or alternatively stated, I'll know it when I see it. Practice does not guarantee improvement. That was on a slide that we had for years. And then finally, feedback. So those things linked to each other inextricably. The issue that we had, the bias that Shawn was talking about that you can't improve what you can't measure is all about that. I know what right looks like or I'll know it when I see it. If we don't have something, then a subject-matter expert is resolved to that bias or we don't have consistency from one biased subject-matter expert to another.

And if you don't have any measurement, then that practice can be ... And trust me, I know this one because I have years of experience in this one. Practicing the wrong things, you don't miraculously change those things. There's that critical element that's missing, and that's that third bit, which is feedback. By delivering feedback, we have the potential for a subsequent change. And that's what training is all about.

Daniel Serfaty: Courtney, I'd like you and Shawn to expand basically on that idea, but focusing it more on a particular toolkit that was developed at Aptima early on we call Spotlight, which is a kit that includes basically a lot of the science by articulating, what should we measure, what can be measured, and what is measured. And how we went from basically an academy concept of scales and measurement into a practical tool. I would love for you to tell me a little bit about that. What is unique about it, and what did we learn in applying it to different setting environments? So Shawn, and then Courtney.

Shawn Weil: Yeah, I love the Spotlight story. Unfortunately, I wasn't there at its inception. I suspect if you listened to some of the previous podcasts and listen to Jean McMillan, she'll tell you the story of resilience that was the origin of the Spotlight application, which at its core seems like a pretty straightforward concept. But in practice, it's a lot more sophisticated, especially in comparison to what people tend to do. So Spotlight at its core is an electronic observer measurement tool. It's a way to provide to observer-instructors the means for comprehensive assessment of live activities. So think about it this way. The way it's been done for years is you have people in a field environment, maybe they're doing some maneuver on an actual field, maybe they're doing some pilot training in simulation environments. And you have experts who are watching them, and they're writing their comments back of the envelope.

Well, back of the envelope only gets you so far. Those biases that we've talked about, the inter-rater differences that creep in, they make it so there's very limited consistency. So enter Spotlight, essentially what we've done is put together a set of measures that are comprehensive and aligned to the activities that the trainees are actually going through, then implemented that in an electronic form factor that then affords a bunch of other things. It affords that feedback that Courtney was describing. It allows for aggregation of data. The measures themselves are designed to encourage what we call inter-rater reliability, essentially consistency from one rater, one expert to another. And we've seen that really transform the way that training is done in a number of environments that Courtney and Evan have really been in charge of and really pushed forward over the years.

Daniel Serfaty: Well, thank you for giving us a little bit of history. Indeed, Dr. McMillan was our previous chief scientist, was actually at the origin of developing that tool, I believe originally for the Air Force. But Courtney, you are actually in one of your many roles the manager of these product line called Spotlight, and you've seen dozens of instantiation for Spotlight. You also mentioned the F word here, feedback. Tell me stories about when you used or recently or previously with the way you've used Spotlight? Which is after all a tablet for our audience that prompts the trainer or the observers to grade the learner or the team of learners according to certain scale that have been established as being essential to their expertise through their mastery. So tell us some Spotlight stories, especially when it comes to how people use it to provide feedback?

Courtney Dean: I've gotten my shoes dirty on a couple of occasions. I've been in the woods of Georgia, I've sat on flight lines, I've hung out and fields next to, I guess, improvised villages or foreign operations villages. And I've been in briefings at two o'clock in the morning that extended until three o'clock in the morning after all of those outdoor activities occurred. And in all of those occasions when instructors had Spotlight with them, their ability to communicate to the learner the delta between what is observed and what is expected. And then to elaborate on that with a picture of how to close that delta is far and beyond what I've ever seen when I watched the same activities go down with an instructor with an envelope and a pencil.

Spotlight is these two core components that Shawn talked about, and I'm not going to try to re-describe it because Shawn did an excellent job. You got the measures, and you've got the application that delivers those measures. And when the measures are in the hands of a competent instructor, they're able to make total sense of the student doing the job that they're supposed to be doing. Why was his arm at the wrong angle? Why did the bullet go offline? Why did the tank not make it to the way point at the right time? Whatever the context is, they're able to thread together the story that led to the undesirable outcome. And they can pick spots within that timeline, and they can communicate with that student, "Here's where things deviated slightly, it led to these consequences. Here's where things deviated slightly again as a result of those consequences." 

Suddenly the student goes from, "I failed, and I don't know why," to, "I failed, and it's because I need this fundamental error here," or, "I received this incorrect information here, and I operated on the wrong frame of reference." Those pieces of information are critical for that subsequent change in behavior that I think I've repeated two and three times now. Ultimately, the student is now empowered to become better in the long run.

Daniel Serfaty: Thank you for that vivid description, Courtney. Evan, I think in a sense we see here that what science and the judicious use of data enable us to do is not just to provide the movie of what happened, which would have been the description of the scenario, of the vignette, of the way the student went through a chapter of instruction. But also some x-ray or a hyper vision, if you wish, of that movie that enable the D word, which is in this case is a diagnostic that Courtney is talking about. And perhaps that's what science gives us, the ability to see inside and then eventually say, "Yes, you didn't do that as well as you could, but here's the reason why," and for a student being able to close that gap. Can you think of in your own development and use of Spotlight in environments that sometime are different from the environments that Courtney described? Can you give us some example of how that was used that way, provide that secret insight into human behavior?

Evan Oster: Yeah, there's a couple instances that I can think of. But one in particular is when it comes to receiving that feedback. So it depends on who your trainee or your student is. And there are times where, like Courtney outlined, you have an instructor doing this back of the envelope notes, they provide the feedback. And that leaves the trainee with an option, they can accept it or they can reject it. And oftentimes when you don't have that environment in that context and the angles, a student is more prone to reject the feedback or to make an excuse for it or whatever it might be. But when using Spotlight, I've seen a number of times where that might be the first response.

And then when the student gets the feedback and the instructor shows them where they might've gone wrong here or there, they are able then to accept it and see, oh, my arm was here, or I did do this. And it's that point in that context that's concrete and objective. And then they're able then to accept the feedback and then use the data and the additional context that the instructor can provide, to use that to make a better decision next time.