It's not the science that is junk, it's the measures

So I recently had occasion to read a whole bunch of studies on charter schools and one type I read was about their effectiveness. I read the CREDO studies and I read critiques of the CREDO studies and I read meta-analyses and I read smaller studies.

Anyway, I want to go back to something I used to say and that I have heard others who are similarly skeptical of Big Ed Reform, and that is the notion of "junk science." A lot of us have called VAM and have called other studies of educational effectiveness "junk science." I know I did, indignantly. But you know what? I didn't really know what I was saying. (This is one reason I went back to get my PhD, so I would have more understanding of these kinds of things.)

And I was reading all of these studies on the effectiveness of charter schools, I remembered reading this post by Matt DiCarlo on the Shanker Blog from over 3 years ago. I remembered that reading it gave me pause about calling what I did "junk science" and I ceased doing so, but even so, I couldn't fully relate to what he was saying:

Now, I personally am not opposed to using these estimates in evaluations and other personnel policies, but I certainly understand opponents’ skepticism. For one thing, there are some states and districts in which design and implementation has been somewhat careless, and, in these situations, I very much share the skepticism. Moreover, the common argument that evaluations, in order to be "meaningful," must consist of value-added measures in a heavily-weighted role (e.g., 45-50 percent) is, in my view, unsupportable. 

All that said, calling value-added “junk science” completely obscures the important issues. The real questions here are less about the merits of the models per se than how they're being used. 

If value-added is “junk science” regardless of how it's employed, then a fairly large chunk of social scientific research is “junk science." If that’s your opinion, then okay – you’re entitled to it – but it’s not very compelling, at least in my (admittedly biased) view.

I am still no statistics expert and I never will be, but I have a much greater appreciation for what these models and analyses can tell us and what they don't tell us and what their limitations are. And these researchers conducting these studies, they may have different ways of conducting the studies and different opinions regarding which factors should be included and which shouldn't, but they know what they're doing, most of them at least, and they go to great pains to be thorough and thoughtful about their design and methodology and to explain the models they're using and to account for the results that these models produce. So the problem is not with the science.

DiCarlo says the problem is in how the models are being used. Yes. But another problem, as far as I could glean, is with the measures they're using. "Student learning” and “student achievement” have come to be represented by test scores. That is not my currency of educational quality, but it is the current currency in educational research and policy. I think many of these tests are of dubious quality and I don't think that they provide a true measure of what students have actually learned and or of the quality of their educational experience. Richer, deeper, more authentic student learning in charter schools, and schools in general, can be measured if we think creatively and holistically about it. But we're not doing that and we're not incentivized to do that. So much of the money for educational research, so much of the recognition, goes to researchers who use these test scores as measures. Because there's not much else. Even researchers who don't agree that they are good measures will say as much in one paragraph and then cite them as evidence of effectiveness or lack thereof in the next. 

To me, it's kind of like chicken nuggets and milkshakes. McFastFood place has a sound process for making chicken nuggets and milkshakes, but once all is said and done, how much actual quality chicken meat and milk come out of the other side? How much actual nutrition? How much actual, recognizable learning and educational quality gets funneled through these tests and comes out of the other side of these statistical analyses that use test scores as measures?

I doubt much.