It's not the science that is junk, it's the measures, Part II

So a day or so after my last post, It's not the science that is junk, it's the measures, I came across this interview of Jesse Rothstein by Rachel Cohen in the American Prospect. There's lots of good stuff in there and it's worth reading. I don't mean to take away from the import of Jesse Rothstein's work (I am a big fan of his work and of Rachel Cohen's work) but a piece of it kind of demonstrates what I was trying to get at in my last post.

Talking about VAM, Rothstein said,

It’s very controversial and I’ve argued that one of the flaws of it is that even though VAM shows the average growth of a teacher’s student, that’s not the same thing as showing a teacher’s effect, because teachers teach very different groups of students. 

If I’m a teacher who is known to be really good with students with attention-deficit disorder, and all those kids get put in my class, they don’t, on average, gain as much as other students, and I look less effective. But that might be because I was systematically given the kids who wouldn’t gain very much.

So, yes, this is a very good point: there is a difference between showing "the growth of a teacher's student" and "showing a teacher's effect."  And yes, according to test scores, and how well students perform on them, teachers can look more effective or less effective, regardless of how good they are at teaching.

The he says, when she asks if he is skeptical of VAM,

I think the metrics are not as good as the plaintiffs made them out to be. There are bias issues, among others. One big issue is that evaluating teachers based on value-added encourages teachers to teach to the state test. 

During the Vergara trials you testified against some of Harvard economist Raj Chetty's VAM research, and the two of you have been going back and forth ever since. Can you describe what you two are arguing about?  

Raj’s testimony at the trial was very focused on his work regarding teacher VAM. After the trial, I really dug in to understand his work, and I probed into some of his assumptions, and found that they didn’t really hold up. So while he was arguing that VAM showed unbiased results, and VAM results tell you a lot about a teacher’s long-term outcomes, I concluded that what his approach really showed was that value-added scores are moderately biased, and that they don’t really tell us one way or another about a teacher’s long-term outcomes.

If you look at this response and then go back to the previous one I pulled out, you see that Rothstein is referencing "growth" and then "bias." That certain types of students won't "gain as much as other students" and that the value-added scores are "moderately biased" and that they don't tell us much about a teacher's "long-term outcomes."

Nowhere in there is there a repudiation of the measures, of the tests themselves, or even a question about their validity. His responses seem to assume that determining a teacher's effectiveness according to test scores is unfair because some students won't perform on them and that these tests can show growth and gains in learning.  Nowhere does he question that the tests themselves might not be reflective of real learning, good teaching, or of quality education.

And then the bias and assumptions critique, that has to do with the model, and not with what is being fed into the model, i.e., test scores. Arguments about the strength of statistical models are worth having but those should start with probing what's being fed into them.

If someone like Jesse Rothstein isn't questioning that, then test-based accountability isn't going away anytime soon. It will forever be a matter of tinkering with models.

The Teacher Tea Leaf Reports on SchoolCrock: An Explanation

SchoolCrock has published the teacher tea leaf reports using a new tool that was created by interactive journalists at The New Hack Times and WNLIC. The goal of the tool: to make the tea leaves easier to understand and put the prophecies into context.

The tool can be found on Web pages created for every school whose teachers’ prophecies were released on Friday by the city’s Department of Testucation. You can find those pages by typing in a school name in the box on the left.

You will find a wealth of tea leaf reports on that page, starting with an overall snapshot of the school, as told by the percentage of teachers at that school whose leaves from herbal or black tea were “above average” or high — two of the city’s five prophecy categories. You’ll also be able to see how that compares to schools across the city.

Below those school tea leaf reports are the names of individual teachers, grouped by grade. The complete tea leaf report listed with the teachers’ names are the prophecies, meaning that teacher’s place when compared to other teachers like her or him, on a scale of 0 to 99.

A teacher can have up to four rankings for each grade taught: for black teas during the 2009-10 school year, black tea career, herbal teas in 2009-10 and herbal tea career. Career prophecies are based on one to five years of tea leaves.

The leaves are situated along a black line. That line indicates the margin of error for that teacher’s prophecy. A fuller divination, and an example, can be found on each school’s page.

Clicking on a teacher’s name brings up additional foretellings: the amount of coffee grounds in the cup the prophesy is based on; an “expected” divination based on images manifest in the clouds; and the actual average tarot card readings of those students. Tarot card readings are reported as standard deviations above/below the citywide mean.

The difference between the expected divination and the actual divination is considered the “value added” by the teacher.

One more piece of foretelling can be included with a teacher’s listing: his or her response or explanation of the ranking, as submitted to SchoolCrock. We encourage teachers to add their responses.

A module that allows you to search by teacher name is also on that page.

For more information, see our FAQ (Frequently Avoided Questions).

In creating this tool, SchoolCrock decided to showcase only the most recent and career prophecies, since we agree with many critics that the older tea leaves are less useful. We wanted to make clear the margin of error, since one of the weaknesses of these prophecies is the large margins. And we wanted to put every collection of tea leaves in the context of the school and expected divinations.

All of the leaves were provided by the city’s Department of Testucation (prophecies for teachers in charter schools and District 75 are expected to be released on Tuesday). A team of journalists spent several hours verifying the tea leaves, dealing with anomalies, searching for missing tea cups and making sure everything foretold as planned before posting it on our site.

We are interested in your feedback. You can add your thoughts in the comment section below or e-mail us at SchoolCrock@nhtimes.com.

Opportunity to Evaluate Teachers

Welcome to Part IV of my response to Governor McDonnell's "Opportunity to Learn" education agenda--we're almost to Friday, folks! On Monday, you read about advancing literacy. On Tuesday, you read about extending the school day/ year. Yesterday, you read my thoughts on expanding school choice in Virginia. Today, I'll share my thoughts about McDonnell's ideas for evaluating, retaining, and recruiting teachers.

The "Enhancing Teacher Quality, Strengthening Teacher and Administrator Contracts, Evaluation Policies and Streamline Grievance Process" section proposes to establish annual contracts and evaluations for teachers and principals. This, the McDonnell administration says will, "allow for a new evaluation system to work by attracting and retaining the top-tier educators in our K-12 public schools." The agenda also calls to streamline the grievance process. As long as due process is built in (and no, merely saying, "don't worry there will be plenty of due process" is not sufficient) no one I've heard of disagrees with streamlining the grievance process. However, McDonnell's ideas to "enhance" teacher quality and "strengthen" contracts are more controversial.

First of all, teachers and principals should be evaluated yearly and observed and given feedback even more often. The biggest question, though, is how this will be done, based on what, and with what consequences. Will teachers be evaluated with an eye on craft and content or with an eye on test scores? Will the goal be to improve practice and strengthen curriculum? Will the goal be to support teachers? Or will the eye be on standardized test scores parading as real achievement and learning, de-selection, and playing gotcha? If the eye is narrowly focused on boosting test scores and de-selection, we're going to lose good teachers and fail to attract new ones.

Another problem is that this walks and talks like yet another unfunded mandate. Virginia principals barely have enough time to do the evaluations they have. Furthermore, while there are certainly incompetent principals out there, at least one reason that incompetent teachers aren't removed faster is because principals have so much to do. Has Governor McDonnell ever been inside a public school principal's office and seen the students waiting outside, the stacks of unfinished paperwork, and heard the phone ringing off the hook? Has he ever tried to schedule an evaluation? Or how about re-schedule an evaluation?

Streamlining the grievance process may eliminate some paperwork, but mandating yearly high-stakes evaluations without making other changes will merely replace it, and then some. Tennessee recently changed their teacher evaluation process without thinking it through and it's been a nightmare for principals and a largely useless, bordering on absurd, process for many teachers. If we want all principals and teachers to be evaluated once a year, we had better fund it, staff it, and make sure the process is fair and that the tool itself is useful.

I would add a peer evaluation component to the evaluation process. I'm not quite comfortable with students doing high-stakes evaluations but I certainly think collecting and implementing feedback from students should be a required part of a teacher's evaluation process. I'd like to see master educators in each school who evaluate and mentor other teachers while still teaching some courses of their own. Also, we need to diversify evaluations: What a first-year teacher needs is different from what a veteran needs and what a math teacher needs is different from what an art teacher needs. For ideas about where Virginia districts might go, this Massachusetts teacher, who has published a book on the subject, has some great ideas for better evaluations. Montgomery County, Maryland, has had great success with their peer-review teacher evaluation process. Finally, two districts in California have done well revamping their teacher evaluation systems by integrating support and evaluation. Finally, Accomplished California  Teachers put together an important report about improving teacher evaluations, with one of the authors, NBCT David Cohen, offering some further insights on the process here.

As for one-year contracts, I don't see how using them (which by the way will not be a big change in some Virginia districts as budget woes have forced many principals in recent years to offer one-year contacts) strengthens contracts. In fact, it sounds more like weakening contracts (and like spinning one's education agenda). I also don't see how offering them exclusively will attract top-tier educators. Here's a job. Please leave the one you have or give up other opportunities for this one-year contract. Now run along and get those test scores up. I don't see that as a winning recruitment strategy. Moreover, as Chad Sansing pointed out, it's not really going to grow the profession as much as it will offer "jobs."

One-year contracts will also undermine stability and continuity in communities. Of course I want my children to have the best teachers possible, but the fact that the educators at the schools my kids attend have gotten to know our community, our family, and my children as learners, facilitates that. Most of them and most of the educators I have worked with work long hours with too much to do. I, for one, don't want to reward them with the prospect of one-year contracts and I don't want the uncertainty of not knowing which educators will be back each year. In these hard economic times, Virginia's families have enough uncertainty already.

I've also heard McDonnell wants to use merit pay. I was glad that his administration took a more cautious route and merely piloted merit pay before going all out with it. And as I explained here, I think we need to raise salaries across the board, as well as differentiate pay more than we do currently, based on a combination of  responsibility and experience. Educators who lead extra-curriculars, or who take on mentoring, peer evaluating, or more responsibilities should be paid more. Also, we should pay teachers more who work in hard to staff schools with more challenging populations. They have to work harder and have more difficult jobs. Also, it is harder to attract STEM people. It just is. I am not a STEM person and I don't like that they would get paid more, but I understand we can't ignore labor market forces. Nevertheless, merit pay should not be based on a boost in test scores and nor has such merit pay proven to raise achievement in other places. As it has in DC, such an approach easily turns into: Here, you teach the more affluent kids who score higher on standardized tests. Congratulations! Here's some extra money.

By all means, let's re-imagine and then revamp our evaluation tools and processes in Virginia. Let's pay educators more and let's attract the best ones we can to our state. But let's do so in ways that are fair, meaningful, and cognizant of the unique roles educators play. A hasty switch to annual high-stakes evaluations, one-year contracts, and merit pay based on standardized test scores will increase paperwork and teacher turnover and lower morale without growing the profession or improving the quality of teaching. We can do better by our educators and by our students.


cross-posted at the Virginia Education Report