New York City Halts Teacher Bonus Program: Another Blow to Evidence-Resistant Ideology

The New York Times reports that the school system has abandoned their teacher bonus system because it is ineffective. I quote:

A New York City program that distributed $56 million in performance bonuses to teachers and other school staff members over the last three years will be permanently discontinued, the city Department of Education said on Sunday. The decision was made in light of a study that found the bonuses had no positive effect on either student performance or teachers’ attitudes toward their jobs.

The research appears to be quite careful and the RAND Corporation is highly respected:

The study, commissioned by the city, is to be published Monday by the RAND Corporation, the public policy research institution. It compared the performance of the approximately 200 city schools that participated in the bonus program with that of a control group of schools. Weighing surveys, interviews and statistics, the study found that the bonus program had no effect on students’ test scores, on grades on the city’s controversial A to F school report cards, or on the way teachers did their jobs.  “We did not find improvements in student achievement at any of the grade levels,” said Julie A. Marsh, the report’s lead researcher and a visiting professor at the University of Southern California. “A lot of the principals and teachers saw the bonuses as a recognition and reward, as icing on the cake. But it’s not necessarily something that motivated them to change.”

Are you surprised? I am not, and if the people running the New York City school system had actually read a large body of existing research, they would never have wasted all this money in the first place. In our opening chapter of Hard Facts, Dangerous Half-Truths, and Total Nonsense, Jeff Pfeffer and I reviewed the extensive literature on the links between incentives and teacher performance, and it turns out that although there always have been people with great faith in pay for performance systems for teachers -- going back to at least 1918 -- careful studies show over and over again that they do not improve student performance.  The New York Times article suggests that despite the ideology supporting pay for performance systems, there is growing evidence that the current round of incentive-based teacher pay isn't working -- just as it never had worked:

The results add to a growing body of evidence nationally that so-called pay-for-performance bonuses for teachers that consist only of financial incentives have no effect on student achievement, the researchers wrote. Even so, federal education policy champions the concept, and spending on performance-based pay for teachers grew to $439 million nationally last year from $99 million in 2006, the study said.

To be clear, pay for performance schemes do appear to have some effects in schools -- most of which are bad. One of the most well-documented (see this post on findings in Freakomomics and related research) is that some teachers and administrators start cheating when their pay is linked to performance on student's standardized tests.  Their are strong hints that this is exactly what happened in Washington, D.C. and other cities where financial incentives for teachers and administrators are linked to student test scores. 

Note that I am not arguing against pay for performance systems in general.  They work in other settings --sports, sales, lots of other places,as we show in Hard Facts.  But they don't work for teachers for a host of reasons, perhaps paramount among them are that teachers rarely have enough control over key student behaviors before, during, and outside of class, over class composition (and when they do, they sometimes use it to cheat the tests.. such as by sending poor perfomers to special education classes), and over other resources they need to have a strong enough impact on student learning.   Also, giving students a test once a year probably isn't a very good way to measure what students are learning.  As The New York Times report argues, another problem with pay for performance schemes is that it turns teachers' attention away from intrinsic rewards (the reason most go into the profession in the first place) and toward extrinsic rewards (See Dan Pink's Drive to learn more about the trouble with extrinsic rewards). 

To be clear, I am NOT a general supporter of the policies of teacher's unions.  Although I do think that way too much blame and way too little credit is given to teachers, I do have an evidence-based pet peeve against how vehemently teacher's union's defend the jobs of bad apples, the rotten and incompetent teachers.  This argument is consistent with the work on "Bad is stronger than good" that I've discussed here before... while it is tough for even a good teacher to overcome a lousy system and have strong positive impact on students, it is pretty clear that really lousy teachers can make a bad system worse, and dampen the positive effects of a good one.   I believe that if unions changed policy here and became even more vehement about reforming and removing bad teachers than their critics it would improve their reputations and the quality of education -- and earn them political capital to battle lousy policies such as tying teacher pay to student test performance.  (See this great discussion and debate at The New York Times).

To return the dismal record of pay for performance systems in schools, some years back, I had an interesting conversation with Tony Bryk, a prestigious educational researcher who is now heads the Carnegie Foundation.  We were were at a think tank, a place called the Center for Advanced Study in the Behavioral Sciences, and I asked Tony why -- even though there is so much evidence against practices like pay for performance for teachers -- they remain popular and come back in waves... until overwhelming evidence emerges again that in fact they are bad.  Tony suggested two reasons.  The first has to do with ideology -- that people hold some assumptions so strongly (like economics and business minded folks who believe that incentives are the best answer to changing any kind of human behavior) that they refuse to accept any evidence that runs counter to their beliefs -- no matter how strong those findings might be.  The second reason was what Tony called "collective amnesia."  He argued that, in the history of educational policy, the same bad ideas seem to come around every 10 or 20 years, and policy makers and their staffs either don't remember or make no effort to dig-up relevant research to guide their decisions, regardless of their ideologies.  In the case of pay for performance, it appears that both of these factors are operating. 

Practicing evidence-based management isn't easy given our various human flaws.  But we sure could save a lot of money, a lot of heartache, and make people's lives a lot better if we all tried a lot harder to do it.  There are plenty of outcomes in life that are impossible to predict.  Unfortunately, what happened in New York was completely predictable, even if people with blind faith in linking test scores to financial rewards for schools and teachers remain unwilling to believe this well-established truth now. 

P.S. A comment below suggests that some recent studies do show a positive effect of financial rewards on student performance.  My reading of the Rand Report suggests that for studies done in the U.S. there are a a few studies that show a positive impact, but the weight of the evidence supports the historical pattern of no effects or negative effects.  The more rigorous studies in particular find no ot weak effects on test scores, and little effect on teacher motivation, although there is some evidence that teachers devote more evidence to teaching to the test and less to teaching other things (not a surprise).  There is also some evidence that teacher cooperation goes down a bit and evidence that teachers game the system more to boost test scores.   A researcher from Chicago explained to me that in the schools she was studying (this was about 10 years ago) that scores were going up but she believed that it was not so much because incentives motivated teachers to work harder, but because it motivated them to get rid of their weakest students (often by sending them to special ed classes) and to refuse to "skip" gifted students because they pumped-up the average test scores in a class.  Finally, the most obvious effects of pressure on teachers and administrators to pump up test scores is cheating on the tests (by the teachers and administrators), as we have seen from evidence from Chicago, Philadelphia, and Atlanta, and just today, a probe started into cheating at schools in New Jersey.

I am not rejecting the value of financial rewards as motivational tools for teachers outright, and it does appear that there are some special conditions under which they may be of some value. But the weight of the evidence suggests that most of the money spent on such incentives could have been to better use, that the ideological support for them is much stronger than than the evidence in support, and that one of the most consistent effects is bad -- teachers and administrators cheat either to get the incentives or because they fear losing their jobs.

This is my conclusion. You may reach a different one.  Here is a link to the 300 page Rand study of the failed New York program, which contains an excellent and very current review of the research.

Comments

[Bob Sutton]

Great comment. I agree teachers should be paid much more and given higher stature in other ways... Singapore is a good example in some ways. But the use of test scores as the metric just isn't working and I also question the need to divide them into winners and losers. Think of quality logic... rather than assuming there will always be a lot of broken parts, why not pay enough money and provide enough supports so that nearly all the parts (teachers) are good -- and be extra aggressive about getting rid of the bad parts (teachers).

[R. Cannon]

One of the problems here is that almost every study cited inthe RAND article emphasizes the short term use of merit pay in isolated contexts with pre-existing employees. Essentially then, you are taking a sample group who self-selected into the profession (and the particular institutions) based on the pre-existing incentive structure. I taught in urban public schools for several years, and have often likened that work to that of rural family practitioners or, at the furthest extreme, nuns. The vast majority of the people in that line of work are essentially doing the work as a form of "service." Merit pay for nuns would probably not make the better and nunning, same for teachers (thus the teachers viewing it as a form of recognition). Where you DO see effects of merit pay are with systematic changes- either in new institutions recruiting a totally new teaching for or use at the national level (more so the latter) because it then attracts a workforce that is drawn to that system of incentives. There are plenty of smart folks who would great make teachers who also like the idea of being rewarded for success; in our country they become corporate trainers, lawyers, physicians, etc. etc. In nations where teaching is a high-status, high-paid, and high-expectation profession, it becomes a real possibility for that subgroup. There aren't enough nun's in the world to run all of our schools.

http://www.hks.harvard.edu/pepg/MeritPayPapers/Woessmann_10-11.pdf

[Anon]

This reminds one of two things, both on dealing with systems -

1) the lessons of W. Edwards Deming's Red Bead Experiment:

Red bead example - http://maaw.info/DemingsRedbeads.htm

2) Donnella Meadows's statement to the effect that the favorite solutions people bring to solving system problems are usually wrong

[Kevin Rutkowski]

I'm glad to see that people are waking up again to the real effects of these incentives.

In addition to everything you said in your post, it seems to me that most of these incentives make the assumption that a student's performance on a test is based solely on what a teacher did with that student during the current year.

I had consistently good test scores in school regardless of whether a particular teacher I had in a given year was good or bad. Most of my teachers were good, so one bad one did not undo all of my education. Similarly, one good teacher cannot undo years of poor education.

Of course, a student's performance is also influenced by many factors outside of school as well.

[[email protected]]

Bob

We work in public and private organisations and find that bonuses (like targets) distort systems.

We agree it is based upon a false belief that people need 'motivating' instead of an understanding that people are purposeful (human nature). Economists need to modify their theories in the face of extensive evidence.

Unfortunately targets, tables and testing are the three things killing education. This is a short video of Professor John Seddon giving evidence in the USA to California University on the export (from the UK) of 'deliverology'. Targets as incentives.

http://www.thesystemsthinkingreview.co.uk/index.php?pg=18&backto=18&utwkstoryid=257&title=Deliverology+destroys+service%3A+Professor+John+Seddon+addresses+the+faculty+of+California+State+University&ind=7

[davidburkus]

Interesting findings. I've just left a position in healthcare, which similar champions pay for performance metric to try and compensate based on preventative measures, not disease treatment. I think it has a similar dilemma: doctors aren't responsible for patients eating, exercising or otherwise taking care of themselves.

[LKMco]

Surely the problem here is with equating performance purely to exam results? If we take a broader understanding of performance then what better to base pay on than performance? See my blog about this here: http://lkmconsulting.co.uk/article/what-earth-should-pay-be-based-if-not-performance-20072011

[David Hinchee]

Bob,

Has any of the research looked at what happens to the quality of education when a bonus that was previously granted is discontinued?

[Stuart Buck]

But it's not true that teacher merit pay has never improved student performance. Here are several studies:

Texas: http://www.performanceincentives.org/news/detail.aspx?pageaction=ViewSinglePublic&LinkID=594&ModuleID=48&NEWSPID=1

Houston: http://www.aefpweb.org/sites/default/files/webform/ASPIRE%20--%20AEFP%20Conference%20Draft.doc

England: http://www.sciencedirect.com/science/article/pii/S0927537108001218

Israel: http://www.nber.org/papers/w10622.pdf

India: http://www.nber.org/papers/w15323.pdf

[Sluggo]

Teachers unions are in a tough position relative to defending bad teachers. They have a legal duty of fair representation in the due process system. They can no more force bad teachers out through the disciplinary process than defense lawyers can testify against their clients. Three things could improve this: 1) Teachers have control over part of the evaluation process through peer review. There are many successful examples of this. 2) Due process should take, at most, a couple of months and not years. 3) 60% of teachers quit in the first five years of service. This combined with the "bad teacher" legends points to a system of recruitment and retention that has failed miserably. Tighten up standards for becoming a teacher, offer loan forgiveness, and improve compensation for younger (not all) teachers.

We are doing the opposite in the US. We are blaming teachers for things that are out of their control. We are constructing evaluation systems that hold them accountable through standardized tests which is rife with moral hazard. The people who claim to be reformers are at best dismissive and at the norm, contemptuous toward the teaching force.

I will actively discourage my kids from pursuing teaching until conditions for teachers improve.