Our New York Times Piece on Evidence-Based Management: The Uncut Version

Jeff Pfeffer and I had a piece appear today in The New York Times "Preoccupations" column called "Trust the Evidence, Not Your Instincts."  We are pleased with the points it makes and how it reads, but as is inevitable given the space constraints in newspapers, the final version is a bit shorter than the piece we submitted. In particular, we wish there had been space to include our point that, not only has linking incentives to standardized test scores been generally ineffective, a nasty side effect is that such programs often drive teachers and administrators to cheat (giving students the right answers or erasing wrong answers and replacing them with right answers).

In addition, one point that we didn't emphasize even in the "uncut version" is that we are NOT arguing financial incentives are generally useless, dangerous, or unwise to use.  They do motivate human-beings, and seem to be especially effective (as Dan Pink shows us) for tasks that do not require high levels of imagination.  But a condition for any incentive system to work is that people need to have enough control over their work.  A big problem with many teacher incentive programs is that, all too often, individual teachers just don't have enough resources, enough influence over the preparation kids had before they enter their classroom, enough influence over what happens to their students outside the classroom, and enough support from the administration.  So no matter how motivated the teachers might be, they can't have a big impact on student's scores, at least through honest means. Although it isn't pretty or ethical, teachers and administration sometimes turn to something they can control: They give the kids answers, erase wrong answers and change them to right answers, or in some cases, find ways to get the weakest performing kids out of their classes and schools, even when those students need the most help.  Unfortunately, in too many schools, this means the weakest students are moved to special education classes, which raises mean test scores in regular classes, but hurts both the kids who don't belong in special education classes as well those who do.

OK, here is the uncut version:

Title:

The Virtues of Evidence-Based Management

Reading lines:

We know a lot now about what it takes to build humane and effective workplaces.   Leaders and managers can avert a lot of unnecessary harm –and do much good – by learning and heeding the best evidence.

Text:

Consider this scenario.  You have a serious illness. Your doctor prescribes an intrusive, painful, and expensive treatment— and you have to pay for it.  What she doesn’t tell you—because she has not consulted the research – is that most studies show the treatment is ineffective and fraught with negative side-effects.  You go through the procedure, suffer severe pain, and spend a lot of money.  Unfortunately, as with most patients, the procedure proves ineffective. You later uncover the research your doctor failed to consult.  When you ask why she didn’t use this evidence, she answers, “Who pays attention to studies?  I have years of clinical experience.  Besides, the protocol seemed like it ought to work.”  

Does that sound like malpractice?  It does to us.  Fortunately, pressures to practice evidence-based medicine are reducing preventable errors.  Not so in most of our workplaces, where failure to consider sound evidence repeatedly inflicts unnecessary damage on employee well-being and organizational performance.   But it doesn’t have to be this way.

No workplace practice is as important—and apparently vexing—as pay.  Many people believe that pay for-performance will work in virtually any organization, so it is implemented again and again to solve performance problems -- even in settings where evidence shows it is ineffective.  Consider the recent decision to end New York City’s teacher bonus program after wasting three years and 56 million dollars.  As this newspaper reported in July, a Rand Corporation study found this effort to link incentive pay to student performance “had no effect on students’ test scores, on grades on the city’s controversial A to F school report cards, or on the way teachers did their jobs.”  This bad news could have been predicted before squandering all that time and money.  The failure of such programs to boost student performance has been documented for decades.  A careful review of pay for performance in schools in the 1980s showed these programs rarely lasted more than five years and consistently failed to improve student performance.   The 300 page Rand report emphasizes that (although exceptions exist) evidence against the efficacy of teacher incentive pay in U.S. schools continues to grow stronger and is especially evident in the most rigorous studies. 

This practice doesn’t just waste money.  As Chicago economist Steve Levitt and others show, strong incentive programs can entice – or scare -- teachers and administrators to “cheat” on the tests, either by providing students with questions and answers in advance or changing student’s answer sheets to increase apparent performance.  Recent well-publicized cheating scandals in Atlanta, Baltimore, Washington D.C., and elsewhere could have been foreseen by anyone who read and heeded this research.  Building a culture of cheating in schools corrupts both students and teachers for no good purpose.

Evidence about numerous other practices is ignored too.  Harvard University’s J. Richard Hackman finds that stable membership is a hallmark of effective work teams.  People with more experience working together typically communicate and coordinate more effectively.  Although this effect is seen in studies of everything from product development teams to airplane cockpit crews, managers often can’t resist the temptation to rotate people in and out to minimize staffing costs and make scheduling easier.  This happens even though, for instance, the National Transportation Safety Board found that 73% of the safety incidents reported on commercial aircraft occur on the first day a new crew flies together. 

Hiring the right people is another key decision in every workplace.  Many studies show that unstructured face-to-face interviews are biased; for example, interviewers prefer candidates who are likeable, similar to them, and physically attractive (even when these qualities are irrelevant to performance).  Numerous selection methods are superior – among the best is to simply see if the candidate can perform the work.  Yet interviews remain the primary selection method used by organizations.  And we’ve often been astounded by the refusal of seasoned managers and executives to even consider evidence that interviews are a flawed selection tool.

Strongly-held but weakly supported beliefs about workplace practices reflect what psychologists call “confirmation bias.”  When people hold firm beliefs about something, they tend to ignore, reject, and forget facts that clash with their beliefs; and remember, accept, and more readily accept facts that support their beliefs.   A related impediment is the excessive self-confidence that plagues many people, especially those who wield power over others.  Decision-makers may acknowledge they use a practice that is ineffective for most other people and organizations -- but believe they are so talented that the usual findings don’t apply to them.  

To illustrate, numerous studies show that mergers typically inflict economic damage on the acquiring company.  Yet when one of us served on the board of a software company that was contemplating an acquisition — a “target” company in a different city and of comparable size (conditions that predict merger failure) -- the CEO argued it would succeed despite the evidence because he wasn’t like most CEOs.  He was wrong.  It failed, just as most acquisitions do under these conditions.

Despite such impediments, there is an evidence-based movement afoot in some organizations.  When Gary Loveman became COO of Harrah’s in the late 1990s, he decided that improving the service provided to the best customers—“the people who made the cash register ring”—was a priority.  Loveman had taught service management at Harvard Business School, so was well-versed in research on customer loyalty -- and how employee turnover undermined it. Loveman’s team implemented numerous evidence-based tactics including realistic job previews. After candidates were offered a job, they were informed about the good and bad elements so they could decide if the work was right for them.  Turnover plummeted, service improved, and coupled with Harrah’s innovative marketing, the company went on a decade-long run of outstanding performance. 

A recent study at Google demonstrates the power of accepting and acting on evidence, even when it clashes with ingrained beliefs.  For most of its history, Google’s leaders believed that deep technical expertise was the most important quality for a manager. They believed the best bosses pretty much left their people alone, and their main role was to help with technical problems when people got stuck.  Yet when Google examined what employees valued most in a manager, technical expertise ranked last of the eight attributes examined.  Attributes like being even-keeled, asking good questions, taking time to meet with people, and caring about employees’ careers and lives were most crucial.  Google found that managers who did these things led top performing teams, had the happiest employees, and suffered the least turnover. In response, Google is making many changes in how it selects and coaches managers, and is devoting particular effort to improving its worst managers.  We applaud Google’s leaders for overcoming their biases. But note the attributes of great managers Google “discovered” were evident in hundreds of prior studies. Perhaps if Google’s leaders hadn’t believed they were so “special” and “different,” they might have launched such efforts to improve their managers years earlier.

The evidence-based medicine movement arose in response to thousands of unnecessary deaths and billions of wasted dollars that could have been averted by implementing proven practices.  Similarly, the growing pile of studies on the human and financial costs of employee disengagement, management distrust, bad group dynamics, faulty incentive schemes, and other preventable damage suggests the need for an evidence-based management movement.  Some organizations are leading the way.  It’s time for many more to do the same.

P.S. Speaking of evidence-based management, Teresa Amabile and Steve Kramer, authors of The Progress Principle, had a great editorial The Times today called "Do Happier People Work Harder?"

Comments

[Bob Sutton]

Kevin,

Thanks, that quote is fantastic

Bob

[Kevin Rutkowski]

Your introduction reminded me of one of my favorite quotes about incentives.

"Treat monetary rewards like explosives, because they will have a powerful impact whether you intend it or not." -Mary and Tom Poppendieck, authors of Implementing Lean Software Development: From Concept to Cash

[Christine Ellen Goepfert MD]

Interesting. But, believe me, you surely would NOT like to be treated by any physician who ONLY goes for evidence-based medicine. You would like to have a doctor who has excellent knowledge, excellent skills, a lot of experience, who is smart and open-minded AND incorporates evidence-based medicine in your treatment.

[George Lehman]

I agree with the basic argument that is being made. However I would like to challenge the Intuition-Evidence dualism that is embedded in the argument. It seems to me that many times we face decisions for which there is no clear evidence. In these situations we need to develop our ability to effectively tap into our experience base that often makes itself known in the form of a hunch rather than as a rational argument.