Good to Great: More Evidence That "Most Claims of Magic are Testimony to Hubris"

Stanford's James March, perhaps the most prestigious living organizational theorist, wrote me an amazing email when Jeff Pfeffer and were writing Hard Facts, our book about evidence-based management. When I asked him if there were any breakthrough ideas in management, his reply was:

“Most claims of originality are testimony to ignorance and most claims of magic are testimonial to hubris.”  Unfortunately, we are in an era where the evidence to support March's statement is rolling in at remarkable rate. Many of the "geniuses" that have been worshiped in recent years have turned out to be false gods -- just ask all those people who lost billions with Bernie Madoff. 

As much as I love reading Jim Collins' Good to Great for its compelling writing style and great stories, I've always found his claims way overblown. Any study that is based on only 11 companies, out of a sample of 1500, is bound to have a lot of methodological problems.  (e.g., How many companies that had Level 5 leaders didn't make the leap? His method makes it impossible to tell.) And it has always bothered me that once his "chimps" (as he called his research team) identified the good and great companies, they went back and interviewed people about the differences. It is sort of like asking players of the team that won the world series or super bowl (and reporters) what the secret to their success was after the series is over, and then comparing their answers to the players on the fourth place team. Compelling research by Berkeley's Barry Staw and others shows that winners and losers offer much different recollections to explain what happen and why, regardless of the facts.  In particular, winners report having better leaders, being more persistent, more focused, and more cohesive -- even in experiments where there is no real difference between groups, and the winners and losers were simply misled about their performance by some seemingly legitimate person.  So I don't know who will win the Superbowl this year, but I can predict what the winning players will say, and so can you.  If you want to read a detailed critique of the methods in Good to Great, see The Halo Effect. 

The other thing that always bothered me about Good to Great is that -- although there are thousands of rigorous peer reviewed studies that are directly on the issues he studies -- he never mentions any of them to further bolster or refine his arguments (for example, there is a lot of research on charismatic leadership that he ignores, which shows it is effective under many conditions -- most striking to me is that he defined it as, essentially, flashy and arrogant leadership, which isn't how most researchers define and measure it. They they do talk about charismatic leaders being articulate and able to set down a clear vision, but the key thing is the ability to motivate and direct followers' energy -- muck like Collins'  Level 5 leaders).   So this second concern also is reflected in March's quote, as a claim of originality that is testimony to ignorance.

To return to the magic, the evidence that anything that is too good to be true usually is -- and there is no such thing as the single magical breakthrough study in business -- has been pouring in lately when it comes to Good to Great. Steve Levitt, of Freakonomics fame (and unlike Collins, a researcher with bona fide research training and a world class scholar), did an analysis in The New York Times called "From Good to Great..to Below Average," showing that Collins' 11 "great companies" were as a set performing below the S&P 500. Note also that this even before the shit really the fan, and the financial crisis drove Circuit City into Chapter 11 and Fannie Mae into much deeper trouble.

Just this morning, I got a broadcast email from Academy of Management  Perspectives, an academic journal I subscribe to that attempts bridge the gap between academic research and management practice. The next issue contains two articles -- both based on data gathered well before the economic meltdown -- that further document that lack of staying power among these 11 so-called great companies.  I think it is worth printing the entire abstract of each here, as given this book has sold 4.5 million copies.  Even if this may be boring to some of you, the ideas in the book are taken as an articles of faith in so much business -- and Collins has acted as if the ideas in his book and those 11 companies provide magic for any company.  In an ironic twist, he has displayed remarkable personal hubris in his (noble) efforts to undermine leaders who suffer from hubris.  His writing and inspiring speeches are moving, but he is selling snake oil, or at least exaggerating the healing powers of some well-known management practices that work pretty well, but are not magical cures.

The first article is by Professors Bruce Renick and Timothy Smunt, and is called "From Good to Great to..." Here is the abstract:

With sales of more than 4.5 million copies, Good to Great by Jim Collins provides an inspiring message about how a few major companies became great. His simple but powerful framework for creating a strategy any organization can use to go from goodness to greatness is certainly compelling. However, was Collins truly able to identify 11 great companies? Or was the list of great companies he generated merely the result of applying an arbitrary screening filter to the list of Fortune 500 companies? To test the durability of his greatness filter, we conducted a financial analysis on each of the 11 companies over subsequent periods. We found that only one of the 11 companies continues to exhibit superior stock market performance according to Collins' measure, and that none do so when measured according to a metric based on modern portfolio theory. We conclude that Collins did not find 11 great companies as defined by the set of parameters he claimed are associated with greatness, or, at least, that greatness is not sustainable.

The second article is by Professors Bruce Neindorf and Kristine Beck. It is called "Good to Great, or Just Good?" Here is the abstract:

Good to Great has been on Business Week's best-seller list since its October 2001 release. In Good to Great, author Jim Collins identified a set of 11 firms as great, then used them to derive five management principles he believed led to "sustained great results." We contend that due to two fatal errors, Good to Great provides no evidence that applying the five principles to other firms or time periods will lead to anything other than average results. We explain the two errors and empirically test our contention. When ranked with the 2006 Fortune 500, the 11 Good to Great firms have an average ranking of 202nd. In addition, in terms of long-term stock return performance, the Good to Great firms do not differ significantly from the average company on the S&P 500. Our evidence is consistent with the conclusion that although the Good to Great firms may be good, they aren't great. 

Both these artifices are published in Academy of Management Perspectives, Volume 24, Number 4, 2008.

Although, oddly, my colleague and friend Jeff Pfeffer continually crossed-out most of the text in Hard Facts that took Collins to task for methodological problems and excessive claims. But the key lessons from this book, and so many others, are:

1. As March implies, there are no magical leadership or organizational practices that will quickly propel your organization to the top of the heap.  Even the greatest organizations struggle to stay at the top and are led by fallible people who make many mistakes.

2. There is no such thing as a single breakthrough study. The best and most valid conclusions and advice are based on a series of studies that have survived the brutal peer review process and that result in a consistent set of findings. In this regard, an interesting contrast is Chip and Dan Heath's Made to Stick, which is based the weight of the evidence from hundreds of rigorous studies (instead of one that could not survive the peer review process unless the claims were toned way down and the hundreds of past studies that were consistent -- and clashed with it -- were at least mentioned).  I especially point to Made to Stick, and I would add Influence, because they are so well-written that they show you can combine good scholarship with a great read. 

3. My main objection, in the end, isn't to the research Collins did -- the stories are interesting and I believe that nearly all of the practices that he suggests would make a manager more effective -- indeed many if not most are bolstered by more rigorous studies (albeit, even as his research now implies, as signs of competence or even ordinary greatness). My objection is -- to use Jim March's words -- the hubris and ignorance about the claims about the rigor of the research and the originality of the ideas.  There are lots of management books, or parts of management books, that are incredibly useful and inspiring, but don't claim to draw on research.  Orbiting the Giant Hairball is a great example.  Another is Tom Kelley's masterpiece Art of Innovation. The difference is that these great books don't make excessive claims - Hairball draws on the author's personal story and Tom Kelley draws mostly on what he and his colleagues have done at IDEO.

I suspect that I am not going to make a lot of friends with this email, but as I struggle to identify evidence-based practices --and practices that seem promising, cool, or have been used to good effect in a few places -- that help managers and others do their work more effectively, to achieve what might be called "ordinary greatness," I think that -- in the spirit of the times -- perhaps all of us who are paid to give advice to managers and leaders owe to them, and to all the employees whose jobs depend on their actions, to come clean about the limits of our claims and our knowledge. 

We are at a moment in history were we all need a lot more truth and a lot less snake oil.

P.S. Here is a post about another great quote from March, on innovation.

Comments

[Kwiefling]

I love your honesty and fresh style, Bob! While Good to Great has valuable lessons, we must remember "All models are wrong. Some are useful."

[tableau]

agree with jack , it's a good book

[jack]

Hi Bob,
I have read "Good to Great" and I think it's really good !

[Dan]

Bob -- Thanks for this post. As an author myself I find myself frequently caught in the dilemma of the demand. I remember, for example, a CEO telling me that my work "didn't amount to a mature business model" one day because I was using too many anecdotes and someone else saying the next that "the data didn't have relevance" to their organization. People want answers that are universal AND particular. The only "truth" in this mess seems to be that it is possible to write a best seller out of recipes and formulas if they look like they might pay off, especially if enough "snake oil" is rubbed on them, and you're not with it if you don't do X or Y.

There's a misbegotten conceptual blunder in all this. We think we can reduce the complexity of business acumen and leadership to something that is actually at a much deeper level of both personal and organizational understanding. It's not that we shouldn't try to articulate how to improve, but to confuse that with a promise, particularly a scientifically verifiable promise, is simply naive. This is not, to my mind, terribly different than the employee, not doing his/her job, who complains: "just tell me what to do!" And if you can't tell me, then you are a hoax. So we do the best we can to offer the expertise, and voila, we are found out. We took the bait of hubris.

We skimmed over the section that said there are no absolute answers. We skimmed over the section that said we don't know. The answer to this MAYBE is a community. One where we talk about the real stuff that's going on in our firms and in ourselves. Seems like, from time to time, that might create a breakthrough.

[dblwyo]

So tell us...how do you really feel ? No wonder GtoG hasn't been a productive topic. Taking all your points as given then what ? What we really need is a book that compiles, compresses and generalizes the huge swath of extant findings that we can use. And on the other side of the coin it's often been my experience that academics trap themselves into using tools they're comfortable with and very unwilling to consider anecodotal evidence and cases until the constitute a statistical universe. That means the small samples get ignored. How to split the differences ?

[Roshan]

Hello Bob,

I have to admit that I have been GTG admirer although I was always skeptical about the research that was conducted to arrive to the 11 companies that they cite in the book.

But I believe that you can always learn something from every book eventhough 90% may be repeat of other books or old wine in new bottle as they say.

Roshan

[Kevin Horgan]

Agree totally with your perspective. Would also apply similar critique to Bill George's True North which consists of retrospective interviews and platitudes. No wisdom unlike Fooled by Randomness or The Halo Effect or your work.

[anon]

"Regardless of the quality of the scholarship, does the book claim that the management techniques that took the companies from good to great result in a permanently great company?

I'd like to know if the companies that went from "Great" to below average (or worse) sustained the techniques described in the book over the past 7 years. "

I couldn't agree more. I feel Bob and others make a lot of great points. Further, Collins should be careful about how 'magical' he considers his claims to be. That said, I don't recall Jim ever saying that his 11 Greats would remain great forever... or even beyond the time frame of the study.

I'm on both sides of this argument.

[Andrew Meyer]

I agree with all of your comments and I would add one other. A business exists in a business environment with a particular set of economic fundamentals. If something causes that business environment to grow and the economic fundamentals to improve, all the businesses in that area will probably do well. Likewise, if something causes that environment and its economic fundamentals to shrink, all of the businesses will be hurt.

As the great Warren Buffett said many years ago, if a management team with a reputation of great success encounters a business with bad economic fundamentals, it is the business with bad economic fundamentals that will retain its reputation.

I agree with you that Collin's book has some struggles, but also consider that many of the companies that did well were in environments that enjoyed good economic fundamentals for a long time. When those fundamentals changed, so did the reputations of the businesses.

Thanks for all your great writing, insights and for challenging people to think. I appreciate it, Andy

[Brad]

One of your best posts ever.

[Jan]

A good post. I have my own example from Sweden. A government backed VC I talked to claimed that: "We can show with statistics that the companies we invest in do better than the other ones (i.e. the ones we are not investing in)."

It's like going to the racetrack and betting on favourites and say: "The horses I bet on do better than the horses I don't bet on." If you only pick favourites, you are more likely to pick winners, that's all. In my case, the only thing the statistics showed was that the VC could pick successful companies, companies that probably would have made it anyway.

When it comes to repeating success in companies, it's surprising how few executives can do just that after they have left for another company. Apple is the most notorious example I can think of. Apple have had a string of executives who left and could not repeat what they did within Apple.

Processes cannot always be described with words. They can be dead simple to execute, but almost impossible to describe with words, like: describe to me how you tie your shoelaces. Easy to do, impossible to explain.

[Nivi]

Agreed with all of the above, but I would like to make the case for the "magic" of management theory.

Imagine a world where some scientists know Maxwell's equations and others don't. The scientists who know Maxwell's equations can predict the future. Those who don't are lost in the wilderness.

That's the world we live in today when it comes to the "equations" of management and organizations. People who understand Peter Drucker and lean manufacturing are magicians compared to those who don't.

Or, at least, they have the potential to be magicians -- unfortunately, you still have to wake up in the morning and do the work. =)

[Kevin Rutkowski]

I bought the book a few months ago because my company's CEO is a big fan of it, but I haven't read it yet.

Regardless of the quality of the scholarship, does the book claim that the management techniques that took the companies from good to great result in a permanently great company?

I'd like to know if the companies that went from "Great" to below average (or worse) sustained the techniques described in the book over the past 7 years.

In the US, it seems that management has a lot of pressure to do something different than what they're currently doing regardless of current success. It seems that many people consider a manager who maintains the status quo to not be doing their job. I suspect that some of the companies changed what they were doing simply for the sake of change.

I have heard that Toyota specifically hires CEOs who will maintain the current successful system and frowns on ones who would make massive changes. Maybe the US companies could learn from that theory (if it indeed is a theory supported by evidence).

I noticed that Wells Fargo is a company on the list that I think would still be considered great by many measures. There aren't too many banks that are as stable as they are right now. Perhaps bank executives have less pressure to change what is currently successful.

[Frank Roche]

Bob, I've always been skeptical about Good to Great. Your points here about evidence-based assertions is so spot on. And it should be chilling to those who drank the GTG Kool-Aid.

I get offered a ton of books to review each year. I won't do it anymore unless there's an academic basis for the work. Making claims based on opinions and self-reinforcing "research" won't cut it. I agree, by the way, with what you say about the Heaths' Made to Stick -- great writing and real research.

Thanks for this post. I'm cheering you, even if I'm sitting on the cheap seats.