A Stanford undergraduate doing a case analysis on using intuition versus systematic analysis wrote me an email last night to get my thoughts on the difference between the two, especially in light of the work that Jeff Pfeffer and I did on evidence-based management. Below is my lightly edited response. This is just off the top of my head (is it mostly intuition?). I would love to hear your thoughts on this distinction -- if it is useful, how the two concepts fit together, when one is more useful than the others, and so on:
I don't think that intuition and evidence-based management are at odds. There are many times when decision-makers don't have very good data because something is new, the situation has changed (e.g., where do you invest money right now?), or because what might seem like intuition is really mindless well-rehearsed behavior that comes from years of experience at something, so even though people can't articulate the pattern they recognize, they still are acting on a huge body of experience and knowledge. And on the very other side of experience there are virtues to the gut reaction of naive people, as those who are not properly brainwashed may see things and come up with ideas that expertise drives out of their brains (e.g, that is why Jane Goodall was hired to observe chimps, in part, because she knew nothing).The trouble with intuition is that we now have a HUGE pile of research on cognitive biases and related flaws in decision-making that show "gut feelings" are highly suspect. Look-up confirmation bias --- people have a very hard time believing and remember evidence that contradicts their beliefs. There is also the fallacy of centrality, a lot more obscure, but important in that people -- especially those in authority -- believe that if something important happens, they will know about it.
My belief -- and it is only partially evidence-based -- is that intuition works best in the hands of wise people (this is all over hard facts), when people have the mindset to "act on their beliefs, while doubting what they know," so that they are always looking for contradictory evidence, encouraging those around them to challenge what they believe, and constantly updating (but always moving forward), then I think that intuition -- or acting on incomplete information, hunches, conclusions -- is right. Here is one place I've talked about it. Brad Bird of Pixar is a good example of someone with this mindset, as we learned when we interviewed him for the McKinsey Quarterly. So is Andy Grove. I think the most interesting cases to look at are those where people with a history of good guesses or gut decisions -- what mistakes has Steve Jobs made? What about Google... indeed, it is interesting that they believed they were going to crush Firefox with Chrome , but their market share remains modest a year later. My point here isn't to say anything negative about Jobs or Google -- they have impressive track records, plus some history of the usual failures that all humans and human organizations suffer from. Rather, my point is that by looking at errors by people and firms that have generally good track records, you can learn a lot about conditions under which judgment fails, because you can rule out the explanation that they generally suffer from judgment.There is a lot written on intuition and the related topic of quick assessments --- see Blink -- and some evidence (although Gladwell exaggerates about the virtues of snap judgments, as the best are often made by people with much experience in the domain, but as always he makes wonderful points). Also see this book by David Myers for a balanced and evidenced perspective on intuition.
My view is that intuition and analysis are not opposing
perspectives, but tag team partners that, under the best conditions, where
hunches are followed and then evaluated with evidence (both quantitative and
qualitative, that is another issue, qualitative data are different than
intuition, and often better) versus when hunches and ingrained behaviors are
mindlessly followed and impervious to clear signs that they are failing.
Work Matters readers: Again, I would appreciate your thoughts, as this is one of those core challenges for every boss and for a lot of behavioral scientists too!