A friend of mine remarked the other day on the uncanny ability of Netflix to recommend movies that he almost always finds interesting. Amazon has been known to barrage email inboxes with book recommendations, among other things. And the entire advertising industry has been transformed by the data-driven ability to target individual consumers in ways unimaginable in the Mad Men era.
The power of Big Data goes far beyond figuring out what we might want, even if we might not know it exists. Big Data helps pharmaceutical companies identify the attributes of their best sales people, so they can hire, and train, more effectively. Big Data can help predict what songs are likely to be hits, which wine vintages will taste better, and whether chubby baseball pitchers have the right stuff.
The benefits of Big Data are so, well, big, that there’s no going back. Yet I don’t need to re-read George Orwell, or scan the ongoing headlines about the massive snooping of personal communications orchestrated by the National Security Agency to feel at least some discomfort with Big Data’s side effects. The side effect I am most concerned about is one that seldom gets notice — in a world where massive datasets can be analyzed to identify patterns not easily identified using simpler analog methods, what happens to genius of the Einstein variety?
Genius is about Big Ideas, not Big Data. Analyzing the attributes and characteristics of anything is guaranteed to find some patterns that are just better at explaining the data. It is an inherently atheoretical exercise, one that requires minimal thought once you’ve figured out what you want to measure. If you’re not sure, just measure everything you can get your hands on. Since the number of observations — the size of the sample — is by definition huge, the laws of statistics kick in quickly to ensure that significant relationships will be identified. And who could argue with the data?
Unfortunately, analyzing data to identify patterns requires you to have the data! That means that Big Data is necessarily backward-looking; you can only analyze what has happened in the past, not what you can imagine happening in the future. In fact, there is no room for imagination, for serendipitous connections to be made, for learning new things that go beyond the data. Big Data gives you the answer to whatever problem you might have (as long as you can collect enough relevant information to plug into your handy supercomputer). In that world, there is nothing to learn; the right answer is given. I like right answers as much as the next guy, but in my experience that is just not enough to motivate people to action. For instance, knowing that email follow-ups to sales calls are the most time efficient is nice, but it’s unlikely to convince a salesperson whose always picked up the phone to change her approach, especially if it’s always worked for her.
People don’t think like data behaves. They need to be convinced, they want to be part of the creation of the solution. They don’t like the solution to be imposed on them. Why is this important? Because you can have all the “optimal” solutions you like, but in the real world managers need to convince other people to execute on those solutions. People have this strange habit of wanting to contribute to the development of that solution.
Big Data is troublesome not just for people in organisations who are unwilling to dedicate themselves to something devoid of emotional resonance. Let’s talk business.
Big Data doesn’t necessarily drive out creativity, it’s just that the scientific imprimatur of Big Data makes it very hard to argue the opposite way. Yes, it is possible for creative people to start further down the field when they have a deeper understanding of the underlying relationships that govern their discipline. Advertisers can design better campaigns if they truly understand what consumers are buying and why. But sometimes you just need to break the rules to create anything new. Apple’s original iPod was such a hit precisely because it emphasized simple and elegant design features rather than what everyone else was competing on – MP3 sound quality.
Just like companies that build their business on “best practice,” which ensures that they will never do more than anyone else, companies that let Big Data dominate their thinking will not be the ones who change the rules of the game in their industry. How many entrepreneurs, even in the leading repository of Big Data thinking — Silicon Valley – start with Big Data to design their business plans? Not Facebook, not Google, and definitely not Apple. These companies actively leverage Big Data today to grow their businesses, but the spark that led to their creation was personal, entrepreneurial and even idiosyncratic.
The inability to understand, or capture, the human element in business isthe biggest danger that comes from Big Data. Has there ever been a major breakthrough whose origin doesn’t reside in the brain of a man or a woman? Imagine in the not too distant future a brilliant person, a genius, proclaiming a new way of thinking that is contrary to Big Data. What would happen to her ideas, or to her for that matter, if she bucked the orthodoxy of Big Data to suggest a different view of the world that is not consistent with the dominant digitally-derived solution? As opposed to yesteryear, we won’t lock her up, but we might lock up her ideas. If anyone paid attention to what she said, she would be denounced as uninformed.
How about one of the best-known geniuses of our time? Einstein. What if Albert Einstein lived today and not 100 years ago? What would Big Data say about the general theory of relativity, about quantum theory? There was no empirical support for his ideas at the time — that’s why we call them breakthroughs. Today, Einstein would be looked at as a curiosity, an “interesting” man whose ideas are so out of the mainstream that barely a blogger would pay attention. Come back when you’ve got some data to support your point.
Companies, like civilizations, advance by leaps and bounds when genius is let loose. What would we do with Einstein today?
Listen to the podcast of this column here: https://www.tuck.dartmouth.edu/uploads/content/SydWeighsIn_BigData.mp3