The illusion of data as objective conceals that it rarely shows you the whole picture. Making decisions based on the easiest data to gather is a recipe for disaster.Last week, the former CMO of Nike published a rare deep dive into a marketing blunder four years in the making. As a firm believer in taking inspiration from analogous domains, I think there is a great lesson here for Product and UX to takeaway.Why do I think its relevant? Just tell me this doesnt sound familiar:Nike invested billions into something that was less effective but easier to be measured vs something that was more effective but less easy to be measured.On the advice of McKinsey, Nikes new CEO John Donahue decided to pivot to a data-driven approach, reorganizing the company towards digital direct-to-consumer sales and eliminating the former model centered on distinct categories. The allure is easy to recognize, and its the same trap that Boeing and other companies fell into over the preceding years.Coming up with new ideas is difficult and requires specialist knowledge. Moreover, it requires specialist knowledge to understand what those specialists are doing and therefore managethem.Meanwhile, slashing costs works the same way across every industry. And so Nike squared up to eliminate duplicate processes, streamline operations, improve efficiency, and increase productivityall those phrases that are meant to say whatever it is you dodo itharder.This kind of strategy was already a punchline in 1997 (Bellwether, ConnieWillis)How well did it work? Well, that depends on what the intent behind the strategy was. If Donahue intended to lose $25B in market cap and tank the stock price 32%, it was a smashing success. For those of us who dont think thats what he meant to do, it bears reflecting on where he wentwrong.The hard limits ofdataIt is wrong to suppose that if you cant measure it, you cant manage ita costly myth.W. EdwardsDemingData isnt worthless. Data is really, really valuable for telling you what has happened in the past. Great expense has gone into producing data that can tell you whats going on in the present. But as the 7- and 8-figure salaries of quantitative analysts at hedge funds show us, using data to extrapolate what will happen in the future is one of the most challenging things you can try to do withit.Typically, the way one would do that is by harvesting whats called warm datathe qualitative data that gives the numbers their meaningand then using that to tell a story about where the numbers aregoing.Unfortunately, thats not what online advertising does. As Phil Bastien calls out here the most common outcome of advertising data is that if you buy a couch, it must mean that you need morecouches.Phils assertion is easy to validatejust turn off Adblock and see if your experience matches that of the 1100 people who agree with user brain.curiosities onThreadsHuman marketers and merchandisers know thisits why you might see paper towels next to the BBQ sauce or buns and hotdogs together at the store. Humans could enrich you are buying bread with the qualitative data point toppings go on bread to create a delightful shopping experience. But getting dataeven data fed to an AIto do this is much harder than youdthink.Nikes decision to eliminate individual product categorieswho could marshal exactly this kind of expertisein favor of a generic one-size-fits-all data model created a problem that you can probably predict. This model performed the equivalent of handing out flyers for your pizza shop in the pizza shops lobby and signaled the business to reorganize from attracting new customers to pumping more money out of the customers they alreadyhad.The data-driven deathspiralMeasuring the wrong things is a fast path to disaster.JaredSpoolOne might propose that the entire profit motive behind the genre of creative work made up by brand, marketing, and design is to drive behavior change. We enable customers to do something newwhether by informing them that an option exists, or making it more attractive and effective to choose that option overothers.And this is where the trouble starts. Nikes incestuous strategy was bringing in data on existing customersthose whose behavior didnt need changing.This is an extremely common mistake in all kinds of industriesthe most vocal, most frequent, loudest participants in your research are the least representative of the total addressable market. The people who dontor cantuse your product barely show up on theradar.Its tempting to reach for the data thats easy to harvest and satisfy yourself with the online shopping data. Thats exactly what Nike did. As a result, their product priorities rapidly diverged from the kind of things ordinary customers buy, while product with mass appeal rotted in warehouses for lack of places to sell it. And the longer Nike chased these fringe customers, the more ordinary shoppers moved on to a competitors product.Letting data drive you is fixed-mindset thinkingA fetish has been made of quantitative methods and whole laboratories have been devoted to solving, with elaborate statistical machinery, problems which had very slight importance.Gregory Bateson,1944With the benefit of time and distance, its easy to condemn Nike for making this mistake. Its even easier to believe that we can avoid itbecause we would simply pick the right data to follow, instead of the wrongdata.Arrested Development: The One Where MichaelLeavesThe fact of the matter is that this mistake was made by experienced, intelligent professionals who also thought that they were doing the right thing. Its not enough to identify that they were wrong; we need to understand why in that moment, the wrong thing seemed like the best wayforward.And its easy to see how it would be. Nike is still one of the worlds most valuable brands, and chasing whales has worked well for many lesser firms. In an environment as risk-laden as 2020 was, it feels good to lean into your strengths and get a practically-guaranteed 10% (or whatever) bump in yourmetrics.Its also very tempting to close your eyes and pretend that the 10% isnt coming at the cost of larger losses elsewhere, because if you speak up it might be your job thats on the line in the next round of layoffs. It takes a lot of courage and conviction to push back against the dominant narrative. It takes research skills to form an informed hypothesis and business sense to convince people with budgets that it ought to betried.Validating what everyone already believes is far easierbut it adds no value. The value of research doesnt come from elevating people who are already shouting. It comes from finding the people who are not being heard, and adding their voices to the conversation.Then you can make a real data-driven decision: a decision driven by all thedata.Otherwise, the best you can do is commodity-grade decision-making.Nikes $25B blunder shows us the limits of data-driven was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.