Potential Dangers of Becoming Data-Driven
Originally published on UXmatters, [January 18, 2021]
Over the past decade, more and more organizations have been doing their best to become data-driven. This is a huge and much-awaited mindset leap — especially given corporate dinosaurs’ typical way of thinking: “I’m the HiPPO (Highest Paid Person in the Office) and my gut tells me this is what the customer needs.”
User research is one of the most powerful antidotes to this outdated mindset and is becoming part of the vocabulary and practice of an increasing number of organizations. It also signifies an increase in UX maturity, so is a very promising trend!
The Problem with Data Being in the Driver’s Seat
Making decisions based on data reduces risk and increases your potential for creating useful outcomes. However, if you make decisions based on bad data, you risk making a similar — or perhaps an even worse — kind of gamble or bet as when making decisions based on untested assumptions.
The way you gather and treat your data can make or break the thing you create, whether it’s a product, service, or any other kind of system that people design — such as an organization or a healthcare system.
If your organization is data-driven, but your data is bad, you’ll just make poor decisions with greater confidence!
According to the Harvard Business Review, the cost of bad data is $3 trillion per year in the U.S. alone.
Let’s consider an example: What if I told you that there is data that shows — with statistical significance (p≤0,05) — that eating ice cream every day is better for your health than eating vegetables? That this data is from a study with more than 20,000 participants, across three continents?
Would you make any decisions based on this data? If you did, would they be data-driven decisions? Who would be to blame when all hell broke loose?
In the aforementioned study, researchers had asked: “What would you like to eat more of so you would be healthier? A) Vegetables or B) Ice cream?” They ostensibly conducted this experiment with over 20,000 kindergarten children. Of course, this is actually a fictional study.
I’ve obviously exaggerated to make a point. But, unfortunately, there are many real examples of bad research design and execution. Problems include asking leading or biased questions during surveys, interviews, or usability tests or cherry-picking data. Introducing bias in user research can generate lots of bad data. In fact, sloppy user research could exponentially increase the chances of gathering bad data. So, as UX researchers, we need to be mindful, vigilant, and rigorous about our research. Unfortunately, it’s not that hard to make mistakes that diminish the quality of UX research. The devil is in the details. Tomer Sharon has shared some potent advice on how to ask questions when interviewing users and stakeholders by avoiding asking leading questions, biasing, or priming interviewees.
What are the chances of your creating something useful on the basis of bad data? At best, you might hit the bullseye, but on the wrong target. You could create something great that no one needs or wants, and you’d be burning a lot of money in the process.
“Crap in, crap out.” — Darren Hood
