“How many people used our new feature last quarter?” That’s a nice and easy question. Run some analytics, and you can have a satisfyingly neat answer. “Why did users abandon our product after using this feature last quarter?” Here, you’re getting into muddy waters.
“Big data” may be sexy, and quant research is great for determining what is happening, but numbers can also be effective at predicting what will happen. However, that doesn’t tell the story of why things are happening. But really, that doesn’t tell the story of why things are happening. That’s where “thick data” and qualitative research come in, and good product managers know that individual stories can explain or contradict big data.
Quant studies are by far the largest spend among market researchers, garnering 81% of the money spent in 2017 compared to 14% for qualitative research, according to ESOMAR, a global market research association. Despite that imbalance, it is hard for one to exist without the other, and results are strongest when they are combined. So why do so many companies default to quant, first and only?
The Garden of Quant
The recent multi-billion dollar exits of SurveyMonkey and Qualtrics point to how dominant online surveys have become in the world of quant research. So much so that the layperson would easily be excused for thinking that online surveys and quant data are interchangeable.
In reality, a multitude of quant data exists that major companies are tracking in their product and beyond, whether it’s behavioral analytics, location tracking, or sentiment analysis.
In aggregate, these numbers add up to big data. And there are enough numbers that one might wonder what other information a company could possibly need in order to succeed.
The temptation to stick with quant data alone is an understandable one. Quant data is scalable, allowing you to quickly reach thousands of people and achieve statistical significance in mere hours. Writing a survey has become more doable than ever for the everyday marketers, PMs, or UX researchers, something that may have before been the sole turf of research agencies. With the advent of online panels, online surveys have also become arguably the cheapest method of gathering user feedback.
With time-to-market becoming an ever more important differentiator, particularly in the retail space, massive quant studies have become an imperative, not a luxury.
And perhaps the least important but most desirable aspect of focusing on quant data alone is that you don’t have to actually talk to anyone. I see this all the time with our clients at Discuss.io—talking to people takes preparation, and it’s nerve-wracking to be driving a conversation with a living, breathing user of your product.
Yet despite all these advantages, qual data persists, as does interest in how to do it. A quick Google search shows 318,000,000 results for “how to write a survey” and 607,000,000 results for “how to run a focus group.”
Why isn’t quant data enough?
To Get Ahead, Go Small
It’s easy to lull yourself into thinking that quant data is all you need. After all, it’s fast, easy, and cheap to acquire. It doesn’t involve any of that messy, talking-to-people stuff.
Anecdotally, we hear that when research budgets are tight, qual is the first thing to go. And yet stories abound of missed opportunities and giants humbled who ignore or don’t gather qualitative data. One of the most telling is from Tricia Wang, whose TED talk details how Nokia’s singular focus on big data in 2009 kept them from seeing and reacting to the smartphone revolution in its earliest days.
In her work as a technology ethnographer, Wang developed a deep understanding of the motivations and spending habits of emerging markets where Nokia was dominant—China, India, and Mexico. She saw that despite limited salaries, low-income Chinese were drawn to iPhones and the shiny ads that came along with them. The device was seen as an entry to a high-tech life, and consumers were willing to spend half a month’s salary to get one. Despite Wang’s deep conviction that Nokia was in dangerous territory, Nokia dismissed her findings as anecdotal. Their “big data” analysis didn’t match with her findings. “Your surveys, your methods have been designed to optimize an existing business model, and I’m looking at these emergent human dynamics that haven’t happened yet,” she said.
And, in fact, that’s the strength of small data.
Small Data, Big Gains
Nokia couldn’t afford to ignore qualitative data in 2009, and today’s product person can’t either. Household names are losing market share to faster, nimbler companies focused more on usage of their products than purchase of their products.
HBR recently outlined the importance of focusing on users over buyers. And how do you do that? How do you understand what influences someone’s perception of your product’s convenience factor, or why they sign up for a yearly license but usage across that customer’s organization remains low?
You guessed it: small data, in the form of exploratory conversations with real people. Really, though, it’s the combination of research methods that yields the most impact. Yasmine Khan advocates for a quant/qual sandwich. As timelines continue to be compressed, “Agile research” has emerged as a way to quickly iterate on concepts using a mixture of qualitative and quantitative methods.
Quant and Qual: A Perfect Pair
Quant data and online surveys, in particular, have become a necessary part of a business’s marketing and product development process. But relying on quant alone can lull you into a false sense of having all the answers and being data-driven vs data-informed.
Perhaps the most memorable recent example of the importance of combining quant and qual studies is “Lady Doritos.” In a Freakonomics podcast episode, PepsiCo’s then-CEO Indra Nooyi cited that men and women prefer to snack differently, leading the internet to both coin and condemn “Lady Doritos.”
PepsiCo does a significant amount of qualitative research and puts a priority on developing empathy throughout their marketing and product development teams. They are a client of ours at Discuss.io, and if you read the podcast transcript you’ll see qualitative research described over and over – watching people eat Doritos, considering how people store products in their fridges and pantries, how they carry around snacks, how they drink beverages in cars.
But before insights like this become actionable (or public), they need to be tested at scale using quant methods. Doing so helps to validate the findings, refine the messaging, and ultimately avoid a Lady Doritos gaffe.
In reality, the game changers in terms of new markets, new products, and true innovation come from human intuition and in-depth, unstructured conversations with target users. Though the sample size is much smaller, and the data is harder to analyze, success hinges on uncovering the “why” behind human behavior.
A version of this post originally appeared on ProductCraft.
