Your customers aren't lab rats

Everywhere you turn these days, someone is talking about running experiments and tests to understand your customers or target market.

They say things like:

  • use experiments to validate your ideas
  • A/B test responses to interface designs
  • build minimal solutions that can be built and released quickly to get ongoing feedback.

It seems that many product people can no longer make a decision without first testing everything out on users. Some would say:

“Of course, why wouldn’t you do things that way?”

but I’m going to take a counter point.

Just because you CAN run lots of experiments or A/B test almost everything, doesn’t mean you SHOULD or that it’s the right thing to do.

In a recent discussion about improving sales effectiveness, someone actually suggested running an A/B test with sales teams; training one group one way, the other, another way, and then seeing which group did better.

Clearly this person didn’t understand that A/B testing requires ALL other variables to be held constant which would be impossible for different sales teams, with different territories, customers, objectives etc.

Additionally, let’s be honest…with exceptions such as A/B testing specific web page designs/layout over MANY (thousands or millions of) impressions, most experimentation and user validation is done empirically with small numbers of customers.

And the results themselves may be skewed or contain significant margins of error.

  • Did you ask the right questions?
  • Did you ask the right people?
  • Did you use the right tests?

Is anyone measuring that when the feedback is incorporated into product decisions?

Don’t get me wrong; I completely support making INFORMED decisions, but the mantra of experimentation is getting out of control. Just because you didn’t ask someone some specific questions(s) about some issue in the past week, it doesn’t mean you can’t make decisions about that issue.

Between Wild Ass Guesses and Calculations

Take a look at the the diagram.

On the left, we have the (infamous) Wild Ass Guess (WAG). Sometimes called “gut feel” — though IMHO gut feel is a bit to the right of WAG — this is a decision based on little or no data and has huge uncertainty.

On the right, we have what I call Solid Calculation. This is a decision made with a complete set of high confidence data and is clearly understood as fact.

In the middle is what I call the Decision Zone. It’s skewed to the right of center, and there is a reason for that. Here, there is some or a lot of data, but it’s never 100% of the data. Thus it’s a decision and not a calculation.

It’s skewed to the right because you want to have a SOME level of information (explicit or implicit), to base your decision on, and as you get further to the right it starts turning more into a calculation.

We are paid to make good decisions

In his book Blink, Malcolm Gladwell writes:

“The key to good decision making is not knowledge. It is understanding. We are swimming in the former. We are desperately lacking in the latter.”

Experiments and particularly A/B testing give us data and some knowledge, but where do understanding and insights come from? You can’t A/B test your way to deep insights. A/B testing different UIs on a website will tell you WHAT users prefer, but it will never tell you WHY they prefer what they prefer.

These come from experience, hindsight, domain knowledge, deeper research and the like. In life, we use these traits everyday to make both big and small decisions, so what about at work?

In fact, one of the main jobs of good managers, including Product Managers, is to make good decisions.

VERY successful products and businesses can and have been been built without excessive amounts of testing and constant experimentation. Apple is the obvious company that comes to mind, and you can read a short post here about how Apple views customer experimentation and iterative feedback.

"Instead of “launching and learning”, Apple waits until their products are much more mature and offer a more complete experience...Now I’m sure you’re thinking “but Apple is Apple and nobody else is”. Yes, that’s true, but consider that this is just another process, albeit a different one. At Apple they have internal checks-and-balances about when to launch a product or not, and it stands in stark contrast to the lean approach. "

So, let’s continue to make informed decisions, but also, let’s decide that not every decision requires A/B testing or customer experimentation before it can be made. Let’s trust in ourselves and in our coworkers enough so that we don’t waste time (ours and our customers) in unneeded data collection activities, simply to tell us what we should already know.

This article originally appeared on LinkedIn and has been republished with permission.

There are many more product training at Tolpagorni

Increase your skills and toolbox in any specific area; Product marketing, Value engineering, Strategy, Product Planning or Product Owner. If you are more senior, train to master the complexity of a High Performance Product Manager.

Check all our trainings


About Saeed Khan

Saeed has been a thought leader and thought provoker throughout his 25 years working in high-tech. He started the influential blog On Product Management in 1997, and has been writing and speaking on the discipline of Product Management and new product development for a number of years. He is a co-founder of Transformation Labs, an advisory firm that helps companies harness innovation, product management and product marketing to drive growth and market success.

We build high-performing Product Management

Markets and technology continue to change. Stay up-to-date as a Product Manager with Tolpagorni training's.

Join in!

No thanks, we enjoy struggling!

Strengthen your business with Smart Consulting

Product managers are explorers.
We have the skills to hike the extra mile. And the courage to take on challenges. We hope to hear from you!

Contact us for free Consultation

We believe in smart consulting with hypothesis creation and executable deliverables. We know we can support your product management in creating outstanding products that grow your business!