Your customers aren't lab rats


Everywhere you turn these days, someone is talking about running experiments and tests to understand your customers or target market.

They say things like:

  • use experiments to validate your ideas
  • A/B test responses to interface designs
  • build minimal solutions that can be built and released quickly to get ongoing feedback.

It seems that many product people can no longer make a decision without first testing everything out on users. Some would say:

“Of course, why wouldn’t you do things that way?”

but I’m going to take a counter point.

Just because you CAN run lots of experiments or A/B test almost everything, doesn’t mean you SHOULD or that it’s the right thing to do.

In a recent discussion about improving sales effectiveness, someone actually suggested running an A/B test with sales teams; training one group one way, the other, another way, and then seeing which group did better.

Clearly this person didn’t understand that A/B testing requires ALL other variables to be held constant which would be impossible for different sales teams, with different territories, customers, objectives etc.

Additionally, let’s be honest…with exceptions such as A/B testing specific web page designs/layout over MANY (thousands or millions of) impressions, most experimentation and user validation is done empirically with small numbers of customers.

And the results themselves may be skewed or contain significant margins of error.

  • Did you ask the right questions?
  • Did you ask the right people?
  • Did you use the right tests?

Is anyone measuring that when the feedback is incorporated into product decisions?

Don’t get me wrong; I completely support making INFORMED decisions, but the mantra of experimentation is getting out of control. Just because you didn’t ask someone some specific questions(s) about some issue in the past week, it doesn’t mean you can’t make decisions about that issue.

Between Wild Ass Guesses and Calculations

Take a look at the the diagram below.



On the left, we have the (infamous) Wild Ass Guess (WAG). Sometimes called “gut feel” — though IMHO gut feel is a bit to the right of WAG — this is a decision based on little or no data and has huge uncertainty.

On the right, we have what I call Solid Calculation. This is a decision made with a complete set of high confidence data and is clearly understood as fact.

In the middle is what I call the Decision Zone. It’s skewed to the right of center, and there is a reason for that. Here, there is some or a lot of data, but it’s never 100% of the data. Thus it’s a decision and not a calculation.

It’s skewed to the right because you want to have a SOME level of information (explicit or implicit), to base your decision on, and as you get further to the right it starts turning more into a calculation.

We are paid to make good decisions

In his book Blink, Malcolm Gladwell writes:

“The key to good decision making is not knowledge. It is understanding. We are swimming in the former. We are desperately lacking in the latter.”

Experiments and particularly A/B testing give us data and some knowledge, but where do understanding and insights come from? You can’t A/B test your way to deep insights. A/B testing different UIs on a website will tell you WHAT users prefer, but it will never tell you WHY they prefer what they prefer.

These come from experience, hindsight, domain knowledge, deeper research and the like. In life, we use these traits everyday to make both big and small decisions, so what about at work?

In fact, one of the main jobs of good managers, including Product Managers, is to make good decisions.

VERY successful products and businesses can and have been been built without excessive amounts of testing and constant experimentation. Apple is the obvious company that comes to mind, and you can read a short post here about how Apple views customer experimentation and iterative feedback.

"Instead of “launching and learning”, Apple waits until their products are much more mature and offer a more complete experience...Now I’m sure you’re thinking “but Apple is Apple and nobody else is”. Yes, that’s true, but consider that this is just another process, albeit a different one. At Apple they have internal checks-and-balances about when to launch a product or not, and it stands in stark contrast to the lean approach. "

So, let’s continue to make informed decisions, but also, let’s decide that not every decision requires A/B testing or customer experimentation before it can be made. Let’s trust in ourselves and in our coworkers enough so that we don’t waste time (ours and our customers) in unneeded data collection activities, simply to tell us what we should already know.

This article originally appeared on LinkedIn and has been republished with permission.

Saeed Khan is one of our main speakers at the Product Leadership Day  2018. Let's find out more about our conference in Stockholm.

About The Author

Saeed has been a thought leader and thought provoker throughout his 25 years working in high-tech. He started the influential blog On Product Management in 1997, and has been writing and speaking on the discipline of Product Management and new product development for a number of years. He is a co-founder of Transformation Labs, an advisory firm that helps companies harness innovation, product management and product marketing to drive growth and market success.