Experimentation in product management

 Traditionally new products were developed according to the founder’s idea that was written down, which the engineers built. The last few years this pattern has changed.  Across the internet there has been a shift in mindset to bring the customer into what we are building. There is a growing awareness that we are wrong about what the customer wants most of the time. Therefore it is necessary to experiment to find out what customers want.

We talked to Teresa Torres about the role of experimentation in product management. The greater part of her career has spent in pre product market fit internet start ups, so if someone should know how to experiment to find a product that is successful it’s Teresa. Today she helps companies make better product decisions as a consultant and coach.

According to Torres it is better to start thinking about product development in terms of experiments rather than requirements. In Marty Cagan’s dual-track scrum article, he recommends using a discovery track and a delivery track. First we should experiment in the discovery track to identify what the right thing to do is. In the discovery track there should be a lot of experimentation in order to to inform what to build. Today there is a tendency to build any and every idea.

But real experiments require quite a bit of rigor and experience in designing the experiment.

“This is my primary focus as a coach. Many teams start to experiment but don’t have the experience to do it well. Most of us don’t have strong science or statistics backgrounds. What happens in practice is instead of generating informed hypotheses and designing experiments to test those hypotheses, we are testing anything and every thing  The problem with this approach is that we risk false positives.  We are running tens and sometimes hundreds of experiments, many with way too many variations.  This guarantees that we will see many false positives – changes that look good but really have no impact.  As a result, we are making decisions on bad data. If we want to build better products,  we need to understand how to run good experiments. The business side needs to be more scientific and the data science side needs to be more business oriented”

According to Torres the ready availability of experimentation tools like Optimizely and Visual Website Optimizer opens up the possibility for experimenting, but you need resources and expertise otherwise decisions will be made on faulty data. Part of the problem is the wide spread “Fear of Math”. Most people shy away from concepts like statistical significance and power. But it is necessary for product managers to begin understanding these concepts. Today there are many online resources that will teach you basic understanding of statistical concepts. Another problem is that we need to be better at hypothesis design. If you have not properly designed your hypothesis before you start you are not setting yourself up to get good data. We also need experimenters that can design experiments that can also test the hypotheses they are designed to test.

I asked Torres if there are any simple rules of thumb or best practices for product managers who want to get started.

“Don’t trust results that are not statistically significant. Surprisingly many teams are not even testing for significance. Define your hypotheses and decide upfront what decisions you will make if it passes, fails, or is flat . Otherwise you will just find yourself rationalizing after the fact why your change is still good regardless of what the data tells you.  Run the same experiment multiple times. It helps reduce false positives. There is no absolute truth. The world is always changing, something that didn’t work in the past may work in the future. Always keep a mindset that everything evolves.”

For more tips, see her article on The 14 Most Common Hypothesis Testing Mistakes (And How to Avoid Them)

It is up to you if you take Teresa Torres’ suggestion to start experimenting. In the mean time visit her excellent blog Product Talk and sign up for her newsletter. It is always packed with interesting content about product management.

What’s around the product?

Today and for the last 10-15 years we have seen an increased focus on the product and bringing it to market ever faster. Lean, Scrum and agile methodologies champion this view. But maybe it is better to extend this narrow focus on the product to reflect on what is around the product instead.

This is the point that Steve Johnson is trying to make when we talked to him recently. Steve is the founder of Under 10 Consulting, a product management consulting company based on the belief that minimal processes and simple templates will result in world class products. Steve was educated in computer science and marketing. He started in programming, but moved into product management and later joined Pragmatic Marketing, where he worked for 15 years.

Focus on the product!
Product focus is, according to Steve, a good thing, but there should be more to product management than product. What about the promotion, selling, support, and services? These are all part of the “product” from the customer’s perspective. An example is Lean Startup where the focus is on making changes to the MVP in order to make it more and more attractive to the customer until you achieve product market fit. But, just because the response is not as good as we’d hoped, maybe we don’t need to change the product or add another feature. Maybe the problem is the promotion of the product or the positioning. For example, you could make the best action film with car chases and plenty of explosions and still get bad response from customers if you happened to promote it in a children’s film festival. That doesn’t mean the product is a failure; it doesn’t mean you should pivot.

The MVP is often too minimal

“Lean Startup will take you up to version 1.0, but what is interesting comes after. The MVP is the bare minimum product, but you don’t get joy from that. We saw that with the iPhone. What they did with the first version was clearly minimal. They didn’t deliver every possible feature in a minimal fashion; they delivered a few critical features brilliantly. And it set a new expectation for all phones. We all knew that future releases would include photos and video and more apps”.

According to Steve, today’s obsession about product means that not enough thinking goes into developing the whole product, and how it is promoted and sold.

The history of product focus
Since he is a veteran I asked Steve whether it had always been like this, but that didn’t seem to be the case. Back in the eighties there was plenty of awareness that a product had to be marketed and positioned. There was an understanding that product management involved business understanding. In “Crossing the Chasm” Geoffrey Moore saw the product manager’s role as bringing the product from idea to implementation, and then a product marketing manager took it from implementation to market. In the mid-90s, the team behind Scrum created a great framework for producing software where a product owner was supposed to give guidance on what to build. Unfortunately the product owner role turned into something that was carried out by a junior person or work someone would do in addition to their “real” job. This has had the effect that the product manager has been pushed back to the technical side of the job and the business side is not adequately represented in product development:

“If you ask developers what they need in a product owner role they say they want to understand more about the market and users. Marketing want people bright on technology. Sales want someone who is an expert on the domain. Executives expect someone to run the product as if it were a business. Those are very different views on what product management is.”

Therefore focus comes to be on the product in isolation. The effect is that what gets built is not always an optimal product, not a product that will delight customers. Johnson uses the metaphor of a movie:

“ In a movie the developers are the artists. Similarly programming is an art form. The product manager is like the director. Let’s say he walked around the set and the caterer says to him ‘I would like to have some cute bunny rabbits in the film,’ the producer demands car chases, and the actors put drama into it although it is supposed to be a comedy. The director, who may be straight out of college or doing it half time on his way home to dinner with his family, just lets them have their way. That would probably not end up as a film customers would love.”

It is unfortunately the process followed by many companies today, where product owners and the process by which they work are not highly prioritized. Quality often comes as a third or fourth priority, according to Johnson.

Favorite product
As a final question I asked Steve what his favorite product was.

“My Kensington remote control for presenters. I do presentations often and usually the organizer will bring me a remote that is actually a mouse. I often use some gags as the first few slides, so just taking the remote I have already gone through the first three or four slides by accident. Kensington is great because it is specifically designed for presenting. You can go forward to the next slide or back. That’s it. That is just so simple! It is also relatively heavy. I used to have one that was very light but I often dropped it because I forgot I had it in my hand. Another great feature is that they put 2GB of data on it, so I can bring the presentation within the remote. It just does one thing, but it does it really well. I do wish it had a super bright laser pointer though. That is the only complaint I have with this product.”

If you want to read more about Steve’s ideas and views buy his book “From Fragile to Agile: The business of agile product management.” It is out on Amazon in both ebook and paperback. Definitely worth a read.
You can also read more about his company Under 10 Consulting
Steve will be speaking on “Have we LEAN’d too far” at the Business of Software conference in September 2014.
Photo by flickr user James Willamor