Posted on

Experimentation in product management

 Traditionally new products were developed according to the founder’s idea that was written down, which the engineers built. The last few years this pattern has changed.  Across the internet there has been a shift in mindset to bring the customer into what we are building. There is a growing awareness that we are wrong about what the customer wants most of the time. Therefore it is necessary to experiment to find out what customers want.

We talked to Teresa Torres about the role of experimentation in product management. The greater part of her career has spent in pre product market fit internet start ups, so if someone should know how to experiment to find a product that is successful it’s Teresa. Today she helps companies make better product decisions as a consultant and coach.

According to Torres it is better to start thinking about product development in terms of experiments rather than requirements. In Marty Cagan’s dual-track scrum article, he recommends using a discovery track and a delivery track. First we should experiment in the discovery track to identify what the right thing to do is. In the discovery track there should be a lot of experimentation in order to to inform what to build. Today there is a tendency to build any and every idea.

But real experiments require quite a bit of rigor and experience in designing the experiment.

“This is my primary focus as a coach. Many teams start to experiment but don’t have the experience to do it well. Most of us don’t have strong science or statistics backgrounds. What happens in practice is instead of generating informed hypotheses and designing experiments to test those hypotheses, we are testing anything and every thing  The problem with this approach is that we risk false positives.  We are running tens and sometimes hundreds of experiments, many with way too many variations.  This guarantees that we will see many false positives – changes that look good but really have no impact.  As a result, we are making decisions on bad data. If we want to build better products,  we need to understand how to run good experiments. The business side needs to be more scientific and the data science side needs to be more business oriented”

According to Torres the ready availability of experimentation tools like Optimizely and Visual Website Optimizer opens up the possibility for experimenting, but you need resources and expertise otherwise decisions will be made on faulty data. Part of the problem is the wide spread “Fear of Math”. Most people shy away from concepts like statistical significance and power. But it is necessary for product managers to begin understanding these concepts. Today there are many online resources that will teach you basic understanding of statistical concepts. Another problem is that we need to be better at hypothesis design. If you have not properly designed your hypothesis before you start you are not setting yourself up to get good data. We also need experimenters that can design experiments that can also test the hypotheses they are designed to test.

I asked Torres if there are any simple rules of thumb or best practices for product managers who want to get started.

“Don’t trust results that are not statistically significant. Surprisingly many teams are not even testing for significance. Define your hypotheses and decide upfront what decisions you will make if it passes, fails, or is flat . Otherwise you will just find yourself rationalizing after the fact why your change is still good regardless of what the data tells you.  Run the same experiment multiple times. It helps reduce false positives. There is no absolute truth. The world is always changing, something that didn’t work in the past may work in the future. Always keep a mindset that everything evolves.”

For more tips, see her article on The 14 Most Common Hypothesis Testing Mistakes (And How to Avoid Them)

It is up to you if you take Teresa Torres’ suggestion to start experimenting. In the mean time visit her excellent blog Product Talk and sign up for her newsletter. It is always packed with interesting content about product management.

Posted on

Is Google Glass the Next Segway and What Can We Learn From It?

About a decade ago one of the most hyped products in Tech was the Segway. It promised to revolutionize public transportation. The hype was almost hysterical. It was impossible to get your hands on one, unless you were especially favored and it was overprized. The hype for google glass is similar today, only a select few can get it and it is overprized. Google glass could have a similar fate to the Segway confined to a niche domain like mall cops or tourist excursions, although it has yet to find this niche. This niche could be internet porn, surgery aid or factory repair.

However, the reason is not the overhyping, price or lack of accessibility. It is a much deeper problem. Something as simple as product market fit. Let us look at a couple of reasons for this.

Product market fit
The Segway was by all accounts an impressive innovation and well thought out idea. It was just not thought out for any particular person or problem. It was technology for technology’s sake. It seems that at no point did the product manager/designer/inventor stop to test whether this product actually addressed an existing pain in any significant customer group.

The reports of google glass is similar. We frequently hear that this is cool technology. People ask and are interested in people wearing them. They want to try it. It works very well. But people have a hard time imagining what they would use them for.

“Glasshole” – how one feature can overturn the whole product
Lately google has been catching a lot of bad press around glass due to its camera function. A new term has even appeared already before it has been released, namely “glasshole”. There is actually only one function, which is the reason for this, that is the camera mounted on the glasses.

It appears that people other than the google glass users do not appreciate the possibility of being taped by the glass wearer. This one function is now threatening to take down glass with it. This problem could have been detected much earlier and addressed if someone had bothered to listen to someone other than the over exited test users.

What to do?
A product has to exist in an environment beyond its immediate users. Analysis of this environment and the humans that live in it could have revealed the emotional reactions. The solution by google has been to ban certain types of application of the camera. Which is a good idea, but most people don’t know that and therefore it won’t make much of a difference in perception.

Google could have considered two other options that are even more simple and would have an immediate effect in the environment where the product would be used. Since the basic problem is that people don’t want to feel that they can be monitored covertly, that is, taped without their knowing it. The first option is to introduce a visible light that is turned on when the camera is recording. This would make it possible for people to know when they were being filmed similar to when people with cell phones film. That is not a problem because it is out in the open. You can go to the person and complain.

The second option is to get rid of the camera altogether from the core product. It could be a visible add on, like a go pro camera, that could be taken off at the request of other people or just courtesy. Again it is important that the immediate environment can see that it is there. The problem here is that one controversial feature has been embedded within the product.

With out the camera google glass would still be an impressive product, that could show you the way, notify you of appointments and emails while driving etc. Sometimes taking out a feature may actually increase the market potential.

Anyhow, I love the google glass idea because it shows that somebody out there has the courage to dream and build products that are beyond what we know. I hope that google glass will find its niche faster than the Segway though.