How to Replace Excel As the Product Management Tool of Choice – Product Talk with Nils Davis

I recently had a very interesting conversation with Nils Davis, who is the former product manager of Accept 360s product management tool about, what it takes for a product management tool to succeed.

The default tool of a product manager is Excel, basically a list of things you need to do, so the first observation is that a product management tool has to do something better than excel.

It is necessary to take some thoughts out of the mind of the product manager and put them into the tool, so he/she doesn’t need to worry about them. Something Nils has detailed in another blog post

Three things that excel doesn’t do well that a product management tool should do grew out of our conversation.

It should be possible to record relationships. For example if a several customers wished for a feature it should be possible to mark a requirement or feature with that customer. That way you can take a report with you to the client and report the status of that feature, when it is scheduled or whether it is part of a new release. IT would also be good to mark a feature as addressing a particular market or segment so you can make a report to document what you do for this particular market.

Another type of relationships is dependencies. I brought this up because it is something we hear sometimes from customers. If one feature depends on another it should be possible to record that in the system. My problem with this is that I rarely see anyone use it. When they do you quickly get everything tangled up in crossing relationships and don’t know how to get out of it. But Nils had a good reason for this: no one ever made a really good visual way for dependencies to make sense for the user.

When you work with features it is important that you can divide a feature up into more parts. A feature could for example be authorisation and the sub features could be traditional log in, linked in authorisation, google+. Then it should be possible to follow up on the status of the whole feature by rolling up the status of the sub features. This would also support the workflow of increasing specification of a suggested feature. We may first agree that we need a log in function, then we will make sub-requirements for design of screen, password validation, account activation, password restore etc.

It is almost impossible to collaborate in excel, so I asked Nils whether this was also important, since we chose to focus on making exactly this easy in Sensor Six. Nils agreed that this indeed was a feature worthy of inclusion, since very often you need input to your excel sheet from various stakeholder and it is quite difficult to send an excel sheet around to different persons. You would want the tool to make it possible to get input easily.

I think this is a good test to look for in a product management tool. Is there one or more of these things that the tool does 10 times better, and you need that feature, it should be worthwhile to invest in it.

Being new in product management

Every day new product managers are recruited, but not so much is written about what it is like being new in this profession. To make up for that, we talked to Christine Luc who is building digital products at a medium-sized skincare company. Before this she was in a few early stage start ups, but still relatively new to the job as product manager.

Who wants to be a product manager?
People grow up dreaming about becoming doctors or lawyers, but why would any one want to become a product manager? We asked Christine why she decided to become a product manager:
“My background is in marketing. When you are in marketing you work with what you have today. You can’t change the product or anything about it. When you are in a marketing role you listen to complaints and it is hard not being able to help the customer more. You can’t change the product and go to the root cause. This was my motivation to move to product management”

But moving into this new position is not without challenges. According to Luc the most difficult thing in the role as product manager is to keep everyone in sync with what is happening. “my earlier stance was that there should be complete transparency with all stakeholders, but the problem is that there is no time dimension to that wish”. It takes some time to communicate and things may change. It could be that the PM knows that a problem is going to be resolved soon, so then it may be better not to share the problem with everyone, since it could provoke confusion and panic. On the other hand it is important to keep everyone on the same page and not let rumors appear in the organization. It is important that there are no inconsistencies in what different parts of the organization thinks or knows. That problem is bigger in larger enterprises than small start ups where everyone knows each other. It is of central importance to get the right information to the right people at the right time.

Being a product manager, we asked Christine what it was about the job that kept her coming back to work every morning. The challenge to see the gradual progress of the product and being part of something that has an impact on the company and the morale of coworkers is central: “internal stakeholders are encouraged when they know that they are part of something wonderful”

Who should go into product management?
According to Luc product management may not be for anyone though. If you don’t like talking to customers, like the status quo and tend to be good at pointing out problems, you may be more happy in another product role like, say, tech lead or test engineer. If on the other hand you are an optimist, a change enabler and you love talking to many different people, product management may be just the thing for you.

A/B testing for product managers

Neil McCarthy is Director of Product Management at Yammer where he has worked for the past three and a half years. Coming from an education in electrical engineering he has worked for the past 10 years in enterprise software in roles bordering between the business and the technical side.

At Yammer they decided early on to become a data informed company and invested heavily in an infrastructure to support this along with a team of data scientists. Today, no new feature is released without an A/B test.

Why A/B test your product?
I asked Neil what A/B testing can do that other methods for getting customer feedback, such as focus groups and surveys, can’t do.

“A/B testing helps product teams move faster by helping them build the right things and validate their assumptions along the way. A/B testing is a great way to test an idea you already have, but it’s not a great a way to come up with new ideas. Gathering user feedback and thinking strategically about the future of the product and industry is a better way to come up with good ideas.”

At Yammer they also do qualitative and quantitative research post project to figure out what people are actually doing. This plays a big part in figuring out what happened when a test fails.

One example of such a test that turned out to be worse than baseline was when they decided to try to alter the sign up flow. Conventional wisdom has it that the more friction you take out of the sign up flow the better the retention of the customer. So, Yammer hypothesized that by taking out a few steps of the sign up flow and putting them into the product, they could increase long term retention. But to their surprise it turned out that taking out these steps had the opposite effect. The sign up flow was helping users understand what Yammer is. Therefore they did not keep the change and instead left the sign up flow as is. Another example of something that was a success was when they tested whether including a module in the feed that suggested the user to follow other users that their friends followed. It turned out that a lot of users started to follow others and this resulted in a lift in the core metric of days engaged.

How to test
Yammer is not Twitter or Facebook who can do significant tests with only 1% of their users. Instead, Yammer usually tests on 50% of their users. Still it take minimum 2 weeks to do a test. The problem is that since you are testing hypotheses, some of which are proven incorrect, it feels like the advancement of the product is slower. In actuality, you’re moving faster because you eliminate a lot of waste and complexity by not implementing features that are unsuccessful.

“The core of A/B testing is to have a hypothesis. At Yammer hypotheses are rigorously formulated into if/then statements. For example “if we increase the priority of groups, then more users will get work done in Yammer”. This will be broken down into smaller hypothesis that can more easily tested, like: “If we increase the prominence of the group join button then more users will join groups and engage more frequently with Yammer”.

How to avoid local maximum
A well known problem with A/B testing and any other incremental test method is the problem of the local maximum. This happens when a product reaches the point where small changes no longer significantly improve it. At Yammer they have avoided local maximum problems by periodically taking big bets, where they work on really big features. Even for bigger features, they’ll break down the project into small pieces so they can execute incrementally.

Getting started with A/B tests
I also asked Neil what he thought the current best practice for A/B testing was. Here is a list of four key ingredients in successful A/B testing for product managers.
1) Having the right hypotheses is necessary. If you don’t have well informed hypotheses, A/B testing will not help you no matter what degree of technical perfection you have.
2) Log everything users do. This is not to help the A/B test in itself, but in order to understand post hoc, what happened. Why did the test go wrong? Why did the users not react as expected?
3) Have a solid A/B testing framework in place. Without the technical framework to do it you won’t succeed.
4) Put statistical rigor into guidelines for conducting the A/B tests. You need to make sure you are considering statistical significance when looking at the results so you only conclude on true positives.

What’s around the product?

Today and for the last 10-15 years we have seen an increased focus on the product and bringing it to market ever faster. Lean, Scrum and agile methodologies champion this view. But maybe it is better to extend this narrow focus on the product to reflect on what is around the product instead.

This is the point that Steve Johnson is trying to make when we talked to him recently. Steve is the founder of Under 10 Consulting, a product management consulting company based on the belief that minimal processes and simple templates will result in world class products. Steve was educated in computer science and marketing. He started in programming, but moved into product management and later joined Pragmatic Marketing, where he worked for 15 years.

Focus on the product!
Product focus is, according to Steve, a good thing, but there should be more to product management than product. What about the promotion, selling, support, and services? These are all part of the “product” from the customer’s perspective. An example is Lean Startup where the focus is on making changes to the MVP in order to make it more and more attractive to the customer until you achieve product market fit. But, just because the response is not as good as we’d hoped, maybe we don’t need to change the product or add another feature. Maybe the problem is the promotion of the product or the positioning. For example, you could make the best action film with car chases and plenty of explosions and still get bad response from customers if you happened to promote it in a children’s film festival. That doesn’t mean the product is a failure; it doesn’t mean you should pivot.

The MVP is often too minimal

“Lean Startup will take you up to version 1.0, but what is interesting comes after. The MVP is the bare minimum product, but you don’t get joy from that. We saw that with the iPhone. What they did with the first version was clearly minimal. They didn’t deliver every possible feature in a minimal fashion; they delivered a few critical features brilliantly. And it set a new expectation for all phones. We all knew that future releases would include photos and video and more apps”.

According to Steve, today’s obsession about product means that not enough thinking goes into developing the whole product, and how it is promoted and sold.

The history of product focus
Since he is a veteran I asked Steve whether it had always been like this, but that didn’t seem to be the case. Back in the eighties there was plenty of awareness that a product had to be marketed and positioned. There was an understanding that product management involved business understanding. In “Crossing the Chasm” Geoffrey Moore saw the product manager’s role as bringing the product from idea to implementation, and then a product marketing manager took it from implementation to market. In the mid-90s, the team behind Scrum created a great framework for producing software where a product owner was supposed to give guidance on what to build. Unfortunately the product owner role turned into something that was carried out by a junior person or work someone would do in addition to their “real” job. This has had the effect that the product manager has been pushed back to the technical side of the job and the business side is not adequately represented in product development:

“If you ask developers what they need in a product owner role they say they want to understand more about the market and users. Marketing want people bright on technology. Sales want someone who is an expert on the domain. Executives expect someone to run the product as if it were a business. Those are very different views on what product management is.”

Therefore focus comes to be on the product in isolation. The effect is that what gets built is not always an optimal product, not a product that will delight customers. Johnson uses the metaphor of a movie:

“ In a movie the developers are the artists. Similarly programming is an art form. The product manager is like the director. Let’s say he walked around the set and the caterer says to him ‘I would like to have some cute bunny rabbits in the film,’ the producer demands car chases, and the actors put drama into it although it is supposed to be a comedy. The director, who may be straight out of college or doing it half time on his way home to dinner with his family, just lets them have their way. That would probably not end up as a film customers would love.”

It is unfortunately the process followed by many companies today, where product owners and the process by which they work are not highly prioritized. Quality often comes as a third or fourth priority, according to Johnson.

Favorite product
As a final question I asked Steve what his favorite product was.

“My Kensington remote control for presenters. I do presentations often and usually the organizer will bring me a remote that is actually a mouse. I often use some gags as the first few slides, so just taking the remote I have already gone through the first three or four slides by accident. Kensington is great because it is specifically designed for presenting. You can go forward to the next slide or back. That’s it. That is just so simple! It is also relatively heavy. I used to have one that was very light but I often dropped it because I forgot I had it in my hand. Another great feature is that they put 2GB of data on it, so I can bring the presentation within the remote. It just does one thing, but it does it really well. I do wish it had a super bright laser pointer though. That is the only complaint I have with this product.”

If you want to read more about Steve’s ideas and views buy his book “From Fragile to Agile: The business of agile product management.” It is out on Amazon in both ebook and paperback. Definitely worth a read.
You can also read more about his company Under 10 Consulting
Steve will be speaking on “Have we LEAN’d too far” at the Business of Software conference in September 2014.
Photo by flickr user James Willamor