What is a successful product?

Every company wants to be a success. One key ingredient in that is successful products. A successful product will look different to different companies, but usually it can be tracked by KPIs. For any business it is very important to find the right KPI and be wary of so called vanity metrics.

What’s your “On Base Percentage”
In Michael Lewis’ “Moneyball” the Oakland A’s changed their scouting and talent acquisition philosophy to focus on only one metric. In the world of baseball and the hundreds of possible metrics you could use it sounds like insanity. Never the less the philosophy went from the premise that the number of bases is what wins games. Not number of home runs, how handsome the batter looks or stolen bases. It all comes down to one thing: “on base percentage”, how often will the player get on base. Never mind how he is doing it or how he looks. Now this insight allowed the Oakland A’s to perform above average given their limited budget and exploit some loop holes in the market.

This is a powerful reminder that focusing on one thing is very powerful. It will eliminate discussions about which of the KPIs is more important in case one goes up and another goes down. It will also ensure that everyone is one the same page and no-one is in doubt about what success looks like. So the first step towards product success is to find your “On base percentage”

Different types of KPI
There are two main types of KPIs objective and subjective. They have different properties and can be used in different circumstances.

Objective KPIs are the best, because they pick out a measure that can be verified regardless of human perception and interpretation. They are things that can be measured like: Frequency, volume, amount, duration. Most of the classical webanalytics and economic parameters fall in this class. A good example of an objective KPI is downloads per day for an app. There is no way you can argue with that. The same goes for units sold per day.

Subjective KPIs measure things depend on human subjective assessment like interpretation feeling and interpretation. That does not mean that they need to be less quantifiable. When you measure satisfaction, there are many good frameworks for doing that. It could be Facebook likes, customer satisfaction ratings, retweets, NetPromoter Score, shares etc. These are all quantifiable measures of a subjective quality, whether that is satisfaction, interest, pride or something else.

The business model determines the KPI
The absolute central KPI will always be monetary, like revenue, margin, equity and the success of a product will in most cases be measured directly by the amount of value it generates. The typical product will be measured by the revenue it generates. It could be margins as well, but it is a bit more complicated to use that since the company´s cost structure doesn’t necessarily have any connection with the success of the product. Equity in that sense is even more complicated to track (but more about that below).

Some people may object to money being the central purpose of a product, especially if they are of an idealist anti-capitalist inclination. But since money pays for salaries electricity the success of the product must somehow be traceable to this. Even if you are a non profit charity organisation you need to pay the rent, electricity and taxes. Some products are offered for free and are viewed as successes even though they generate little or no revenue, such as twitter, Facebook, snapchat. But the reason they are valuable in the eyes of investors is the promise of revenue, a kind of deferred revenue. Typically the KPI is number of users. But this figure is usually recalculated into revenue by a value that you think you can generate from them (eg. $10 per user). In that sense you are back at money as a measure for product success.

The most straight forward measure of product success is the revenue it generates. The reason is that it is the cleanest and most intuitive way. Nobody can argue with that figure – money in the bank is money in the bank. Sometimes that is more important than anything else, since you can’t argue with liquidity. Examples of these types of products are SaaS products where it is popular to track Monthly Recurring Revenue per product. That is an accepted standard. More traditional software companies may track revenue from licenses sold. Transaction based companies like credit card companies or other types of infrastructure companies who charge by transaction may track the number of transactions.

It may however occur that the revenue is not directly referable to the product in itself. That is the case for many free products (not freemium products). A good example are open source software products. If we take the database Cassandra as an example. It is free and can be downloaded. It is built and maintained by Datastax which received gazillion in funding recently $45 million ($83,7 in total). You could track Cassandras success in installed base. Then again, most people need support, since it is after all a complicated product, and guess who is selling SLAs, support and implementation consultancy for Cassandra? Datastax, so again the KPI is a proxy for revenue (more interestingly this creates a drive towards making the product more needy of consultancy, so what looks like a successful product to the company is not necessarily the same as to the customer in this case).

Revenue is a good around indicator if the business model dictates you earn money on all the units shifted.

Margins are a more precise way of tracking product success because it directly supports business success. Revenue may not, since you could be selling products for a lower cost than the production cost. More sales would therefore not lead to success for the company. Margins will almost always do that.

It is a good thing to track margins when cost per unit is easy to calculate. That is usually the case for hardware and physical products, where cost per unit is often already calculated. So, in the manufacturing industry that would be a good measure. The same goes for retail, where the cost per unit is the purchase price, plus transport and storage. Sometimes the margins are just calculated as the purchasing price minus the sales price. That is good rough measure of success.

In other industries, like the service industries, it is a bit more difficult. If we take a hotel, the cost per Unit is somewhat more difficult to calculate, since heating, cleaning and electricity is not typically calculated per room. The same goes for professional services, where one our sold may have the cost of the consultant for that hour, but also all the other hours he or she didn’t bill to any customers and that amount is very variable.

Margins are a good KPI for industries where the Cost per unit is easy to calculate and intuitive to understand.

In some cases revenue and margins may fall short as measures of success. To measure product success in terms of equity is more prevalent in isolated fields. In accounting terms, equity is the the amount of liabilities minus the assets. For a product that would mean how much you owe (for buying, producing, building etc.) minus the market value. That doesn’t make much sense in terms of serial produced products like cell phones.
If we take real estate it makes a lot of sense. If the product is a house or an apartment, the ability of investments in this apartment to raise the market value is a very good measure of success. Something similar is also the case for incubators, where the start up itself is the product. Here you may want to track the investment against the market value in order to see whether the product is successful. I would guess that Y-combinator, Rocket internet or 500 start ups would track the equity of each individual start up rather than their revenue or margins.
One problem with equity though is that market value may be very hard to assess before you sell the product. That makes it hard to use as a proactive KPI. But usually there are other proxy indicators. In consumer technologies, the standard one is number of users, downloads, visitors etc. because that can be used to calculate predicted revenue and therefore market value.

Equity is a good measure if the product is unique.

Non monetary measures
There may still be instances where you would measure product success by some other KPI, which is not monetary. One example is satisfaction. Many products from state departments are offered as a service to citizens. They are still products, but they do not generate any revenue, margins or equity, indeed, they were never meant to. Instead they generate a service, the success of which can be measured by satisfaction in one sense or another.
Satisfaction is somewhat more difficult to measure, because it depends on subjective measures. A rating scale is a typical solution, but it could also be sentiment analysis from social media or clicks on icons such as likes and smileys. Number of issues registered or complaints related to a product would be another way of tracking satisfaction. Sharing on social media is similarly usually a good indication of satisfaction, but you can’t tell whether it is shared because of a positive or negative experience. Some public sector products are however not meant to have high user satisfaction ratings: Jails will not be measured by user satisfaction either, since the users are meant to not have great satisfaction from it.

Foundations are atypical, but they will typically have a KPI that can be derived from their particular purpose. If it is investing in alternative energies, or saving the world like the Bill and Melinda Gates foundation, then somehow a KPI would be found for that use. But then again, foundations typically don’t have that many products.

Non monetary measures are good when the business does not generate it’s operating budget from the product.

Product succes is key to the success of the entire enterprise. You should carefully study the business model of the enterprise and pick just one KPI as the measure of success. The KPI should be quantifiable and possible to track continuously. Otherwise you don’t have anything to steer after.

You could and should still track other KPIs if they in some way influence the central KPI. That way you can learn the best ways to optimise your product and see early warnings of problems.

Experimentation in product management

 Traditionally new products were developed according to the founder’s idea that was written down, which the engineers built. The last few years this pattern has changed.  Across the internet there has been a shift in mindset to bring the customer into what we are building. There is a growing awareness that we are wrong about what the customer wants most of the time. Therefore it is necessary to experiment to find out what customers want.

We talked to Teresa Torres about the role of experimentation in product management. The greater part of her career has spent in pre product market fit internet start ups, so if someone should know how to experiment to find a product that is successful it’s Teresa. Today she helps companies make better product decisions as a consultant and coach.

According to Torres it is better to start thinking about product development in terms of experiments rather than requirements. In Marty Cagan’s dual-track scrum article, he recommends using a discovery track and a delivery track. First we should experiment in the discovery track to identify what the right thing to do is. In the discovery track there should be a lot of experimentation in order to to inform what to build. Today there is a tendency to build any and every idea.

But real experiments require quite a bit of rigor and experience in designing the experiment.

“This is my primary focus as a coach. Many teams start to experiment but don’t have the experience to do it well. Most of us don’t have strong science or statistics backgrounds. What happens in practice is instead of generating informed hypotheses and designing experiments to test those hypotheses, we are testing anything and every thing  The problem with this approach is that we risk false positives.  We are running tens and sometimes hundreds of experiments, many with way too many variations.  This guarantees that we will see many false positives – changes that look good but really have no impact.  As a result, we are making decisions on bad data. If we want to build better products,  we need to understand how to run good experiments. The business side needs to be more scientific and the data science side needs to be more business oriented”

According to Torres the ready availability of experimentation tools like Optimizely and Visual Website Optimizer opens up the possibility for experimenting, but you need resources and expertise otherwise decisions will be made on faulty data. Part of the problem is the wide spread “Fear of Math”. Most people shy away from concepts like statistical significance and power. But it is necessary for product managers to begin understanding these concepts. Today there are many online resources that will teach you basic understanding of statistical concepts. Another problem is that we need to be better at hypothesis design. If you have not properly designed your hypothesis before you start you are not setting yourself up to get good data. We also need experimenters that can design experiments that can also test the hypotheses they are designed to test.

I asked Torres if there are any simple rules of thumb or best practices for product managers who want to get started.

“Don’t trust results that are not statistically significant. Surprisingly many teams are not even testing for significance. Define your hypotheses and decide upfront what decisions you will make if it passes, fails, or is flat . Otherwise you will just find yourself rationalizing after the fact why your change is still good regardless of what the data tells you.  Run the same experiment multiple times. It helps reduce false positives. There is no absolute truth. The world is always changing, something that didn’t work in the past may work in the future. Always keep a mindset that everything evolves.”

For more tips, see her article on The 14 Most Common Hypothesis Testing Mistakes (And How to Avoid Them)

It is up to you if you take Teresa Torres’ suggestion to start experimenting. In the mean time visit her excellent blog Product Talk and sign up for her newsletter. It is always packed with interesting content about product management.

Big Data From a Product Perspective – Different Views

The hype surrounding big data at the moment is reaching a climax. While it is evident that we have more and more data and that there are valuable insights hidden, the situation is different if we look at big data as the products that are actually offered.

If we look at big data from a product perspective I think the situation is a bit more mixed. As a product category big data is not yet mature enough to warrant these huge valuations, but that could happen if a couple of things happen. But first let us look at how big data is viewed by different groups

Different views on Big data
From inside the big data community the focus is on technologies like Hadoop, Hive, nosql databases and the companies supporting these technologies and a plethora of other more or less obscure (to the uninitiated of course) products that are part of the big data ecosystem. It is not closely related to business intelligence although it is vey much the same problem big data is solving.

If we look at how the media sees it we are looking at something similar to the invention of the wheel. Something that will have profound effect on human civilisation and the way we live for millennia to come.

Investors see big data investments like the quest for the holy grail (which might explain some of the silliness): Hortonworks has raised $248 million, Coudera $1,2 billion, Datastax $189 million, Elastic search $104 million, Couchbase $106 million etc. All of these companies don’t have a proprietary product, but support open source products. The business model is one of building closed source tools that let customers run the open source better.

The CEOs who invest in big data really just want a big pile of money. They are not interested in the curios patterns you can find like the correlation between search terms containing the word coconut and the migratory patterns of the African swallow. They see in big data a new way to make more money and just want to get to that immediately.

The CIO is usually completely sidelined in decisions involving big data. Maybe because he is increasingly becoming the custodian of legacy technologies, but the need for big data often come from isolated infrastructure projects or from business development.

Developers view big data as models of the real world with intricate detail like the matrix. Soon we will be able to model the entire universe and predict what will happen with big data technology.

What end users see is of an alarming complexity. You need to have semi programming skills in order to extract even simple queries. You also need to be adept at manoeuvring applications with hundreds of functions similar to the sys admin. This is often the case with open source development that usability suffers, because the community wants to take the product in different directions. Furthermore developers are users and they already know the product so there is no real pressure to make the product easy to use for the uninitiated.

What it really is? In the end big data may very well turn out to be just like the segway. I am not saying that it will only be used by mall cops and tourists, but rather that it might end up servicing very limited segments and industries with very specific needs.

Enter the genius – the five specialisations of the big data employee?
The problem today is that in order to get any value out of big data you need to be a virtual genius. you need to master at least four areas that are usually specialisations

  1. First of all you need to be a developer. You might not need to code an actual application if you are just using it for analytical purposes, but you need to be able to write code to extract the information you need one way or another.
  2. Second, you need to be an infrastructure architect and sysadmin because you need to set up a great number of servers and networks. You need to know about the multitude of different infrastructure elements.
  3. Third, you need to be a database administrator. You need to set up databases and maintain them. You need to set up ETL processes, sharding and the like (you do not have to worry about database schemas though).
  4. Fourth, you need to be a data scientist since you need to know a fair amount about machine learning algorithms in order to extract patterns from the data.
  5. Fifth, you need to be a business analyst. If big data is to make sense from a business perspective it is necessary to understand the business model, the revenue streams, the cost structure etc. You also need to know a fair amount about the customers like what parameters to segment them by and what their pains are.

Naturally you don’t have to have all that in one person. In principle it can be spread across several employees, but quickly you will have to hire a complete team in order to just get started, although it is still difficult to find specialists that know just one or two of these things. On top of this you need very tight integration, because big data is more integrated than other technologies.

If you succeed with this the problems are not over unfortunately. Most organisations already have established procedures where work is split up along the lines mentioned above. You have application developers, operations, DBAs, analysts and business developers. Each department has it’s own governance and procedures describing hand offs to other departments. Now you are asking the organisation to circumvent all of these established procedures.

So big data products still have a long way to go before they are ready for the mass market and the really big bucks.