Posted on

From the Super Bowl to Super Products

Later this evening Super Bowl XLIX is played. One of the teams playing has reached it more frequently than any other team in recent decades. The Patriots are a remarkable team, that we could learn a lot from about product development and winning against the competition in a highly competitive market (disclaimer – I always hope the Patriots loose so this is not a fan post).

Rarely has a team performed consecutively at such high a level for so long. That fact alone proofs that it can not just be explained as luck or random variation. There must be something fundamentally right about what they do. That I think is to be found in their culture. I think that culture could be transplanted into product development as well. After all some companies, like the Patriots, also consecutively produce amazing products.

The secret sauce

I read too many sports books, I know, but I excuse myself with the slim possibility that what I read could potentially be transferable to real life. Some years ago I read “Patriot Reign” by Michael Holley. He was the first to gain continuous access to Bill Belichick and the Patriots over a period of two years. And Belichick is the key to understand Patriots success and their secret sauce.

When Bill Belichick came to the Patriots they were nothing special. They had a losing record and no one expected anything from them. One of the first controversial moves was firing all-star quarterback Drew Bledsoe and substituting him with a 6th round draft pick rookie quarterback: Tom Brady. This move, in my opinion describes what is truly unique about the Patriots.

What this move shows is that the organization is a rigid meritocracy. It doesn’t matter what you did last year or even five minutes ago; it is what you are doing now that counts. It also means that you don’t hire the rock stars and if you do be ready to fire them the minute they don’t perform. A meritocracy is built around rewarding performance continuously. This is opposed to being rewarded for friendship with management, good looks, past achievements, loyalty or even a good sense of humour (although these things can be very good).

In a meritocracy you trust potential when it is demonstrated and give inexperienced employees a chance if they have proven worthy. Now this means that you also have to have a good idea about what merit is, that is, what counts as good and what counts as bad. If you are developing new products a 40 yard dash may not be the thing like it is for a football team.

 

Establishing a framework for merit

What you need to do is to establish a framework for merit. This framework doesn’t have to be very formal. You actually just have to have an idea of what good performance is and then measure employees up against this.

These performance goals should however be tightly aligned with the strategy of the company and the industry it is in. The Patriots are not necessarily looking at the same KPIs as everyone else. They have found their own KPIs based on their analysis of the game. This is interestingly visible in the recent “Inflateagate”: It was discovered that the balls the Patriots played with were not properly inflated. Puzzling as that information it is tied very closely to strategy and performance goals. A recent article by Warren sharp uncovers the reason

The article starts from a peculiar fact: The Patriots were freakishly outperforming all other teams in number of plays per fumble since 2006 when a new rule was made that allowed you to bring your won balls. Why are fumbles important? It turns out that you significantly increase your chance of winning a game if you have fewer fumbles than your opponent. The more plays you have per fumble the better.

Part of the Patriots strategy was to improve plays per fumble, since this would increase their chance of winning. One way of doing this is to deflate the balls, but I am sure that fumble ratio is a very important parameter in player evaluations as well.

 

From Super Bowls to Super products

This is an example of how an analysis of the game or market if you are a tech company can help you spot a weakness to exploit. The challenge is then to find a KPI and establish a framework of merit for evaluating performance and then implement a rigid meritocracy around these insights. Doing that will help you consecutively outperform the competition because your organization will consecutively produce super products.

Posted on

What is a successful product?

Every company wants to be a success. One key ingredient in that is successful products. A successful product will look different to different companies, but usually it can be tracked by KPIs. For any business it is very important to find the right KPI and be wary of so called vanity metrics.

What’s your “On Base Percentage”
In Michael Lewis’ “Moneyball” the Oakland A’s changed their scouting and talent acquisition philosophy to focus on only one metric. In the world of baseball and the hundreds of possible metrics you could use it sounds like insanity. Never the less the philosophy went from the premise that the number of bases is what wins games. Not number of home runs, how handsome the batter looks or stolen bases. It all comes down to one thing: “on base percentage”, how often will the player get on base. Never mind how he is doing it or how he looks. Now this insight allowed the Oakland A’s to perform above average given their limited budget and exploit some loop holes in the market.

This is a powerful reminder that focusing on one thing is very powerful. It will eliminate discussions about which of the KPIs is more important in case one goes up and another goes down. It will also ensure that everyone is one the same page and no-one is in doubt about what success looks like. So the first step towards product success is to find your “On base percentage”

Different types of KPI
There are two main types of KPIs objective and subjective. They have different properties and can be used in different circumstances.

Objective KPIs are the best, because they pick out a measure that can be verified regardless of human perception and interpretation. They are things that can be measured like: Frequency, volume, amount, duration. Most of the classical webanalytics and economic parameters fall in this class. A good example of an objective KPI is downloads per day for an app. There is no way you can argue with that. The same goes for units sold per day.

Subjective KPIs measure things depend on human subjective assessment like interpretation feeling and interpretation. That does not mean that they need to be less quantifiable. When you measure satisfaction, there are many good frameworks for doing that. It could be Facebook likes, customer satisfaction ratings, retweets, NetPromoter Score, shares etc. These are all quantifiable measures of a subjective quality, whether that is satisfaction, interest, pride or something else.

The business model determines the KPI
The absolute central KPI will always be monetary, like revenue, margin, equity and the success of a product will in most cases be measured directly by the amount of value it generates. The typical product will be measured by the revenue it generates. It could be margins as well, but it is a bit more complicated to use that since the company´s cost structure doesn’t necessarily have any connection with the success of the product. Equity in that sense is even more complicated to track (but more about that below).

Some people may object to money being the central purpose of a product, especially if they are of an idealist anti-capitalist inclination. But since money pays for salaries electricity the success of the product must somehow be traceable to this. Even if you are a non profit charity organisation you need to pay the rent, electricity and taxes. Some products are offered for free and are viewed as successes even though they generate little or no revenue, such as twitter, Facebook, snapchat. But the reason they are valuable in the eyes of investors is the promise of revenue, a kind of deferred revenue. Typically the KPI is number of users. But this figure is usually recalculated into revenue by a value that you think you can generate from them (eg. $10 per user). In that sense you are back at money as a measure for product success.

Revenue
The most straight forward measure of product success is the revenue it generates. The reason is that it is the cleanest and most intuitive way. Nobody can argue with that figure – money in the bank is money in the bank. Sometimes that is more important than anything else, since you can’t argue with liquidity. Examples of these types of products are SaaS products where it is popular to track Monthly Recurring Revenue per product. That is an accepted standard. More traditional software companies may track revenue from licenses sold. Transaction based companies like credit card companies or other types of infrastructure companies who charge by transaction may track the number of transactions.

It may however occur that the revenue is not directly referable to the product in itself. That is the case for many free products (not freemium products). A good example are open source software products. If we take the database Cassandra as an example. It is free and can be downloaded. It is built and maintained by Datastax which received gazillion in funding recently $45 million ($83,7 in total). You could track Cassandras success in installed base. Then again, most people need support, since it is after all a complicated product, and guess who is selling SLAs, support and implementation consultancy for Cassandra? Datastax, so again the KPI is a proxy for revenue (more interestingly this creates a drive towards making the product more needy of consultancy, so what looks like a successful product to the company is not necessarily the same as to the customer in this case).

Revenue is a good around indicator if the business model dictates you earn money on all the units shifted.

Margins
Margins are a more precise way of tracking product success because it directly supports business success. Revenue may not, since you could be selling products for a lower cost than the production cost. More sales would therefore not lead to success for the company. Margins will almost always do that.

It is a good thing to track margins when cost per unit is easy to calculate. That is usually the case for hardware and physical products, where cost per unit is often already calculated. So, in the manufacturing industry that would be a good measure. The same goes for retail, where the cost per unit is the purchase price, plus transport and storage. Sometimes the margins are just calculated as the purchasing price minus the sales price. That is good rough measure of success.

In other industries, like the service industries, it is a bit more difficult. If we take a hotel, the cost per Unit is somewhat more difficult to calculate, since heating, cleaning and electricity is not typically calculated per room. The same goes for professional services, where one our sold may have the cost of the consultant for that hour, but also all the other hours he or she didn’t bill to any customers and that amount is very variable.

Margins are a good KPI for industries where the Cost per unit is easy to calculate and intuitive to understand.

Equity
In some cases revenue and margins may fall short as measures of success. To measure product success in terms of equity is more prevalent in isolated fields. In accounting terms, equity is the the amount of liabilities minus the assets. For a product that would mean how much you owe (for buying, producing, building etc.) minus the market value. That doesn’t make much sense in terms of serial produced products like cell phones.
If we take real estate it makes a lot of sense. If the product is a house or an apartment, the ability of investments in this apartment to raise the market value is a very good measure of success. Something similar is also the case for incubators, where the start up itself is the product. Here you may want to track the investment against the market value in order to see whether the product is successful. I would guess that Y-combinator, Rocket internet or 500 start ups would track the equity of each individual start up rather than their revenue or margins.
One problem with equity though is that market value may be very hard to assess before you sell the product. That makes it hard to use as a proactive KPI. But usually there are other proxy indicators. In consumer technologies, the standard one is number of users, downloads, visitors etc. because that can be used to calculate predicted revenue and therefore market value.

Equity is a good measure if the product is unique.

Non monetary measures
There may still be instances where you would measure product success by some other KPI, which is not monetary. One example is satisfaction. Many products from state departments are offered as a service to citizens. They are still products, but they do not generate any revenue, margins or equity, indeed, they were never meant to. Instead they generate a service, the success of which can be measured by satisfaction in one sense or another.
Satisfaction is somewhat more difficult to measure, because it depends on subjective measures. A rating scale is a typical solution, but it could also be sentiment analysis from social media or clicks on icons such as likes and smileys. Number of issues registered or complaints related to a product would be another way of tracking satisfaction. Sharing on social media is similarly usually a good indication of satisfaction, but you can’t tell whether it is shared because of a positive or negative experience. Some public sector products are however not meant to have high user satisfaction ratings: Jails will not be measured by user satisfaction either, since the users are meant to not have great satisfaction from it.

Foundations are atypical, but they will typically have a KPI that can be derived from their particular purpose. If it is investing in alternative energies, or saving the world like the Bill and Melinda Gates foundation, then somehow a KPI would be found for that use. But then again, foundations typically don’t have that many products.

Non monetary measures are good when the business does not generate it’s operating budget from the product.

Recommendations
Product succes is key to the success of the entire enterprise. You should carefully study the business model of the enterprise and pick just one KPI as the measure of success. The KPI should be quantifiable and possible to track continuously. Otherwise you don’t have anything to steer after.

You could and should still track other KPIs if they in some way influence the central KPI. That way you can learn the best ways to optimise your product and see early warnings of problems.

Posted on

How to manage a globally distributed product team

Building a product alone is difficult. Building it with a team is harder, but building it with a distributed team is even harder. Still, today it is rare that any company is building any product with a 100% local, in-sourced on site team. That is why one key ability in building a succesful product is to be able to manage a globally distributed team. That is not easy and there are some crucial differences between being a small start up and an established enterprise. Here is what I learned from teaching this at the University and doing it in practice.

 

Specific challenges – Startups vs Established Companies
A start up typically differs slightly in the way globally distributed work is carried out, from how more established companies go about it. A start up will typically have:

More freelancers compared to full time resources. An established firm will typically have an offshore team with a local project manager either through an agency or if it is a larger company an independent legal entity. Start ups don’t typically have those resources, so they rely on freelancers. The primary challenge of a start up is to locate the right freelance resources. Luckily services like oDesk, Elance and Fiverr makes it easy to get in contact with skilled people in off shore destinations. They also have a system of ratings to spot the good from the bad freelancers. You can get very far in identifying good prospects, but it is still difficult. If you need someone to code, do research or write you may want to check how they work first. It is common practice to do a tender for a small test task if it is a bigger (say, more than 50 hours of work) task you need to get done. Just design a task that will be similar to what you need and invite three people to do it. It is important that you remember this is a test, so you need to guide and inform them in exactly the same way. Also, don’t expect to use this: keep recruitment and operation separate. Another thing you need to do in advance is to list the criteria important to you and rate each contestant in the tender and compare them. This way you can also hold it against the price. It is also important to decide in advance whether you are looking for commodity or premium services. Even out there in off shore countries there is a relation between price and quality. Invite people according to this. If you need a premium copywriter, you will usually not be happy with the $5 per hour vietnamese freelancer. Even if he has four stars. You may on the other hand be very happy with him if you need a researcher to do product reviews, that you will yourself edit.
More specific individual tasks compared to a continuous flow of tasks. larger companies have a continuous flow of tasks. They may still work on projects, but when one project finishes resources are transferred to another. In a start up you will usually be restricted in funds and have much more defined tasks to be completed by off shore resources, like: design a landing page, implement an algorithm in PhP, define a colour scheme, set up a server etc. This shifts the management focus to being very clear and precise about what you want. If it is a specialised task you will usually not have the luxury of working with a resource who has any knowledge of what you do, so they need to be told very precisely what you want from them. This means you have to do a lot of work in preparation. This is by the way where most people fail and get a bad experience with off shore resources, but although the skill level may differ between the US and India it does not differ as much as you would think. The reason why you tend to hear about bad quality from India or other off shore destinations has more to do with them not being instructed in enough detail what to do.

General challenges
There are also some general management challenges when you work globally that are the same whether it is a team, a freelancer or an entire division they are:

Timezone differences. Timezones are usually a problem, but they can actually be an advantage. The best companies manage to build an organisation that works 24 hours a day. That means you can have 24 hours support and develop new functionality twice as fast if you do it right. You need to set up a rhythm for your company where you define the hand offs between timezones. For example a morning meeting in New York with the off shore development team team in Ukraine (where it is the afternoon) where progress and new tasks are discussed. And then again an afternoon session with the Singapore team who have just started working.
Cultural differences. yezz, let’s talk about the elephant in the room. Different countries have different patterns of behaviour, but they all expect everyone to behave like themselves and interpret behaviour in that light. There is ample research that specifies particular countries culture, which would be worthwhile to become acquainted with if you expect to do a lot of business with that country. From a management perspective the first step is to become aware that you may not make sense to your off shore team because of how you frame it. Working with many different cultures it may not be feasible to get to know one in detail, so a little sensitivity will take you a long way and do try to read something about that culture, maybe even on quota. On the other hand for core tasks I would strongly suggest to pick just one destination, that you are either familiar with, or is ready to become familiar with. For my own company I chose Moldova for the core development tasks. I know the country from travels, speak some Russian and it is a similar size to my Native country, Denmark. These things for me resulted in a close cultural match where we have a really good connection. For Western countries I usually recommend Eastern European countries although they are usually a bit more expensive, but you earn it back really fast by not having to do everything three times because of misunderstandings. They are generally very familiar with Western culture as well.
The human factor. Off shore resources are human too and you need to show that you care. If you can, go there every once in a while or invite them to you. No Skype/webex/hang out will ever replace face to face interaction. When we talk face to face there is so much more information and you build bonds. This is important in any type of collaborative endeavour and especially in software development. I went to stay with my off shore team for a week on several occasions. I think they thought it was weird that I was there in the office, but we do have a really good relationship today.

Development phases. it is important to think about what development tasks are carried out close to the home market and which can be carried out elsewhere. A rule of thumb supported by empirical studies is that the earlier design phases are best and most often placed in-house or close to the target market. The middle phases where solutions are developed according to specification are more commonly performed by off shore resources, but the later phases such as integration and deployment is again best performed in-house. In research you talk about tightly and loosely coupled taks. Design is tightly coupled because every change can effect everything else. Coding a module designed properly with an interface definition is loosely coupled since changes to the module does not affect the workings of the whole system. Integration is again tightly coupled because one small change can make everything fail. Tightly coupled tasks also need more and quicker iterations which is intractable at large distances. For my own company I do all web and front end design with local resources although it is much more expensive.

Tools. Having a particular tool is not important (difficult as it is to admit since my company supply such a collaborative tool), but having the same tool across everyone who work together is very important. One common source of problem is the two versions of the truth problem. If everybody is not on the same page the product will not work and you will spend endless time clearing up what is the correct version. So by all means have only one system of record for one type of information. Deciding which tool to use you should consider several things: the functionality of the tool, the familiarity with the tool by the team members and yourself. Even if you have the greatest tool in the world an inferior tool that does the job is better. If the people you work with are used to work with Trello and you are a die hard Basecamp fan: ditch base camp and learn Trello. For each type of information decide which system to use. It could be lucid chart for mock ups, google docs for spread sheets, Jira for issue tracking etc. For example I went with my off shore developers choice of Redmine although I preferred to work with Jira.

Processes. People are used to working differently and you need a pragmatic approach to how you are going to work that will work for everyone. How do you plan a release or sprint? how will you test? How will you do status reports etc. Again it pays off to go with how your off shore resources are used to working rather than forcing the latest scaled agile framework down on their heads.

This is not an exhaustive list, but I do believe that it covers some of the more important ones you should keep in mind. It is difficult to work with a global team, but not impossible. It can actually be very rewarding. Just don’t expect it to be like working with a local team.
Posted on

Experimentation in product management

 Traditionally new products were developed according to the founder’s idea that was written down, which the engineers built. The last few years this pattern has changed.  Across the internet there has been a shift in mindset to bring the customer into what we are building. There is a growing awareness that we are wrong about what the customer wants most of the time. Therefore it is necessary to experiment to find out what customers want.

We talked to Teresa Torres about the role of experimentation in product management. The greater part of her career has spent in pre product market fit internet start ups, so if someone should know how to experiment to find a product that is successful it’s Teresa. Today she helps companies make better product decisions as a consultant and coach.

According to Torres it is better to start thinking about product development in terms of experiments rather than requirements. In Marty Cagan’s dual-track scrum article, he recommends using a discovery track and a delivery track. First we should experiment in the discovery track to identify what the right thing to do is. In the discovery track there should be a lot of experimentation in order to to inform what to build. Today there is a tendency to build any and every idea.

But real experiments require quite a bit of rigor and experience in designing the experiment.

“This is my primary focus as a coach. Many teams start to experiment but don’t have the experience to do it well. Most of us don’t have strong science or statistics backgrounds. What happens in practice is instead of generating informed hypotheses and designing experiments to test those hypotheses, we are testing anything and every thing  The problem with this approach is that we risk false positives.  We are running tens and sometimes hundreds of experiments, many with way too many variations.  This guarantees that we will see many false positives – changes that look good but really have no impact.  As a result, we are making decisions on bad data. If we want to build better products,  we need to understand how to run good experiments. The business side needs to be more scientific and the data science side needs to be more business oriented”

According to Torres the ready availability of experimentation tools like Optimizely and Visual Website Optimizer opens up the possibility for experimenting, but you need resources and expertise otherwise decisions will be made on faulty data. Part of the problem is the wide spread “Fear of Math”. Most people shy away from concepts like statistical significance and power. But it is necessary for product managers to begin understanding these concepts. Today there are many online resources that will teach you basic understanding of statistical concepts. Another problem is that we need to be better at hypothesis design. If you have not properly designed your hypothesis before you start you are not setting yourself up to get good data. We also need experimenters that can design experiments that can also test the hypotheses they are designed to test.

I asked Torres if there are any simple rules of thumb or best practices for product managers who want to get started.

“Don’t trust results that are not statistically significant. Surprisingly many teams are not even testing for significance. Define your hypotheses and decide upfront what decisions you will make if it passes, fails, or is flat . Otherwise you will just find yourself rationalizing after the fact why your change is still good regardless of what the data tells you.  Run the same experiment multiple times. It helps reduce false positives. There is no absolute truth. The world is always changing, something that didn’t work in the past may work in the future. Always keep a mindset that everything evolves.”

For more tips, see her article on The 14 Most Common Hypothesis Testing Mistakes (And How to Avoid Them)

It is up to you if you take Teresa Torres’ suggestion to start experimenting. In the mean time visit her excellent blog Product Talk and sign up for her newsletter. It is always packed with interesting content about product management.

Posted on

What product development strategy is right for your start up?

There are two primary product development strategies and multiple ways of executing them, but there is only one that is right for your company. Finding that strategy and understanding the dynamics may be the difference between success and failure. It may also revert your perception of whether you are doing well.

Nicholas Nassim Taleb has an interesting concept of antifragility. Antifragility are strategies that benefit from shock or randomness, where fragility is the opposite.

A fragile strategy is the classical stock investment or lending money where most likely you will get an ok return, but there is the odd chance that you could loose all your money.

An antifragile strategy is counting on loosing money with the odd chance that you could gain a lot. Options trading would be an example. Here the unexpected is your friend.

Product development
I was thinking about how that could translate into product development strategies. Developing a product is similar in that you invest in it and expect to get a return.

The fragile strategy would be to continually try to develop the product so it performs a bit better in the market place. This is the typical way, and we know it from techniques such as A/B testing and constant tweaking to increase performance on some central product parameter. But this is also risky, because suddenly you may wake up and find out that your entire product line is obsolete even though you have continually improved it. This is what happens when companies are disrupted.

So, could an antifragile strategy be the solution and how would it look? I think it would be a strategy where you try a lot of things that are most likely going to fail, with the odd chance that you could hit it big time.

One scenario is launching a lot of products just to try and see. This is what google and Richard Branson do. There is no end to what they launch. A cheaper way to do this is to probe for demand through market research, prototypes or some other way to get input from the customer about the perceived value. The Lean start up movement is the champion of building cheap “fake” products that are good enough to verify a potential demand.

Another scenario is to launch a product in a market that you feel there could be a potential, but where no product has really had any success. Continuing to develop this product is certain to loose money, but something might happen. You may hit the right combination or market conditions could change. You just need to make sure not to make the same mistake twice

Choose the right strategy for you
Choosing one or the other strategy is not not self evident. Whereas the financial system seems to have most people following a fragile strategy it seems that the start up “industry” is following an antifragile, which is great for the very few people who have time, resources and luck enough to stumble upon a product that hits it big time. It seems to me that your choice of strategy should depend on what sort of life you want to live. Is it ok, although very unlikely, to one day not have a product? Can you live with continuing protracted loss? Do you just want a predictable return? or do you want to have the remote possibility to become a billionaire.

Posted on

From User Interface Design to Virtual Product Design

User interface design is very important for any company who has virtual products, but when more and more people access virtual products through more and more different user interfaces it becomes increasingly important to not design the entire virtual product. Not just the user interfaces.

Not just the user interface

I have been researching how different psychological theories can inform and improve virtual product development. One important idea is the idea of object permanence. Object permanence is a principle from Jean Piaget’s psychology. It describes how children learn that objects continue to exist when they are out of sight (to put it very simply)

The reason this is interesting is that it points to something quite fundamental (if you ask me). Object permanence is a quality of the user interface, but also between user interfaces. When we leave something in a system, like a document in a filing cabinet, we expect it to be the same place when we return to it. We also expect object permanence when it comes to virtual products. A brilliant example of a company built entirely around object permanence is dropbox. When you leave a file in your dropbox it is there in exactly the condition you left it when you return regardless of the device you use to access it.

An example of the difference between when it works and it doesn’t is Netflix and HBO Nordic, which is HBOs experimental online movie streaming competitor to Netflix. When you see a movie on Netflix on your iPad and pause it only later to resume it on your playstation you will find it paused in the same place at the front of your screen. HBO nordic also allows you to watch movies and episodes. But every time you return you have to relocate the content you wanted. You have to find the series you were watching, click to the season, find the episode and then try to find where you left off manually.

Designing in a virtual world
What we need to remember is that, even though the virtual world does not have the same constraints as the physical world, we bring with us the expectations of these very same constraints. This is why we need to stage the virtual products in accordance with our expectations and therefore implement some of the same features that we would find in the physical world. The more the user interfaces live up to our expectations the more natural they feel.

So, what we can learn from this is that we have to broaden our view towards designing a virtual product that has multiple user interfaces instead of just one user interface in multiple versions. We have to think of the product as the same, but accessed from different windows into the virtual world.