Being new in product management

Every day new product managers are recruited, but not so much is written about what it is like being new in this profession. To make up for that, we talked to Christine Luc who is building digital products at a medium-sized skincare company. Before this she was in a few early stage start ups, but still relatively new to the job as product manager.

Who wants to be a product manager?
People grow up dreaming about becoming doctors or lawyers, but why would any one want to become a product manager? We asked Christine why she decided to become a product manager:
“My background is in marketing. When you are in marketing you work with what you have today. You can’t change the product or anything about it. When you are in a marketing role you listen to complaints and it is hard not being able to help the customer more. You can’t change the product and go to the root cause. This was my motivation to move to product management”

But moving into this new position is not without challenges. According to Luc the most difficult thing in the role as product manager is to keep everyone in sync with what is happening. “my earlier stance was that there should be complete transparency with all stakeholders, but the problem is that there is no time dimension to that wish”. It takes some time to communicate and things may change. It could be that the PM knows that a problem is going to be resolved soon, so then it may be better not to share the problem with everyone, since it could provoke confusion and panic. On the other hand it is important to keep everyone on the same page and not let rumors appear in the organization. It is important that there are no inconsistencies in what different parts of the organization thinks or knows. That problem is bigger in larger enterprises than small start ups where everyone knows each other. It is of central importance to get the right information to the right people at the right time.

Being a product manager, we asked Christine what it was about the job that kept her coming back to work every morning. The challenge to see the gradual progress of the product and being part of something that has an impact on the company and the morale of coworkers is central: “internal stakeholders are encouraged when they know that they are part of something wonderful”

Who should go into product management?
According to Luc product management may not be for anyone though. If you don’t like talking to customers, like the status quo and tend to be good at pointing out problems, you may be more happy in another product role like, say, tech lead or test engineer. If on the other hand you are an optimist, a change enabler and you love talking to many different people, product management may be just the thing for you.

Review of Bitium

Bitium is an app for provisioning and deprovisioning acces to cloud applications for employees in company. You can add the applications that your company uses and log into them.

Add employees to your organization and assign them apps that you have in your company. They will be invited and you can decide whether they themselves should choose password or bitium should do it for them. If bitium does it for them it is easy to deprovision access once they leave the company, since they don’t know the password that bitium uses to log them in. Bitium solves two central challenges in using SaaS for enterprises

  1. overview and a central portal or starting point for the employees work
  2. authorization, controlling who has access to make sure new employees are up and running quickly and more importantly former employees loose the access just as quickly. You can use most authorization methods like saml, LDAP, oAuth, 2FA and so forth.

On top of this it even has some instant messaging features.

How can you use it?
You can use it in onboarding making sure that all systems are ready when a new employee starts.

Identity and access management is a central process that Bitium can be used for, but in general it is meant to be the portal towards the web of SaaS applications a company uses. So use it as a start page for your company.

Strong points
Easy to use connection to the different apps that exist. No nonsense you click the add and supply log on credentials and you’re up and running

A directory of more than 1000 SaaS apps is daunting and guaranties you will be able to move all or at least the vast majority of access to apps to one place

Clear and clean user interface makes it easy to get started. There is absolutely nothing more or less than there should be for a cloud portal.

Weak points
Some app connections do not work well. Maybe the connection fails or you can’t initialize the app for some reason, but it is probably not only Bitiums fault and they do write that it is in beta. Never the less as a user you expect and hope everything works.

Access management is out of your physical control. Since Bitium is a SaaS company you have effectively trusted it with all your company’s passwords and user data. That is however a tradeoff. If you want to control it you can set up different methods of authorization against your own user base, but the information about which users use which apps will still be stored with Bitium

Suggestions for the future

  • It is an opportunity to recommend other apps that similar users use. We hope that they won’t develop into a marketplace where the user interface is cluttered with adds though.
  • Develop an integration framework, so you could also manage integration between applications.
  • It could also be an idea to integrate the application itself with HR systems, since authorization is a key element of onboarding and off-boarding employees.
  • Develop a strong enterprise mobility offering. Bitium could provide companies with their own app store and administer employees’ access to apps centrally.

There is a free plan that gives you basic functionality. Then there are plans for $199, $599 and $999 per month, which offers more functionality, such as IP whitelisting, SAML and LDAP authorization, which makes it attractive for larger corporations with that infrastructure in place.

Notable facts

  • Integration to more than 1000 apps
  • Created as a spinoff from a previous online game company
  • The vision of Bitium is to become the SaaS operating system of the future

Anyone looking for a single method to access cloud applications could find good use of Bitium. Probably small and medium sized companies who have a lot of SaaS products will be the early adopters, but there really isn’t any reason why larger enterprises shouldn’t use Bitium as well.

A/B testing for product managers

Neil McCarthy is Director of Product Management at Yammer where he has worked for the past three and a half years. Coming from an education in electrical engineering he has worked for the past 10 years in enterprise software in roles bordering between the business and the technical side.

At Yammer they decided early on to become a data informed company and invested heavily in an infrastructure to support this along with a team of data scientists. Today, no new feature is released without an A/B test.

Why A/B test your product?
I asked Neil what A/B testing can do that other methods for getting customer feedback, such as focus groups and surveys, can’t do.

“A/B testing helps product teams move faster by helping them build the right things and validate their assumptions along the way. A/B testing is a great way to test an idea you already have, but it’s not a great a way to come up with new ideas. Gathering user feedback and thinking strategically about the future of the product and industry is a better way to come up with good ideas.”

At Yammer they also do qualitative and quantitative research post project to figure out what people are actually doing. This plays a big part in figuring out what happened when a test fails.

One example of such a test that turned out to be worse than baseline was when they decided to try to alter the sign up flow. Conventional wisdom has it that the more friction you take out of the sign up flow the better the retention of the customer. So, Yammer hypothesized that by taking out a few steps of the sign up flow and putting them into the product, they could increase long term retention. But to their surprise it turned out that taking out these steps had the opposite effect. The sign up flow was helping users understand what Yammer is. Therefore they did not keep the change and instead left the sign up flow as is. Another example of something that was a success was when they tested whether including a module in the feed that suggested the user to follow other users that their friends followed. It turned out that a lot of users started to follow others and this resulted in a lift in the core metric of days engaged.

How to test
Yammer is not Twitter or Facebook who can do significant tests with only 1% of their users. Instead, Yammer usually tests on 50% of their users. Still it take minimum 2 weeks to do a test. The problem is that since you are testing hypotheses, some of which are proven incorrect, it feels like the advancement of the product is slower. In actuality, you’re moving faster because you eliminate a lot of waste and complexity by not implementing features that are unsuccessful.

“The core of A/B testing is to have a hypothesis. At Yammer hypotheses are rigorously formulated into if/then statements. For example “if we increase the priority of groups, then more users will get work done in Yammer”. This will be broken down into smaller hypothesis that can more easily tested, like: “If we increase the prominence of the group join button then more users will join groups and engage more frequently with Yammer”.

How to avoid local maximum
A well known problem with A/B testing and any other incremental test method is the problem of the local maximum. This happens when a product reaches the point where small changes no longer significantly improve it. At Yammer they have avoided local maximum problems by periodically taking big bets, where they work on really big features. Even for bigger features, they’ll break down the project into small pieces so they can execute incrementally.

Getting started with A/B tests
I also asked Neil what he thought the current best practice for A/B testing was. Here is a list of four key ingredients in successful A/B testing for product managers.
1) Having the right hypotheses is necessary. If you don’t have well informed hypotheses, A/B testing will not help you no matter what degree of technical perfection you have.
2) Log everything users do. This is not to help the A/B test in itself, but in order to understand post hoc, what happened. Why did the test go wrong? Why did the users not react as expected?
3) Have a solid A/B testing framework in place. Without the technical framework to do it you won’t succeed.
4) Put statistical rigor into guidelines for conducting the A/B tests. You need to make sure you are considering statistical significance when looking at the results so you only conclude on true positives.