Can Prediction Markets be Used to Prevent IT Projects From Failing?

Can the wisdom of crowds help improve the hit rate of IT projects? It is well known that many if not most large IT projects fail. It is also well known that they could have been saved if proper intervention had been undertaken at an earlier point. The question is just how to get an early warning? How will management or the project manager know when things are going wrong?

One way could be prediction markets. Prediction markets are based on the basic assumption that crowds are wiser than any individual even an expert. The obvious question then is, why not just send out a questionnaire on a regular basis to ask how people are feeling about the situation. The problem with questionnaires is that they are more easily distorted.

One example is the past few elections for President of the USA, where the outcome has been more precisely predicted by the bookmakers than the polls. The reason for this is that an opinion in a poll is free, but when you bet you have a stake in it. It is not free.
Prediction markets are modelled on this dynamic. Participants in the market are given a virtual currency to invest in the likelihood of future events. This currency then will eventually be turned into some sort of reward in real life.

The British company qmarkets has developed software that allows companies to use prediciton markets to predict the future*. Companies have used it to predict which new product to invest in or which drug to bring through to clinical trials, or which risks are most likely to occur. If applied properly, prediction markets are the most accurate way to predict uncertain events.

The project portfolio managers challenge
One of the most challenging parts of project portfolio management is to manage the uncertainty in the portfolio. It is inherently uncertain whether a project will be finished in time for the deadline, whether resources are available or whether the quality will be good enough.

Standard textbooks prescribe the use of complicated algorithms and reference class data to determine whether the project is under spending or overspending with regard to the burn-down rate of a project. A project that burns less hours than expected may experience resource bottlenecks or lack of commitment. A project that burns more hours than expected may have underestimated the amount of work. Either way, the likelihood that the project will be on time and on budget is diminished.

The challenge is to be able to say exactly when a project is going off track. This can be done if we know when exactly a project is overspending and when it is underspending. The problem, however, is that all projects are different. Some have a much slower start and explosive finish, others start really energetically and then level off.

I Predict
When traditional ways of detecting when a project is going off track don’t work, prediction markets may offer precisely the solution needed. Prediction markets are good at early detection of sentiments and information that is distributed among a group of people. It could therefore serve as an early warning system.

The prediction market could be implemented as a market where the question of whether any project in the portfolio, or at least the largest of them, would meet their deadline. All employees would be given a pool of virtual money to buy stocks in the project. These stocks can be bought and sold at any time. The stock price therefore reflects the probability of a project meeting its deadline.

The portfolio manager and senior management could therefore, just by monitoring the market, much earlier spot projects that had a high risk of failure. That would allow them to do one of two things: discontinue the project or give it more attention to bring it on track. Either of these options would benefit the company. It would either minimize losses or maximize likelihood of delivering quality on time.

These are the hard benefits, but there are also, soft benefits that would not follow from the traditional method. Since this is a game, it is a way for employees to take the mind of their work for a second and still be doing something relevant, instead of posting links to youtube movies on facebook.

It is also a way to create awareness of all the projects your company is running. Suddenly employees will know what other departments are doing and they may even serendipitously discover synergies or redundant projects

Since it is a game it can be entertaining, which may boost morale.

Further it may create positive incentives to make the real projects finish on time. If you invested in it, you want to protect your investment (this is why it is probably a good idea to disable the possibility for shorting project socks).

Future potential
Unfortunately this solution is not in use anywhere yet. It is only a possibility, but with the market maturing and ideas about collective intelligence becoming more widespread, it is probably only a question of time. the current economic climate calls for solutions that would help you minimize losses and maximize success. The problem is just whether the cultural gap of actively letting employees play and even rewarding them for it is too big for most companies.

The typical risk analysis is inherently flawed. It is only based on the project managers subjective opinion and furthermore it is typically made with a 5 point scale idiosyncratically interpreted. This doesn’t guarantee a good estimate of the future
Furthermore we are natural born optimists. We overestimate the likelihood of something good happening and underestimate the probability of negative events occurring.

* Best Buy used decision markets to find out whether a new product idea would succeed ( They built a market where it was possible for about 2000 employees to trade imaginary stocks that related to questions about the future. So, if a question was very likely to be true the price of the stock would go up and if it was likely to be false, the price would go down. This means that the price of the imaginary stock would match the probability of the answer being true.

Justin Wolfers and Eric Zitzewitz: “Prediction Markets
James Surowiecki: “The Wisdom of Crowds – Why the Many are Smarter than the Few
Eduardo Miranda: “Running the Successful Hi-Tech Project Office
Photo: thetaxhaven @flickr

“Heat” – How to Train Your Decision Making

In one of the greatest gangster movies of all times,”Heat”, lies hidden deep wisdom for all decision makers big and small. It is not that you should change your business model and start robbing banks, hire Al Pacino or even that you have to grow a moustache and goatee to succeed. But our decision making is clouded by two problems that can be attributed to how our brains work. They can be overcome by heeding the advice of “Heat”.

Cool feature…
Before deciding to do something most people will investigate the possibilities. This is the first place where your decision making could go wrong. People have a tendency to attribute too much importance to the first information they get. This is sometimes called the anchoring bias. The effect is that all subsequent evaluations will be affected by that first thing you see even though that information is irrelevant.

It could be that you are looking to invest in a new Customer Relationship Management system for your company and you hear about a vendor that has close integration with facebook. Again this is a cool feature, but essentially irrelevant. All subsequent CRM systems will be reviewed in the light of whether they have facebook integration.

Jeans suck!
After having seen a couple of alternative solutions to your problem you will start to develop hypotheses based on your gut feeling, for example that products from Germany are inherently more robust. You may have seen two or three examples of this. Here comes the next hurdle: the confirmation bias. The confirmation bias makes you look primarily for confirmatory evidence and attach more importance to it than to contrary evidence.

It could be that in your recruitment process for a new ambitious account manager you encountered two polite applicants from, say, London. This has made you develop the hypothesis that people from London are polite. If an impolite applicant from London comes along, who may forget to shake your hand, you may not attribute his geographical origin any value in your evaluation. Chances are that you will use another hypothesis, for example that people in Jeans are impolite and this is why he is not as polite (at this point you have forgotten that the first applicant from London also wore jeans, but at that time you didn’t pay attention to it). You just made that up on the spot to make sense of the evidence.

Psychologically it just feels a lot nicer to have your hypotheses confirmed, than having them contradicted, even to the point where you make up new ones just to have some confirmation.

30 seconds flat
Flash back to “Heat”. At one of the absolute highlights of the movie the policeman, Vincent Hanna, who hunts the gangster Neil McCauley decides to have coffee and talk with Neil. At the climax of this conversation Neil McCauley says the sentence that has also given the movie its name: “Don’t let yourself get attached to anything you are not willing to walk out on in 30 seconds flat if you spot the heat around the corner.”

This is deep wisdom, because, if applied with sufficient discipline it will eliminate the above named decision problems. You should never get attached to any idea. Period. It doesn’t mean that you should change your opinion or hypotheses incessantly. Rather it means that you should always be psychologically prepared to abandon an idea if you find that evidence does not support it.

This is more difficult than you think. But so is pole vaulting. Yet, pole vaulting can be learned. This is done by training. And train, you can. Teach you, I will: Next time you are making a decision try the following.

  1. Before you start searching for information write down your hypotheses. That way it will be easier to challenge them
  2. When you research your alternatives write down all the new hypothesis you develop
  3. Make sure to continually test your list of hypotheses in your evaluation process
  4. All previously evaluated alternatives should also be checked against the new hypotheses
  5. Every time you find sufficient evidence against an hypothesis. Strike it from the list.

You can train this on everyday decisions like choosing the right restaurant for you wedding anniversary (yes, it’s not enough to bring home Chinese take away), buying a new TV (we all know you need it) or even finding a new boyfriend/girlfriend (I know, not necessarily an everyday decision, but still..)

Photo by Michael Gil, MSVP @flickr

The Hexagon Framework for Selecting the Right IT system

Companies are wasting millions of dollars every year on failed Commercial Off The Shelf (COTS) system acquisitions. The symptoms are well known: budget overrun, missed dead lines, bad quality and bad client supplier relations. Even when the acquisition project in itself is a success there is no guarantee that the system will ever deliver the expected benefits, because the expectations of functionality were never properly aligned with the business goals.

In order to avoid this we have developed an evaluation framework called the Sensor Six Hexagon Framework. It will guide your COTS evaluation through the most important criteria and increase the likelihood of success for your COTS acquisition. The Sensor Six Hexagon Framework operates with six groups of criteria:

Functionality – This aspect considers how the software is to be used. For example which business processes it is meant to support, the usability and specific functions. The purpose of evaluation here is to ascertain the degree to which the functionality is useful. What does the system do?
Implementation – The purpose of this group is to evaluate implementation process. How long will it take, how big is the suppliers experience and cultural compatibility. Here all criteria related to the project of making the COTS functional are considered. How will we implement it
Risk – As the name suggests the risk group has to do with all aspects of what could go wrong. Is the product immature or is security low? Is the supplier just about to go bankrupt? This type of questions should be asked here.
Strategy – Has to do with a long term orientation for the supplier as well as the customer. The customer might want to consider whether this supplier fits with his strategy, but also whether he thinks that he as a customer belongs to a key segment of the supplier.
Integration – is about how this system will work with all other systems in the enterprise. Is there a well known way to integrate it? Does it have a well defined interface?
Operation – Once it has become implemented how is the systems operational aspects: Support, performance, upgrades etc.

The framework is based on empirical research from 10 client cases of COTS acquisition. They were grouped into successes and failures. Based on what was done in the evaluation process we were able to conclude the following
1) All succesful projects had evaluated criteria from all six groups of the Sensor Six Hexagon framework
2) All failures could be predicted from the failure to evaluate properly before acquiring the COTS system
3) The more criteria and the more thoroughly they were evaluated the higher the likelihood of success
4) No single criterion stood out as the most common cause of failure. Every failure was due to a missing evaluation of a unique combination of criteria

The Sensor Six Hexagon is available as a template in Decision Orchestrator and we also use it in consulting as basis of our Accelerated COTS Acquisition Process (ACAP), but anyone can benefit from it. Contact us for a whitepaper describing how. The benefits of using the Hexagon Framework is larger success rate, quicker decision process, and alignment to your companies strategic and operational goals.