Stuck in beta? 

– It is time to stop experimenting and start implementing

Many good experiments have been done in the last few years, especially involving Artificial IntelligenceI but they often become stuck in beta. Experiments with significant potential representing a potential value of millions if not billions are stuck in the innovation pipeline. The reason is that it takes a different effort to bring something to production than producing an experiment. Different cultures, technology standards and risks and regulation contribute to this clogging of the innovation pipeline, but it is only a scalable production-grade solution that creates value. We therefore need practical tools to unclog the pipeline. 

Experiments and the innovation pipeline

Experimentation is great! It allows you to discover new and better products or more efficient ways of doing things. Without experimentation, innovation would be difficult to sustain. Experiments are used to keep the effects away from the population. Chemists conduct small experiments in isolated labs so as not to harm anyone with noxious substances. 

Organizations do much the same thing even if the harmful effects may be different. The target population is most frequently the one impacted and involved in the organization’s value creation, that is, customers and employees. The negative effects of an experiment are shielded from this population, but so are the positive effects. As long as the experiment is in the lab it doesn’t do anything bad but it also doesn’t do anything good. 

Many experiments, even successful ones, unfortunately don’t make it out of the lab. Therefore they never create any real value. There are a couple of reasons for that and they need to be handled for the experiments to make it out of beta to the end of the pipeline. 

Innovation and operations culture 

The people experimenting frequently have different perceptions of what is acceptable risk and what counts as a sufficiently robust solution. The IT organization, tasked with managing all live solutions of the organization, may have somewhat different views of what counts as production-grade, and business units may have different views on acceptable risks. 

Aligning these views is imperative to make progress. Through communication operations and business units  can get a better understanding of how the solution works and experience a better sense of what is necessary in the real world. Simple reviews and briefings especially through the development phase are very helpful to create a common goal. This can align expectations of how feasible the solution is and how far it is from being ready for the real world.

Technology standards 

Organizations have technology standards for a reason, and that reason is not to frustrate developers and people performing creative experiments. It is to reduce overhead and risk. Only a small subset of possible technologies are used because competency needs to be retained to support them. Technology standards are thus in place to ensure efficiency and reliability. These factors are orthogonal to what counts in experiments. Here freedom and speed are the primary factors. 

This obviously makes for different design decisions and technology choices. We want our experiments to show as quickly as possible whether they can create value,  not whether they can align with standards. The results however are invariably that the experiment is far from aligned with the standards in place.

The key here is to resolve the design so as to be consistent with enterprise standards. That means sometimes it has to be rebuilt in part or in full, but it can also mean that standards need to be updated because the experiment found a significantly better way or because current standards don’t allow for the type of solution. Creating the space where this kind of reflection can take place is crucial.  

Risks and regulations 

Differences in how risky experiments are perceived are a third category of reasons experiments get stuck in beta. Doing experiments in the lab shows little about how it will be in real life. That is why the pharmaceutical industry has elaborate procedures to take a new drug through different phases with increasingly comprehensive tests. For technology this is typically not the case. But it illustrates the gap in perception of how risky a solution is. The business is at the end of the pipeline, where the solution meets the market. How the solution performs it is their responsibility. 

Furthermore AI products often have even greater uncertainty because dynamics may occur that cannot be accounted for in the lab. These include public sentiment, model drift, user behavior, ethical considerations etc. These are always important to some extent, but AI technology exacerbates this. Regulation is part of this uncertainty because new AI regulation is in place in many places globally, such as the EUs AI Act, and it is not always clear whether a solution would be compliant. Then it feels better not to take any chances.

It is possible to address this difference by modeling the risks by involving the right competencies. It is necessary to have someone who understands the human dynamics, the regulatory environment, the business and even HR side of the equation. These groups are not routinely involved in technology development but they are needed to provide an adequate picture of the risks. Once these are established  and understood it will be easier to create a mitigation plan to bring the experiment through the pipeline. Maybe additional tests are needed, maybe aspects of the solution must be changed or even something as simple as communication to users and stakeholders is sufficient. 

From experimentation to implementation 

Experimentation is good but also deceptively simple. Even though the experiments can be incredibly complex and the brightest minds are involved, from an organizational perspective it is simple: you just select a problem, and then you get a budget and then you solve it. But even though you can feel and taste the solution it might be much further from prime time than you realize. 

There are many potential blockers in the process that need to be removed. The culture differs but communication can go a long way to resolving those differences. Technology choices are made with varying goals but can be aligned through design reviews and alignment plans. Finally risks may be mitigated through a better understanding involving all aspects of the solution. 

None of these are necessarily technical challenges, so more investment in technology development may not help. It is time to look up from the engine room to the surrounding ship to understand how to change the course.