Credits:Image generated with Copilot

How to simplify IT using the system complexity ratio

Big, old, large companies have large, inefficient, and complex IT systems associated with erratic operations, difficulty maintaining them, and difficulty adapting them to new needs. Consequently, reducing complexity across the entire system portfolio has been a common focus in most large and complex organizations. Conversely, trendy young tech companies are touted as the ideals of system simplicity and efficiency. Indeed, Netflix can sport an eyewateringly simple architecture, leaving more mature companies and public organizations behind with a perception that their legacy systems are hopelessly complex.  

The issue is that sometimes systems are complex because they handle complex problem spaces. By comparison, Netflix and most SaaS companies build systems to handle kindergarten complexity compared to most banks or tax authorities, for example. Netflix just needs to be able to deliver a file to you on demand and subsequently invoice you. In contrast, tax authorities need to interpret tax laws and rulings continuously (and they are rarely made to minimize system complexity). Banks have multiple complex regulatory requirements to guide their system development. 

Conceptualizing system complexity

That does not mean legacy systems cannot be made simpler. What we want to know is whether a system is too complex and has the potential to be simplified and, in particular, the degree to which this potential exists. 

To understand that, we first need to distinguish two types of complexity: intrinsic complexity and extrinsic complexity. Intrinsic complexity refers to the complexity of the problem space the system is intended to manage. In contrast, extrinsic complexity refers to the complexity introduced in handling the problem area, that is, the total system complexity. While there are different meanings of complexity in other disciplines, system complexity is usually considered to be driven by three factors

  1. Number of elements – all else being equal, a system of three components is simpler than one with 3.000
  2. Number of relations between elements – the relations are defined by interactions between the elements of the system. The more different elements interact with each other, the more complex
  3. Number of states of elements – the states of the elements drive complexity because the different states usually have consequences for the interactions between elements. This is to be distinguished from other accidental information about the element.

Intrinsic complexity is, thus, the inherent complexity of the problem space. If we look at a way to gauge this, we can look at it from an information angle. (1) The number of elements would be the number of entities in a logical model and the number of attributes that describe the problem space. (2) The number of relations are the relationships between entities in this logical model, and (3) the number of states is defined by the number of different states the attributes of the entities can take on. Note here that we are not talking about information about the entity but only predefined states. For example, if we have a person entity, the name is not a state, but the gender is. Only the state is likely to drive complexity since rules are likely to be built on it, not the name. This measure of complexity is computable not only in principle but also in practice. 

We can compare this with the extrinsic complexity, which is the actual complexity of the resulting implementation of the system handling the problem space. The same approach we took to the logical model can be applied to the system implementation. Here, we would calculate the complexity of the physical data model of the system implementation designed to handle the problem area. Again, we would count the number of entities, attributes, relations, and states. 

The system complexity ratio

These two concepts, intrinsic and extrinsic complexity, provide an interesting tool with which to measure and conceptualize complexity and simplification potential. The critical ratio, which we can call the system complexity ratio, is extrinsic to intrinsic complexity, that is, how complex the system is compared to the complexity of the problem it is made to handle. If it is one, the system perfectly contains the problem space, and no further simplification potential exists. If it is below zero, the resulting system does not capture the problem space adequately, i.e., it is too simplistic. If the system complexity ratio is above one, the system is too complex and could meaningfully be simplified. 

The system complexity ratio would be expected to be above one because it is necessary to create artifacts other than those in the problem space to render the system usable, such as managing users, their roles, and different technical artifacts. It should, however, not be much greater than one. If the system complexity ratio approaches five, something is terribly wrong. 

The boundary of simplicity 

Calls for simplification are justified, but there is a boundary of complexity that cannot be passed unless the system’s ability to deal with the problem space is compromised. If the system complexity ratio is below one and we make the system simpler than the problem space it is supposed to handle, it will necessarily be deficient in one way or another.  

Einstein is often attributed the aphorism:  “A theory should be as simple as possible but no simpler.” This is a concise statement of the general gist of this thinking. But there seems to be no evidence that he actually did say that. Instead, he said the following: 

“It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.” (‘On the Method of Theoretical Physics’, lecture delivered at Oxford, 10 June 1933)

While not as quotable, it is more precise and even better as a way to think about complexity. Paraphrasing it, we could say that the supreme goal of a solution is to reduce the extrinsic complexity of the system as much as possible but not beyond the intrinsic complexity of the problem space or, in other words, to reduce the system complexity ratio towards one but never beneath one.