The origin of recommendation algorithms was driven by the increased need to manage and navigate vast amounts of digital content. As the internet expanded exponentially in the late 1990s and early 2000s, users found themselves overwhelmed by the sheer volume of information available. Search engines like Google had revolutionized information retrieval, but a significant challenge remained: helping users discover relevant new content they didn’t know existed, especially in the numerous webshops that emerged in the wake of the dot-com boom. This challenge catalyzed the development of recommendation algorithms.
Early days of recommender systems
The early days of recommendation algorithms can be traced back to the so-called collaborative filtering techniques in the 1990s. The concept refers to the idea that users collaborate through their behavior to filter content, so content like articles, movies or products that other users like you preferred. These can then be shown to you based on how closely they are to your preferences. There are two types: user based and content based collaborative filtering. In user based filtering user behaviors are compared to find users with similar behavior to you. The items they liked are then presented to you. In content based filtering the focus is on the items, and items that are often go together are identified.
Pioneering efforts in this domain included systems that leveraged user behavior data to suggest items. The GroupLens project, developed at the University of Minnesota, was one of the first to employ collaborative filtering in the context of Usenet (an early worldwide discussion forum on the internet) articles. It was also used to recommend movies at the MovieLens site. Another early precursor was Amazon’s use of user based collaborative filtering, which recommended products based on the purchasing behavior of other users providing recommendations for other products to purchase.
Content-based filtering also played a crucial role in the evolution of recommendation systems. This approach focused on recommending items similar to those a user had shown interest in, based on an item’s likeness to other items in terms of purchases. For instance, early implementations in the domain of music and movies analyzed attributes like genre, artist, or director to provide recommendations.
The development of recommendations for any need
A number of developments helped drive the rapid advancement of recommendation algorithms. The proliferation of user-generated content on platforms like YouTube, Netflix, and Spotify required more sophisticated recommendation systems to be developed to enhance user experience. The first and most famous example was the Netflix prize, which was an open competition to find the best collaborative filtering algorithm. Netflix provided more than a hundred million ratings from half a million users. Based on this data contestants had to improve on Netflix’ existing algorithm.
Advances in hardware and software also helped propel the development of recommendations systems. Moore’s law, provided an ever increasing amount of computation and storage to become available at ever lower price, but open source software also played a significant role in disseminating and improving algorithms. In machine learning the use of new techniques like matrix factorization, neural collaborative filtering, and reinforcement learning significantly improved the accuracy and relevance of recommendations.
These forces led to the Big Data revolution, where distributions of open source software and large volumes of commodity hardware made it possible to improve significantly the scope, accuracy and performance of recommender systems.
Commercially, the availability of recommender systems prompted companies to quickly realize the economic benefits of personalized recommendations, which could drive engagement, increase sales, and improve customer satisfaction This happened to such an extent that it became a customer expectation to have recommendations.
These developments came together to drive the development of a rich varety of recommendation algorithms, techniques and solutions to fit any need.
How recommendations transformed the modern world
The impact of recommendation algorithms on the contemporary digital experience has been profound and multifaceted. Personalized recommendations have become a staple of user experience across various platforms, from social media feeds curated to individual preferences to streaming services that suggest content tailored to personal tastes. After all, can you imagine using an online streaming or audio book service today without any recommendations for what to watch or listen to?
From the business angle it is clear that E-commerce giants like Amazon and entertainment platforms like Netflix who understood how to harnessed the power of recommendation algorithms to boost sales and viewer engagement established a competitive advantage that helped propel them into the top of most valuable companies in the world.
Recommendation algorithms have fundamentally reshaped how information is consumed and disseminated in our contemporary world for better or worse. While they help users find content they might like, there is also a concern about the creation of information filter bubbles and echo chambers, where users are exposed primarily to information that reinforces their existing beliefs, which has been part of an increasing political polarization and the emergence of concerns over fake news.
Making waves
Rarely has AI had such a transformative impact on the world. Few people even realize it today, but recommendation systems are an example of how technology has fundamentally changed our behavior and shaped expectations about how technology should work. It also provides a model for how AI can recede into the background and optimize the interaction between digital systems and humans. Accurate and relevant recommendations save thousands of hours every hour that would have been spent finding information. Much of this occurs privately and is therefore not reflected in productivity statistics, but make no mistake, it has transformed the consumer experience. I can still remember how it would take an hour to find, rent and, most importantly, return videos from Blockbuster. This is time I now have to spend on other things.
As always with technology there is also a cautionary tale because the widespread use of recommendation algorithms plays a role in narrowing the scope of new information. We run the risk of being trapped in echochambers of people like ourselves. That is, however, nothing new to humanity but it should be acknowledged.
This is part two of a series about the Six Waves of AI in the 21st Century
You can read part 1 about Search here
English 
Danish