The Infinite Loop Part I: The Problem

Trying to understand why the software industry is so inefficient and ineffective #

In this series of posts, you will embark on a journey through the history of software development methodologies (SDMs) and other principles, from Waterfall to Product-led growth, trying to understand the main forces that drive failed agile implementations.

You will then be introduced to The Infinite Loop (L∞P), a new but not revolutionary (on purpose) SDM inspired by the principles that drive the most successful open-source development communities.

We will learn how L∞P leverages lessons from open-source development to create a data-driven culture of trust, ownership and continuous experimentation that fosters sustainable growth and high-performance digital product teams.

Note: This is going to be a long post! Please note that if you don’t have time (or don’t fancy) to read this much, the contents of this post are also available as a more concise slide deck.

Update: PART II is now available.

1. The software industry is inefficient and ineffective. #

“Efficiency is doing things right; effectiveness is doing the right things” - Peter Drucker.

The Standish Group’s CHAOS Report is probably the most extensive and longest-running research study in the software industry. The CHAOS Report examined thousands of software projects over three decades and found some very disappointing performance metrics:

The Standish Group's CHAOS Report

The problems go beyond project failures. Other reports put the spotlight on other equally depressing metrics:

As part of my job, I interact with many developers and technology businesses daily, and sadly, I believe the CHAOS report is accurate. What I have witnessed across our industry is mostly a widespread estate of despair. The following list contains some of the most common problems:

As a developer: #

As a business:  #

2. How did we get here? #

“Risk comes from not knowing what you are doing” - Warren Buffett.

We have been building software products for a few decades now. I would expect our industry to have an excellent understanding of what is required to achieve high levels of efficiency and effectiveness by now. However, we are not there yet.

We have managed to identify and document the main risks involved in developing software products. We have developed Software Development Methodologies (SDMs) and other principles that allow us to mitigate some of these risks. So, how is it possible that after decades of progress, we have failed to improve the rate of success of software projects significantly? To attend to answer this question, we first need to take a little trip through the history of SDMs and other principles.

2.1 Waterfall (1956-1995?) #

The Waterfall is an SDM that uses a linear approach in which each phase must be completed before the next one can begin. The name “waterfall” comes from the idea of representing each stage as a “water dam”. As we progress, the dam is filling. Only when the stage is complete can the water overflow and starts to fall into a “lower dam”, representing the progress in the next stage.


Developing software without any plan is almost certainly a recipe for chaos. The waterfall SDM was one of the earliest attempts to prevent projects from falling into the abyss in which chaos resides.

The good thing about Waterfall is undoubtedly better than chaos.
The bad things are that it has a very rigid structure. It does a poor job managing uncertainty and leads to low customer and stakeholder engagement, infrequent & deferred testing, and scope creep.

Like in finance, in software, “Risk comes from not knowing what you are doing”. We create SDMs in the first place to “mitigate risks” or, in other words, “eliminate unknowns”. Since Waterfall does a poor job managing uncertainty, it is no surprise that Waterfall is today recognised as a recipe for failure in software projects.

2.2 The Agile Manifesto (2001) #

After many years of failed waterfall software projects, 17 renowned software developers met at a resort in Snowbird, Utah, to discuss lightweight development methods. Together they published the Manifesto for Agile Software Development.

Based on their combined experience of developing software and helping others do that, the authors of The Agile Manifesto declared that they valued:

That is to say, while both sides have value and the items on the right should be considered, the authors felt that the items on the left should have more influence on how people approach their work.

The agile manifesto also declared 12 principles:

The 12 Principles of Agile

  1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

  2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

  3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference for a shorter timescale.

  4. Business people and developers must work together daily throughout the project.

  5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

  6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

  7. Working software is the primary measure of progress.

  8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

  9. Continuous attention to technical excellence and good design enhances agility.

  10. Simplicity–the art of maximising the amount of work not done–is essential.

  11. The best architectures, requirements, and designs emerge from self-organising teams.

  12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

The Agile Manifesto is not a methodology. It is just a set of principles, but it significantly impacted the industry and popularised the use of Agile methodologies, such as Scrum and Kanban, which are now widely adopted by software development organisations worldwide.

2.3 Scrum (1995) #

Scrum is an Agile SDM that presents organisations with a prescript process. The organisation that governs Scrum did an excellent job documenting the process and facilitating its mainstream adoption through certification programmes.

Scrum quickly became the most adopted Agile SDM. Process changes in larger organisations are usually riskier, but Scrum provided executives with enough resources to mitigate some of their fears. Some of the leading businesses in the software industry adopted Scrum, and their influence quickly pushed smaller players to follow.


Scrum can be summarised as follows:

The best thing about Scrum is that it facilitated the mainstream adoption of agile and managed to “kill” Waterfall. Scrum helped businesses embrace the idea of planning being something that adheres to the “law of diminishing returns”. At first, planning seems to improve things, but there is a point at which investing more in planning fails to provide any meaningful returns.

Some of the worst things about Scrum include the following:

2.4 Lean UX (2008) #

As we have already mentioned, the reality of building software products is that there will always be a high level of unknowns. Planning and estimating can help clear some unknowns but never eliminate them. Scrum did an excellent job by recognising that a little bit of planning and estimating can help to make things better, but attempting to plan and estimate the entire product upfront is ultimately a waste of time.

The problem with Scrum is that it failed to formally recognise that introducing a discovery or experimentation phase into the product team’s workflow can eliminate more unknowns than planning or estimating, which is the main principle behind Lean UX.

Like the Agile Manifesto, Lean UX is more of a list of principles than an SDM. The main principles of Lean UX include the following:

Lean encourages us to build as little as possible and collect as much feedback from users as early as possible. We must then decide our next move based on the collected user data.

Lean UX

The best thing about Lean UX is that it introduces the idea of using discovery, experimentation and data analytics over planning and estimation as the primary way to mitigate risk in software development projects.

The bad thing about Lean UX is that it is not as prescript as Scrum and requires an upfront research investment. The nature of experimentation makes it hard to plan an estimate. These reasons make Lean UX much scarier for management than Scrum, especially for large organisations.

2.5 Kanban (2010) #

Kanban was introduced as an alternative or complement to Scrum, which was already widely used in the industry. While both Kanban and Scrum are Agile SDMs that aim to improve the delivery of software products, they have different approaches and strengths.

Kanban is a pull-based approach that emphasises visualising and managing the flow of work. It does not have time-boxed iterations like Scrum and focuses on continuously improving the delivery process.

The following list includes the main principles of Kanban:

Kanban principles

The best thing about Kanban is that it introduces the idea of reinforcing focus by limiting work in progress and removing time boxes, leading to increased quality.

Like Lean UX, the bad thing about Kanban is that it is also harder to implement than Scrum. Scrum has a more prescriptive framework and a stronger focus on Agile principles, which can make it more appealing for organisations looking to adopt Agile practices. Scrum also has a well-established certification process and training offerings, which can help organisations adopt and implement the methodology more effectively.

Another potential problem with Kanban is that its focus on metrics like cycle and lead time can reinforce the idea of outputs over outcomes. Leading to reduced customer value and decreased quality.

2.7 The three ways (2013) #

The three ways are a set of principles designed to improve the efficiency of software development projects. These principles are highly influenced by The Agile Manifesto, Lean UX and Kanban and are grouped into three main categories.

2.7.1 The First Way: Systems thinking and the principles of flow #

The First Way

Move work from Business, through Development, to Operations, and ultimately to the Customer (where the value is created) as quickly as possible.

2.7.2 The Second Way: Feedback loops and the need for amplification #

The Second Way

Increase the feedback loops from right to left. Focus on increasing the number of feedback loops and their speed. Treat problems as opportunities to learn how to prevent them and create an ever safer and more resilient system instead of a cause for punishment and blame.

2.7.3 The Third Way: Creating a culture of continual experimentation and learning #

The Third Way

Developing and fostering a culture where constant experimentation and learning are encouraged and where people acknowledge that the way to mastery is through repetition and practice:

The best thing about the three ways is that these principles leverage ownership within the development team to eliminate hostilities between technology disciplines such as frontend or backend development, testing, infrastructure, and site reliability engineering. The three ways also encourage implementing a high level of automation to prevent human errors, speed up the development feedback loops, increase anti-fragility and avoid repetitive tasks. The three ways can mitigate some inherent risks associated with developing technology products, particularly those associated with handovers and operations.

The bad thing about the three ways is that these principles are not a methodology. These principles are not prescript enough and are open to interpretation. As I have mentioned several times, not being prescript makes mainstream adoption much more complicated. However, the three ways and DevOps seem to have escaped this “curse”, and today, they are mainstream in businesses of all sizes across the world. How did the three ways gain popularity despite being hard to implement? Two factors can explain the adoption of the three ways:

2.8 Product-led growth (PLG) (2016) #

The three ways principles and the DevOps movement leverage ownership to eliminate hostilities between technology disciplines. PLG aligns the development, marketing, and sales teams by focusing on creating a product that is the driving force behind customer acquisition, retention, and revenue growth, which creates a shared objective and a common goal for all teams to work towards rather than relying on traditional, siloed sales and marketing tactics.

The following list contains some of the main principles of PLG:

Product-led growth

The best thing about PLG is that it aligns the sales, marketing and customer success departments with the development team. When all groups focus on delivering a high-quality user experience and adding value to the product, the marketing and sales teams can use the product as a selling point and reference for customer acquisition and retention. Ultimately leading to a virtuous cycle where the product drives customer acquisition, and the customer acquisition drives product development and improvement. By aligning the development, marketing, and sales teams around a product-led growth strategy, companies can create a more cohesive and efficient growth engine that drives long-term success.

The bad thing about PLG is that, once more is not a methodology and is not very prescript, making it hard to gain widespread adoption. Like Lean UX, PLG is hard to predict and requires a significant upfront research investment.

3. Why are we still failing? #

“Ease of use may not be the most important feature, but it’s the one that’s most important to get right.” - Jef Raskin

Answering this question is a big challenge. Our collective failures cannot be attributed to a single cause. The following list contains the main reasons I believe are preventing our industry from achieving a greater rate of success:

In PART II, I will share how I believe we can solve these problems.


Now read this

My new book has just been released! Here is how I did it and how you can do the same

My new book, Learning TypeScript 2.x, has just been released! I’m not going to talk about the book contents in this post because you can learn everything about it at However, I would like to talk about the... Continue →