Exposing technical debt in agile software development

15 June 2021

Application modernization

Technical debt is a familiar concept in the world of software development. In software development, technical debt refers to the extra effort you have to make in the future because you – whether consciously or unconsciously – do not (or cannot) always favor the best or ideal solution over the most acceptable one. As mentioned in a previous blog post, the reasons for technical debt can vary. This form of debt is not inherently bad or even insurmountable, providing it is not permitted to reach critical levels.

Where things go wrong is the tendency to associate technical debt with 'outdated' legacy systems rather than 'modern' agile software development. It is precisely the iterative, incremental nature of agile working and the associated desire to accomplish predefined sprints that can increase our tendency to take shortcuts or detours that may burden our software projects with technical debt. It is therefore important to keep a lookout for technical debt building up during agile projects.

Essential and accidental complexity as benchmarks for technical debt

Anyone who works in software development has a certain intuitive understanding of technical debt. Concepts such as essential and accidental complexity can help turn that notion into something more tangible and serve as benchmarks, facilitating awareness and remediation.

The idea of essential and accidental complexity originates with Fred Brooks^1,who argues that software solutions will always have a certain essential complexity due to the inherent complexity of the problem they are intended to solve. By contrast, accidental complexity is added, avoidable complexity. It is the result of human action – whether deliberate or not – and that brings us to its relation to technical debt.

Essential complexity is intrinsically related to the creation of value, meaning it is unavoidable. If we affect it or try (too hard) to constrain it, the result is often simplification and a lower value, which we definitely don't want. What we need to do is minimize accidental complexity, as it does not add any value (quite the opposite in fact).

Don't play hide and seek

This all sounds so simple and obvious, but it isn't always the case in practice. Software development is frequently complex. It is important to expose accidental complexity and its associated technical debt. As is the case for essential complexity, it is important to identify it and make it tangible to all stakeholders, including the business.

By systematically including accidental complexity in a product backlog, you can ensure that much-needed transparency. The backlog is a living thing and is ideal for this purpose. The product owner can then work with the development team and the business to decide which elements of the technical debt to address and when, as well as those to put on the backburner for the time being. These must be deliberate decisions, taken with the understanding that there is technical debt and that if it is not eliminated it will continue into the future.

Prioritizing according to the Pareto principle (the 80/20 rule), based on value creation, is a good start. The ability to remove all technical debt entirely is an illusion. There are 'economic' concerns, for one thing. By the way, deciding not to eliminate a specific debt is a perfectly valid choice, as long as it is a conscious one.

Avoid confusing Babylonian speech

Although technical debt is termed 'technical', the technology is actually just one of the reasons technical debt may occur. Those reasons can arise at any point in the process. In fact, the potential for technical debt is often created in the stages preceding the actual software development.

For example, an important strategic driver could be the lack of a clear business strategy, meaning the IT department does not know what is required or how to take this into account. It could be that IT and strategy are not sufficiently aligned, leading to a focus on the wrong IT initiatives.

In other words, technical debt is an end-to-end responsibility that concerns both IT and the business. There's the rub. Traditionally, and understandably, IT concentrates on the solution – the software – while the business focuses more on the problem, or what needs to be solved. However, it is crucial that both parties investigate the problem thoroughly together and sufficiently comprehend its essential complexity before IT can move on to finding a sustainable solution. This does not necessarily imply extensive research beforehand, but it does require collaboration with a focus on that complexity. It is also perfectly possible in an agile context.

However, both business and IT will need to step out of their comfort zones, and true alignment will only be possible if both parties make the effort to speak a common, mutually intelligible language and treat each other as democratic equals.

The good news is... solutions exist

Select the right enablers to formalize the alignment of business and IT in advance. For example, you could opt for domain-driven design (DDD), with the cloud (microservices) as an architectural building block. DDD is a development approach that uses a common language, the so-called ubiquitous language, to bring business and IT closer together by starting with the core (the relevant domain). Communication and cooperation are key; the goal is to eliminate friction between stakeholders and increase agility.

The initial, primary focus is on the problem, only later shifting to the technical solution, which is mainly treated as a facilitator. By making the essential complexity comprehensible to everyone, you can improve your organization's business agility considerably. Additionally, accidental complexity becomes easier to detect and avoid if possible. As DDD makes the business a stakeholder in the development of the software, value creation is faster and the likelihood of errors is reduced. Furthermore, actively considering accidental complexity minimizes the risk of its occurrence. And that's mission accomplished.

^{1 Frederick P. Brooks, 1987, “No silver Bullet - Essence and accident in software engineering”}

Don't take your modernization journey on your own

We are ready to help you transform and optimize your applications and IT workloads and make them future-proof.

More info

This blog post is the fourth of a five-part series. Previously published:

Discover all our blogs