I’ve been aware of the debt metaphor for a few years now. When I heard of it at first it instantly clicked with me. I grew up on a farm and given my parents were self employed I thought about debt a lot. It inform my view on finances and now programming. I’ve been reading a lot about it over the last while to get a better understanding of the nuances of the metaphor. In this post I want to explain what it is and what I’ve learnt.
What is Technical Debt
Say you are tasked with developing a feature. It’s pretty big and could, by your estimates, take anywhere from 5 to 6 weeks. At this point you make a calculated decision (more on this later) to take a short-cut and get the feature out in 2 weeks. Those 3 or so weeks would be considered technical debt.
To put it in simple terms. Technical debt is any decision (inadvertent or deliberate) that results in a lower quality code base, or more importantly in making it harder to maintain the code base and build future features. When you take a short cut everyone who has to interact with that short cut takes a hit on productivity in either understanding or usability, that is the interest payment if you will.
For a more thorough explanation it would probably be worth hearing what the man who conceived the metaphor thinks, the following video is pretty short at 4:44 so have a watch/listen.
The main take away from the video for me is the need to manage and understand debt. The trouble I’ve seen in my career to date is with the inadvertent type of debt that projects usually end up with.
Martin Fowlers: Debt Quadrant
When Ward conceived the debt metaphor it was a way of explaining to his managers and to the business people who he worked with why one would conduct a refactor to a code base. A code base/module that for all intents and purposes worked, was fine and probably even now in use in the field. To Ward it was about putting back the understanding/learning you gained from the process of building the product.
It’s clear from that explanation that Ward is a conscientious and professional programmer. Given that it would seem to me that he is probably going to have design and risk management at the forefront of his mind. For a lot of developers out there it might feel like that’s not the case where they work. Martin explores the metaphor a bit more thoroughly for rest of us.
The main element of Martins musings is the concept of the Debt Quadrant.
I think this keenly describes what is felt in most projects. Martins career and writings are based mainly on the principals that design is good and helps you to move faster. And that is what he explores here.
What is interesting about Martins quadrant is where certain concepts fall. Does the idea of “move fast break things” lay in deliberate & reckless or deliberate & prudent? What about if you solved a design problem with a lot of inheritance where a visitor pattern would of been more appropriate? Were you being reckless by not being fully versed in all the design tools you needed? Is there a base level of design you should of had before being given the helm of that project?
The main distinction is probably what you do after you ship. In my view a “prudent” decision becomes a “reckless” one when you don’t face or even admit consequences. Lets say you make that decision to knock three weeks off the schedule. Where have you tracked that decision? Have you informed your team mates or colleagues about that decisions? If you have a bug/task database has you logged that decision to be addressed at a later date?
I must warn you, I’m a bit of a purest
Thankfully in my life I have not had to have the burden of a lot of financial debt. I live in Ireland; we have “free” education which come with some costs though nothing like North America and elsewhere. As such I’m a touch adverse to the idea of debt. I save up to buy things like cars, televisions…etc where many people use loans and financing options. I have values by which I live my life. I’m a bit different to most people.
Given this I am trying to understand the need and point of debt. Debt is a tool and like any tool when used correctly it can allow you to do things you could never achieve without it. But like a tool it can very easily turn into a deadly weapon. One of the main causes of business failures is cash flow, guess what can impact cash flow? That’s right loan repayments.
Steve McConnell is a very smart guy and once said:
"I think there is some non-zero amount of technical debt that is okay from a business point of view for the project to take on."
This is a tough pill to shallow for most programmers and Steve even had a potential explanation for this perspective:
"A lot technical staff have been burnt by implicit debt often enough that they have gravitated towards to the position that the only good amount of debt is zero"
And once again I think this comes down to what ones perspective is on their work. Firstly are you explicit about the decisions you are making? And are you explicit about the short cut you are taking? Finally have you got enough confidence and more importantly professionalism to fight for the need to fix this short cuts at a later date. Obviously if it makes sense to fix those short cuts. But there again is the point, deliberateness.
What constitutes technical debt?
Some of the more generally agreed upon examples of technical debt are:
- Lack of documentation
- Hacks and short-cuts we take when under schedule pressures
- Modules of code that everyone hate working on and will do their best to avoid.
- Compiler/linter warnings/errors.
- Extremely low quality code obtained during an acquisition.
- Technological progress, the solutions of today are the legacy systems of tomorrow.
- And perhaps even types of vendor lock in
There is a bit of debate about this point. Some professional out there, such as Bob Martin, would be inclined to think that a mess is not debt. What Uncle Bob means by “a mess” in this context is unprofessional practises like poorly formatted code, badly named variables, badly written comments…etc. To Bob these are down to laziness, they are not decision taken with the aim of achieve anything in particular.
"The more technical debt you take on, the tighter your disciplines need to be. You should do more testing, and more pairing and more refactoring. Technical debt is not a license to make a mess. Technical debt creates the need for even greater cleanliness."
And on reflection I suppose I’d have to agree with him. I listen to a great podcast by a very experienced entrepreneur Sean McCabe. He is a part of the design industry. The whole point and purpose of his podcast is about bringing more professionalism to his industry. He’s a hand letter and given that premise you’d think there wouldn’t be that much to learn. You would be wrong to think that. In particular there is one thing that Sean says a lot about designers and artists a lot. He talks about the situation where someone ends up with awkward clients and they lament it. What Sean says about this situation is, “There’s no such thing as clients from hell because only designers from hell take on those type of clients”.
I think Uncle Bobs perspective of debt and a mess is coming from the same place. So to paraphrase Sean:
There are no code bases from hell just programmers from hell who would choose to work on those code bases
How to manage and track debt?
To be completely frank. I’m not sure. I’ve been reading a lot about this topic in preparation for writing this post and it would seem there are a variety of good options, all with their pluses and minuses.
There is a great article by David Laribee called Code Cleanup - Using Agile Techniques to Pay Back Technical Debt. In this article he talks about using a system thinking process called Theory of Constraints (ToC) which was developed by Eliyahu Goldratt. The idea behind it seems to be identifying bottle necks and some degree of root cause analysis. It certainly seems like an idea I will be looking into in the future.
A few presentations I watched talked about using Scrumm and Agile development methodologies to help manage and track debt. In these development system you are talking about what you are doing at a very fine grained interval (daily stand ups), perhaps even pair programming. In Scrumm you might make a story or task which you would estimate when taking on debt. After a time you re-estimate you backlog items you get to find out if your debt is going up or down, and if you are monitoring velocity you can perhaps correlate those decisions against those value too.
Other means of measuring debt would include using tools like Sonar to track metrics like cyclomatic complexity. Though it generally consider a good idea to reduce values such as cyclomatic complexity it is dangerous to have people focus on numbers. Metrics are a powerful tool, but in the wrong hands or just used incorrectly can do more damage than good. Just look at what happened with most justice systems around the world. Many countries are coming around to the dangers of metrics in their police forces, the health care system, their schools. Should the business and engineering world pay attention as well?
I’m not an expert by any means. I have an Internet connection and an interest. From what I can tell they type of things that would be worth doing might be:
- Audit your system. Find out how much debt there is.
- Static analysis tools can tell you a lot about your system. Try them out, understand what the metrics mean, use them.
- Read your code base, look for “code smells”. Do a bit of code grepping, a tool like ack! really helps with this process.
- Keep an eye on all new debt being generated.
- Use peer code reviews to ensure no obviously technical shortcuts are being taken. And if they are track them.
- If possible use pair programming
- Train you programmers to have a higher bar of quality.
- Start in house technical talks.
- Invest in good books and on-line courses.
- Create a coding standard.
- Enforce you coding standard.
- “An elegant code base is one that was worked on by multiple people but looks like it was worked on by one person” - Nina Zakharenko
- Stay accountable
- Pre commit hooks (run tests)
- Continuous integration
If the above seems potentially overwhelming and perhaps you are worry about the cost it will have on your productivity it might be worth considering this perspective from David Simner.
"Instead of looking at the entire application and looking at metrics over it, you basically just look at what you're going to have to change. Because some bits of the application are just going to sit there and not have to change. More interesting is the bits you are going to have to change and how those bits affect technical debt."
There are a lot of intertwining issues when it comes to using the debt metaphor. It’s first and foremost role is to help you as a programmer find a way of talking to business staff. The language we use as engineers is alien to them. We need a common vernacular. Debt is way of talking about technical decisions that a lot of people can understand. Much more common and in reach than talking about “refactoring” a “clean code”. One of the biggest problems for me in my career is communication and developing my “soft skills”. Ultimately programming is about people. We are making tools for people with people. Finding novel ways of thinking and talking about problems is what separates programmers from professionals.
Below are a series of articles and videos I used to research this topic. You might find it interesting to dive into them and get a deeper/better sense of this topic yourself.
Videos & Talks
- “Technical Debt - The code monster in everyone’s closet” by Nina Zakharenko
- “Measuring Technical Debt” by David Simner
- Steve McConnell talk “Managing Technical Debt”
- IEEE Computer Society Podcast “Steve McConnell on Technical Debt”
- The OnTechnicalDebt episode “How to Categorize & Communicate Technical Debt” featuring Steve McConnell
- 10x Software Development - Technical Debt by Steve McConnell
- “An Appropriate Use of Metrics” by Patrick Kua
- “Design Stamina Hypothesis” by Martin Fowler
- “Technical Debt Quadrant” by Martin Fowler
- “Code Cleanup - Using Agile Techniques to Pay Back Technical Debt” by David Laribee
- “A Mess is not a Technical Debt” by Bob Martin
- http://www.ontechnicaldebt.com/ - “OnTechnicalDebt is the premiere social media community for discussing Technical Debt”
- Ack! - ack is a tool like grep, optimized for programmers
- Seanwes Podcast - The seanwes podcast is to help people make a living with your passion.
- SeanWes Podcast Episode 122 “10 Mistakes You’re Making With Clients That Cost You”.
- “Eliyahu Goldratt Consulting”