Overengineering and Overadoption

As coders, we usually begin our careers by throwing together commands sprinkled with single-letter variable names and wild contortions of logic. The code usually does the job inefficiently and fails in catastrophic ways in unexpected situations. As we progress, our code becomes more robust and correct, but for a while, we continue to put together code that isn’t very well-designed. Code like this is “underengineered.” Underengineering is bad, unless it’s the result of a conscious decision to get the job done faster by skipping most of the design phase and taking on some “technical debt” (AKA “design debt”).

After a couple years of unintentionally writing underengineered code and dealing with the fallout, we compensate by planning ahead and designing better systems which anticipate future needs. Our code becomes easier to read, understand, and work on. That might be the sweet spot, because the next step for most of us is overengineering.

Once we get good at designing systems in abstract ways, we often proceed to develop them in the most abstract ways possible. Overengineered projects often take months or years to complete. It’s difficult to get consensus when every last detail has to be perfect. When we overengineer, it’s likely that by the time we’re done, the business or technology environment will have changed in ways that render much of that excessively well-planned work irrelevant. And, we can’t see the future, no matter how hard we try. When we overengineer, despite all of our planning and abstractions, we often fail to anticipate the abstractions we’ll actually need. Voltaire is credited with the saying, “The perfect is the enemy of the good.”

Why do we overengineer? It seems smart at first. If some abstractions are good, then lots of abstractions are better! Programmers also get bored. We like to solve new problems and develop new skills. A coder friend both wise and tired beyond his years once told me, “There are only seven problems in software engineering, and we just solve them over and over again.” It took me years to understand what he was saying, because I’ve always said, “Software engineering is the best job in the world, because it’s never the same job twice. If it is, you’re doing it wrong!” In fact, we’re both right. His point was that although we may be solving new problems with new languages, abstractions, techniques, and technologies, at the bottom of it all will always lie just a few fundamental solutions. Loops. Conditionals. Functions. Reading from and writing to files and databases. Because of this, programmers can get bored even while engaged in one of the most interesting jobs in the world.

Overadoption┬áis a similar problem that doesn’t get nearly as much attention as it deserves. Programmers love new technology. But the newest, coolest technologies are also the most buggy. Last year I found and fixed a bug in our payment system that could have caused us to approve all payments of a certain type without actually checking to see if those payments were authorized. The bug was not introduced directly through the fault of any of our coders. It had been made possible long ago by the inclusion of an open-source software package that we were using before it was quite ready for enterprise use. After we installed it, it went through so many changes so quickly that we couldn’t keep up. Updating the package would have broken our code, so we were stuck living with a bug that had been fixed years ago. We were early adopters. But it’s possible to adopt too early. We’ve also quickly adopted and trashed various wiki systems, file systems, project management software systems, and version control schemes. We’ve gotten to a pretty good place today, but several of the steps in between could have been skipped entirely if we’d just been a little more patient and reflective before jumping on those rickety bandwagons.

Engineers sometimes, usually unintentionally, make themselves indispensable by hoarding knowledge about arcane legacy systems. It’s quite possible to become indispensable for the opposite reason as well. If the latest, greatest technology is installed the moment it’s released, it will be difficult for everyone in the organization to keep up. We all enjoy learning new things, and building our skills helps the organization as a whole. But sometimes the best thing we can do for the company is to go to work and solve those seven boring problems all over again for a few hours. Sometimes we just have to sit down and bang out some boilerplate.

William F. Buckley, Jr. said, “A Conservative is a fellow who is standing athwart history yelling ‘Stop!’” I’m not a conservative. I’m just standing athwart technology calmly advising, “Slow down a little.”

Interested in working at Shutterstock? We're hiring! >>
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

6 Responses to Overengineering and Overadoption

  1. Kyle says:

    I think that abstraction is good, but it’s wrong to measure “abstraction” as an ends in itself. Abstraction is not good for some mystical or aesthetic reason. It’s good because it means we can write less code! And less code is what we really want. Problems like “the business or technology environment … changed in ways that render much of that excessively well-planned work irrelevant” are really just symptoms of having too much code to maintain. If we had 1/10th as much code, we could adapt to the new requirements in 1/10th the time, and then it wouldn’t be a problem at all.

    Sometimes abstraction is a good proxy for code size. Sometimes it’s not. One way that works really well is programming languages: a programming language with more abstraction (either built-in, perhaps through a big and useful standard library, or by the language’s features, in letting us build higher-level abstractions ourselves) is always a win, in my experience. I’m sure there are exceptions, but for every program I’ve written professionally for 20+ years, using C is always better than using assembly, and using Ruby/Python is always better than C, and so on. You write less code because the guys who wrote the compiler and runtime for the language did it for you already. You’re outsourcing the dull stuff, to good programmers, who have already finished writing it, and debugging it, and you’re using a standard interface that everybody knows: this is a great way to avoid writing code!

    It sounds like you’ve run into problems when people can’t distinguish between “abstraction” (as an abstract concept) and “less code” (which is very easy to measure), and make trade-offs that give you more abstraction, but at the cost of having more code to maintain. I’ve never seen “too much abstraction” itself be a problem. “Too much code” is almost always a problem, though.

  2. Fire says:

    This is an awesome blog, I particularly relate to making over abstractions born of bordom, very good point.

  3. Gerardo says:

    Excellent post, couldn’t agree more with your point of view. =)

  4. Pingback: Yet Another Perl Conference: The Shutterstock Tech Team Goes to Austin | Reina Gallery

  5. Lennie says:

    You only listed 7 problems, I think you need at least networking or task/messaging-queuing as well.

  6. Val says:

    I looked back at usage numbers for a feature we originally expected to get much more traffic and showed them to our product manager. The response was – better safe than sorry.
    In other words, waste in the form of overengineering is generally accepted, if not required. Since there are rarely any metrics to document the waste, it’s easy to get away with it and hard to learn from it.