The Complexity Monster
As developers, we all know that we should write code that can be easily understood by fellow developers. Machines don't care about how you name things, about the fact that you use global variables everywhere or that you want everything to be multi-threaded. Humans do. We can only keep a few things in mind at any given time and we really struggle when things get too complex. We create more bugs in our software and change becomes difficult.
To avoid that, we should constantly focus on simplicity. When faced with two or more alternatives that deliver the same value, we should pick the one that one that leaves the system in a simpler state. But simplicity is not easy to achieve. In fact, is damn difficult.
Nowadays everyone does (or tries to) use agile processes to deliver things faster and avoid wasted time. But agile principles also help in reducing complexity. By iterating quickly on a project and doing small steps towards our goal, we avoid over-engineering our solutions or building things that we don't need.
That said, doing small iterations might lead to different parts of the system evolving independently and becoming non-cohesive. It's important to step back and identify pieces that could be changed so the whole system becomes simpler.
Here we get to the monolith vs. micro-services debate. Is it better to have a large application that knows and does everything or a swarm of small applications that do only one thing? As all interesting debates, it depends. Among other things, it depends on your domain model and the size of your company. But most of all it depends on the complexity of the system. Is your monolith so complex that it takes weeks for new starters to understand? Does reasoning about a feature require you to print out a graph of all the calls made between micro-services?
The solution is well known. We have to split the functions of the system into loosely coupled services that have high internal cohesion. By having high cohesion, adding a new feature requires modifying only a small number of services. By decoupling, you can focus on solving your problem without having to keep the whole system in mind. But in practice it's not always easy to find out the perfect place to split the system.
Build vs. Buy
Paying for third-party services is a great way to externalise complexity.
It's obvious that most of the companies shouldn't be building their email services, but it should be pushed further than that. Do we want to maintain our own hardware or can we use cloud computing? Do we want to maintain our servers or can we use a platform as a service? What about the database?
To decide, it's important to determine whether the service/function is strategic or not. Will building and maintaining X give you a strategic edge against your competitors? If that's the case, go ahead. Otherwise, you'll probably be better off paying for it.
Tools & Frameworks
Using external tools and frameworks is another great way of minimising complexity. Want to retrieve a piece of data? Just run the right query and it's yours. No need to think about how the data is stored on disk or about replication. Someone else did that for you. Of course, big companies build their own databases, web frameworks and programming languages. But the rest of us we should externalise as much complexity as possible.
A good way of deciding between two tools that solve your problem is to pick the most generic and flexible one. And then using it in the most simple way. It's not only the hardware that you might reuse, it's also the knowledge to use and maintain it. Need a cache? By using Redis instead of Memcached, you will be able to use it for many more functions in the future. The same goes for SQL vs. NoSQL databases, Spark vs. Storm and generic vs. niche programming languages.
At a line-of-code level, I've found that TDD works great to achieve simple solutions. By only focusing on making one test pass at a time, frequently I've arrived to a solution that is much simpler than the one that I would have designed. It might be not as elegant or extensible, but the fact is that it's simpler and many times it won't need to be extended at all. TDD is doing agile at a micro level.
Another way to achieve simplicity at code level is to use more restrictive languages. By limiting what you can do with them, it becomes much easier to reason about the code. Good examples are strongly-typed languages, immutability and functional programming.
When building systems, there are many more considerations to take into account than complexity. Things like cost, vendor lock-in, performance, team expertise and legal constraints clearly have to be considered as well. But by explicitly trying to build simple systems we'll be able to iterate quickly and scale teams and companies.
Do you have more tips to reduce complexity?