Today I opened Chapter 4 of Martin Kleppmann’s Designing Data-Intensive Applications. I think this is the fourth time I’ve read this chapter over the years. And this time, a sentence I had seen before suddenly felt different.
The chapter opens with this quote:
“The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.”
The first time I read it, I understood the joke.
There are things that might go wrong, so we prepare for them. And there are things that we believe cannot possibly go wrong, so we don’t. Then, when they do go wrong, we discover that we have no idea what to do.
But today I kept thinking about the second part of the sentence. Not about things going wrong. About the moment when we decide that something cannot go wrong.
The comfort of certainty
There is a strange thing that happens when we become certain about something. We stop looking at it. We stop checking.
We stop leaving room for another possibility. And perhaps that is where the real danger begins.
I don’t think the lesson is that we should expect everything to fail. That would be exhausting. I don’t want to wake up every morning wondering which part of my life is about to collapse. There is a difference between expecting disaster and accepting possibility.
Maybe the healthier position is somewhere in between:
I don’t expect this to go wrong. But I know it can.
That small difference changes how we approach things.
The same idea appears in engineering
This is where the quote becomes much more than a philosophical observation for me. Consider a database. You might have a production system that has been running for three years. The database has never been unavailable for more than a few seconds.
Eventually, the statement
“The database is always available”
stops sounding like an assumption. It starts sounding like a fact. So perhaps there is no meaningful timeout on some calls.
Perhaps there is no useful degraded behavior. Perhaps nobody has tested restoring the backup. Perhaps the application has never been run against an empty database. Everything works beautifully. Until the database doesn’t.
Now the problem isn’t simply that the database went down.
The problem is that the system was designed around the idea that this couldn’t happen.
And that is exactly where Adams’ quote becomes interesting.
A failure that was considered possible usually gets somewhere in the design.
A failure that was considered impossible often doesn’t.
We do this everywhere
Take message processing.
Suppose we consume messages from a queue and assume every message will eventually be processed successfully.
So the consumer does this:
receive message
↓
process
↓
delete message
It works. Until the process crashes halfway through. Or the message contains data that the code doesn’t understand.
Or a downstream service is unavailable. If we had considered those possibilities, we might have designed for retries, idempotency, dead-letter messages, visibility timeouts, and a way to inspect what failed. If we considered them impossible, the system may simply get stuck.
The queue didn’t become unexpectedly complicated.
Our assumption was too simple.
The most dangerous assumptions are often boring
We tend to think about catastrophic failures. The data center burns down. The network disappears. The entire database is corrupted. But many painful incidents begin with much smaller assumptions:
“This field will never be null.”
“This ID will never exceed this size.”
“This endpoint will never receive more than 100 requests per second.”
“This configuration won’t change.”
“This table won’t get that large.”
“This service will always return within a second.”
“This person knows how this works.”
“This code is temporary.”
None of these assumptions sounds unreasonable on its own. That’s what makes them dangerous. The problem isn’t having assumptions. We cannot build anything without them.
The problem is forgetting which parts of our design are assumptions in the first place.
Preparation doesn’t mean predicting the future
This is the part that I find more interesting outside engineering.
We often think preparation means knowing what is going to happen.
But we can’t know.
A good engineer doesn’t need to predict the exact failure.
They need to leave the system with a way to respond when their prediction is wrong.
That’s a different mindset.
We don’t need to know which machine will fail. We need monitoring.
We don’t need to know exactly when a deployment will break. We need a rollback.
We don’t need to know which message will be malformed. We need a way to isolate and inspect it.
We don’t need to know whether a database will become unavailable tomorrow. We need a recovery procedure that has actually been tested.
The goal isn’t to predict every possible future.
It is to make the system capable of surviving the fact that we cannot predict every possible future.
Leave a way back
And maybe this is where the engineering idea becomes a little more personal for me. If I know a door can eventually close, I don’t have to stand beside it all day. I just shouldn’t build my entire life around the assumption that it will remain open forever. If I know a plan can fail, I don’t have to abandon the plan.
I just need to leave myself enough room to change direction. If I know my circumstances can change, I don’t have to live anxiously. I just shouldn’t mistake stability for permanence. The same principle applies to engineering. A system should have a way back. A way to understand what happened. A way to recover. A way to replace something. A way to change a decision. A way to roll back.
A way to discover that one of our assumptions was wrong without having to rebuild everything around it.
Perhaps good engineering is not about making systems that never fail.
That is a much harder promise than we usually realize.
Perhaps it is about making systems where failure doesn’t become a dead end.
The practical lesson
Maybe the practical lesson is very small:
Leave a little room.
For things to change. For assumptions to be wrong. For dependencies to disappear. For requirements to move. For systems to fail. For ourselves to change our minds.
Not because everything is going to go wrong. But because nothing has promised us that it cannot. The dangerous thing isn’t that something can fail. Things failing is ordinary.
The dangerous thing is believing that something cannot possibly fail, and then building a world in which there is no way back when reality proves us wrong. This was my fourth time reading the chapter. The words were the same. The quote was the same.
But somehow, I wasn’t. Maybe that’s the real reason we reread things. Sometimes the book doesn’t give us a new idea. It gives an old idea a new place to land.
Leave a Reply