The Three Dichotomies of State

If you’ve worked with a database before, you’re almost certainly familiar with the many countless hours spent writing code that translates the database’s layout into the application’s in-memory structures. Like a treadmill, this work is entirely for the outcome, not the journey.

But why? After decades of modern databases and many billions of dollars invested, why can’t the application and database be unified? Haven’t object-relational mappers (ORMs) mostly fixed this? What about document and object databases?

The answer is not a convenient one. The fact is that the structure of state is governed by one of two life spans: in-memory or at-rest. Structurally they are superficially similar, but this obscures truly fundamental differences. Differences that can be expressed in three dichotomies.

Durable-Ephemeral. Most fundamental is that state at-rest is expected to be perpetual. It is universally assumed that in-memory state is temporary. Once at-rest, the structure of state is resistant to change. Altering the layout of in-memory state is trivial.

Generic-Specialized. The laborious nature of changing the structure of state at-rest has a major implication. To avoid repeating this time-consuming and risky process, it makes sense to design generic structures that can suit many potential future use cases. This compromise distances the at-rest state from the application’s in-memory model, which is typically far more specialized for today’s requirements.

Centralized-Distributed. The trustworthiness of state at-rest is of supreme importance. Just restarting the process will never fix it. A centralized, single source of truth eliminates many problems that cause loss of trust. But it brings along with it the opportunity for bottlenecks and cascading errors. In-memory state is different. It is local to an address space. Changes in structure as well as mistakes and errors do not automatically propagate to other processes.

So what about these supposed solutions like ORMs as well as document and object databases? The reality is that they just bend the application towards an at-rest data model. They can ease the tedium of interacting with data at-rest, but the fundamentals remain.

While document and object databases have attained relatively limited adoption, Redis is everywhere. The key to much of its popularity is the implication that it is good to store state at-rest using a more convenient and familiar in-memory approach. While there are certainly numerous use cases where it shines, those who have faced changing requirements for at-rest state stored in Redis can attest to the consequences of its convenience.

Instead of fruitlessly searching for a fix or ignoring these dichotomies, they should inform the way that software is built. Most importantly, investment in the design of at-rest state should be prioritized while fretting too much over the long-term implications of in-memory state should be avoided.