The only way out of this box, IMHO is proper layering of software and proper interfaces. Human brains have a limited capacity to manage complexity. There is only so much you can fit into your head at one time. (I dunno, maybe it's just me, but I've been around long enough to have hit this limit repeatedly, and started forgetting about code I have written in the past).
To combat this, humans require abstraction. A proper abstraction fully encapsulates the implementation details and provides an interface that does not require knowledge of the details behind the abstraction.
We have too many leaky abstractions. Bugs, performance issues, brittle software broken by updates. Language features that break in complex ways when users make mistakes (hello C/C++!) A leaky abstraction lets its details spill out through the backdoor. These cause cognitive load and make it impossible to layer software properly.
The only way to build a big system is proper abstractions.
Until Spectre and Meltdown, ISAs were good abstraction layers.
Today, still, my personal opinion is that the kernel system call layer is one of the strongest abstraction boundaries that we managed to enforce. Above this layer, it's all a mess. Below this layer, it's all a mess. (To a first approximation).
We aren't getting anywhere until we get the layers right.
To combat this, humans require abstraction. A proper abstraction fully encapsulates the implementation details and provides an interface that does not require knowledge of the details behind the abstraction.
We have too many leaky abstractions. Bugs, performance issues, brittle software broken by updates. Language features that break in complex ways when users make mistakes (hello C/C++!) A leaky abstraction lets its details spill out through the backdoor. These cause cognitive load and make it impossible to layer software properly.
The only way to build a big system is proper abstractions.
Until Spectre and Meltdown, ISAs were good abstraction layers.
Today, still, my personal opinion is that the kernel system call layer is one of the strongest abstraction boundaries that we managed to enforce. Above this layer, it's all a mess. Below this layer, it's all a mess. (To a first approximation).
We aren't getting anywhere until we get the layers right.