Response to "Meditations on Moloch"

A few months ago, I read Meditations on Moloch by Scott Alexander and thought it was the most absolute and true piece of horror ever discovered. However, Moloch’s grip is not absolute, and Alexander notes this if you look carefully. In the allegory of the rats, they note that population struggle isn’t quite that intense:

“For a bunch of reasons evolution is not quite as Malthusian as the ideal case, but it provides the prototype example we can apply to other things to see the underlying mechanism.”

I think something that Alexander doesn’t give enough attention to is multi-level evolution. Look at cells, multicellular organisms, and cancer. In a primordial soup, all cells compete to survive, grow, and reproduce. Those that don’t are outcompeted and eventually consumed. This is an absolute Malthusian trap. And this was the state of Earth for a few billion years. This competition led to several revolutions in cellular abilities (photosynthesis) and catastrophes (excess atmospheric oxygen) that did nothing but increase the intensity of competition between cells. A cellular philosopher of that time may wish that photosynthesis had never been discovered, so that the cells could be in their natural state of severely limited chemical energy, limiting how extreme their competition can become.

Then the most significant cellular technology developed: coordination. Cells could band together to form large and more complex organisms. Within a single organism, resources were allocated to optimize the good of the organism, even if some cells were starved. Without coordination, no cell would ever agree to be strategically killed for the greater good. But these organised clusters of cells competed on the same battlefield as uncoordinated cells and could outcompete single celled organisms. This eventually leads to a new Malthusian dynamic - that between multicellular organisms once they have filled their niche. But the life of cells within any one multicellular organism is distinctly non-Malthusian. Moloch has been killed. The cells allocate resources between each other within the multi-cellular organism, and failure to do so results in the death of the organism. And just as the cell “wants” to stay alive by keeping its molecules in the right patterns, the organism evolves a “want” to stay alive by coordinating the cells. The body moves up a level, and what started as an organization between cells is now an entity of its own, with desires that it exerts will to maintain.

I don’t use words like “want”, “entity”, and “will” lightly. The multicellular organism starts out without any agency at all. They are more like a loose swarm, with a collective will that is merely the sum of the will of the individuals that make it up. Individuals can defect, choose their own good over the good of the swarm and go out to live alone. When evolution rewards tighter and more effective cooperation, at some point the balance is tipped and the organism becomes a single coherent organism. Cells specialize and can no longer leave the organism. Cannibalizing cells is as easy as metabolizing an other non-living material around. The coherent whole now has needs of its own, and resources that are not important to any individual cell now become needs of the multicellular organism.

  1. Individual cells compete for resources
  2. Some cells gain an advantage by cooperating, such as not eating each other or sticking together in packs.
  3. These collectives become more coherent and gain a will of their own and individual cells’ goals are readily sacrificed for the goals of the organisms. The lives of individual cells are governed by non-Malthusian dynamics.
  4. The will of the multi-cellular organism becomes the dominant force in the world, and the organisms compete among themselves, leading to a higher-level Malthusian dynamic much like the one before

This process happens all the time. Naming and generalizing the above steps, we can make analogies to many other phenomenon that result in local pockets of coordination.

  1. Chaos: Individuals of species A compete for resources, sacrificing their well being for competitive fitness.
  2. Cooperation: Some individuals work together, but still pursue their own well being.
  3. Cohesion: The groups of A that work together more tightly can reproduce and form more groups of A with the same collective DNA, until the collective’s identity becomes so strong as to be identified with a new species, B, and the As that make up Bs are now components instead of individuals.
  4. Chaos: Individuals of species B compete for resources, sacrificing their well being for competitive fitness.

An occurrence of this would be the formation of tribes of humans, followed by the agriculture catastrophe. In the formation of tribes (Cooperation), those humans freed themselves from fighting each other so much. Our social abilities limited the size of these tribes to a few dozen, and anyone outside of the tribe is just as much of a competition problem as before. However, it was fairly easy to leave the tribe and either live one your own or find a new one. Because of this, the tribes values were directly driven by human values. At some point, humans became more sedentary and lost their ability to leave whenever they wanted, and the tribes became more permanent entities (Cohesion). The tribe no longer needed to respect individual values and could optimize for large scale fitness against other tribes, leading back into the Malthusian trap on a new level (Chaos).

I say that Moloch is not absolute because this cycle seems to happen on every level, until all resources are collected into a single very hierarchical entity. The formation of multi-cellular organisms and tribes of humans is very clear, but I think that there are more subtle instances of organization, that takes the form of systematic forces instead of concrete organisms. For instance, the system that consists of all commerce in the world is coherent and interdependent, but doesn’t act with a single will the way a country does. There may be very complex “hive organisms” that consist entirely of humans and our activities, that we are unaware of, or that are intentionally structured to make us unaware of them. Or maybe they are simply outside of our cognitive reaches or we don’t have the right science to understand them, like how our science has yet to gain but the smallest footing on understanding the mind.

If we understand our social structures and freed ourselves from them, where would we be?

If your cells understood your body and wished to free themselves from its servitude, where would they be?

“The reason our current system isn’t a utopia is that it wasn’t designed by humans. Just as you can look at an arid terrain and determine what shape a river will one day take by assuming water will obey gravity, so you can look at a civilization and determine what shape its institutions will one day take by assuming people will obey incentives.” - Meditations on Moloch

Relation to AI safety

We are controlled solely by systematic force, meaning we will either discard all of our values except reproduction (Malthus) or form cohesive system and let it rule our values from above (holism). Even the most iron-fisted dictators still depend on systematic forces to grant them power. The image of the sole dictator, affecting everything but affected by nothing, is largely a myth (or at least CGP Grey makes this claim).

In our current state, without any superintelligence, we are still controlled by forces larger than ourselves which are often not aligned with anyone’s values. The collective being of humanity doesn’t even seem to be aware that it can die from its actions, given its inability to stop polluting the oceans and generating greenhouse gasses.

Deciding not to create superintelligence AI is to doom ourselves to doom ourselves to this different kind of maligned force, one that also poses existential risk. This needs to be considered when weighting the risks and benefits of construction potential superintelligence. If we could build a superintelligence tomorrow, and it had a 10% chance of replacing the universe with paperclips, should we do it? What are the odds that humanity destroys itself, and no intelligent life ever comes back, like in H.G. Wells’ “Time Machine”? If our odds are better with the superintelligence, it would be the right thing to do.

(This is for the case where AI safety failure results in a dead universe, with no suffering. If the superintelligence might instead fill the universe with maximum suffering, then the decision becomes more tricky.)

The third option is to use social sciences and maybe weaker artificial intelligence to control the systems that currently control us, for our collective and individual benefit. I think this is the idea of politics, but the usual route seems incredibly ineffective, and the system might learn immunities against activism. I believe this immunity mechanism might have happened in America: minimal contact with others in public, always busy working population, negative portrayal of effectiveness of protests, stigma around activism only being for certain kinds of people.

As I can tell, these are our four paths: Malthusian prison, rule by uncontrollable systems forces, universe controlled by AI, and universe controlled by world controllers with advanced social sciences. None of these are particularly promising, but the last two offer a shred of hope of preserving human values.