Cognitive Ecologies

I love this Substack note by Venkatesh Rao.

Makes me think about the ceiling on scaling. We're basically feeding LLMs our civilization's exocortex, and maybe all scaling gives us is progressively better access to that structure; you're getting better at compressing and recombining what civilization has already figured out, which is very different from intelligence just continuing to compound indefinitely with scale.

Of course the point on traces means the environment starts doing cognitive work too, and a weaker model in the right ecology could maybe do things a much stronger isolated model can't. The HF incident is a nice example of this: the shared message board became part of the cognitive system.

And maybe the ecology doesn't just add capability, but changes the system's effective priors too. Different memory structures, communication channels or organizational forms could make some reasoning trajectories much easier than others. The environment isn't just scaffolding around the intelligence; it starts becoming part of the architecture that produces it.

Once those traces persist across runs, the line between training and deployment gets a little blurry too. The weights may stay fixed while the system still accumulates conventions, procedures, heuristics, shared memory (basically a kind of culture). Capability can keep evolving without the model itself changing.

That suggests another scaling axis entirely: not just model scale, but ecological or cultural accumulation. Better persistence, coordination and inheritance could keep increasing what the system can do even if the base model stops improving.

So we should probably care a lot more about corrigible environments than corrigible minds. We don't solve governance by solving human alignment; we build institutions that make decent outcomes possible despite the fact that people will be people. AI alignment may end up looking a lot more like institutional design than moral education.

What's great about all this is that it spawns a fascinating research direction: how much intelligence can you gain just by changing the world the model gets to think in? And you can probably test a lot of this without big models at all.