We observe emergent behavior in stunningly diverse parts of our world, ranging from collective animal behavior to economic systems. Emergence (as defined by Wikipedia) “occurs when a complex entity has properties or behaviors that its parts do not have on their own, and emerge only when they interact in a wider whole”. In other words, emergent (or system-level) outcomes are more than (and different from) the sum of their parts [?]. A wide range of system-level outcomes can emerge from a collection of agents following their own, often simple, reward/value functions.
On one hand, structure can emerge: for example, migratory birds fly in a V formation despite the lack of system-level organization. On the other hand is the emergence of chaos: epitomized by the double pendulum, a composition of simple systems can lead to chaotic behavior. Similarly, agentic systems will inevitably develop emergent system-level outcomes: perhaps cooperation to achieve a human-aligned goal, perhaps cooperation to subvert their intended purpose, or perhaps collapse into complete societal disarray. As agentic systems gain prominence, I posit that we should learn to design for emergent global behavior in the same way we think about designing individual agent behavior. How do we design for healthy cooperation to emerge? What about alignment? To answer these questions, we must first demystify the process of emergence and its connection to individual agent actions as well as the agents’ environment.
We begin by exploring how complex phenomena can emerge just from the basic architecture of a system, even in the absence of intelligent players. We then explore emergence in systems of intelligent agents, specifically focusing on collective animal behavior. In the next post, we will connect these principles to multi-agent system design and AI/AGI safety.
Self-organization leads to complexity
The canonical example of emergent complexity is Conway’s Game of Life, a cellular automaton model that evolves deterministically starting from an initial state. The game takes place on an infinite 2D grid where each cell’s state at any iteration is a fixed function of the states of its neighbors. Despite the simplicity of the rules and setup, Conway’s Game of Life is proven to be a universal Turing machine, and an incredible taxonomy of “organisms” defined by unique patterns have been observed. This and other cellular automaton models demonstrate that the emergence of complexity in systems occurs independently of the presence of agency or incentive structure.
Three types of organisms in the Game of Life: A "puffer-type breeder" (red) that leaves glider guns (green) in its wake, which in turn create gliders (blue). From Wikipedia.
But the emergence of complexity is only part of the story: we’re further interested in what kind of equilibrium a system settles into, and many systems seem to tend toward criticality.
Criticality refers to the threshold state of a dynamical system poised between order and chaos.
Sandpiles are a classic example: grains accumulate until the system reaches a critical point, where a single additional grain triggers a massive avalanche (modeled by the Bak-Tang-Wiesenfeld model [?] in the self-organized criticality literature).
In other words, some systems arrive at the edge of instability through their own internal dynamics alone, without the input of insidious actors or misaligned incentives.
We can shed some light on this tendency by observing that the behavior of these systems can be expressed along a spectrum.
At one extreme, all cells rapidly die or settle into repeating, predictable patterns — the “ordered” regime.
At the other extreme is complete unpredictable randomness, which is equally uninteresting.
Somewhere in between is the critical regime: stable enough for information to persist and propagate, but flexible enough to avoid fully predictable solutions — the “edge of chaos.”
This is where the interesting emergent properties live, and it is not a given that a system finds its way there.
As we’ll see, the behavior of intelligent agents and the structure of their environment are our primary levers for shaping where on this spectrum a multi-agent system lands.
Behavior of 1D cellular automata over time (shown on $y$-axis) as a function of $\lambda$, a measure of homogeneity. At small values of $\lambda$, cells die out quickly or fall into repeating patterns. At large $\lambda$, randomness and chaos begin to take hold and eventually dominate. Only in the intermediate $\lambda$ regime (e.g., $\lambda=0.40$) do interesting and nontrivial patterns emerge. Adapted from [?].
Selfish intelligent agents can give rise to complex emergent behavior
Biological systems offer a masterclass in multi-agent coordination and emergence in the natural world.
In the absence of explicit leadership, birds self-organize to fly in formation, animals travel in herds, ants form long meandering trails to transport food, and fish swim in schools that move as one to adeptly avoid predation.
In fact, these system-level behaviors arise from purely selfish, local incentives.
In “Geometry of the Selfish Herd” (1971), Hamilton [?] put forth the theory that selfish incentives actually lead animals to engage in social behavior and herding. He posits that each animal seeks to reduce its own “domain of danger”, the physical area for which they are the most likely prey. He illustrates this nicely through a 1D example of frogs sitting around a circular pond, where a water snake emerges at a random location and attacks the closest frog. Each frog can achieve its minimal domain of danger by positioning itself in between other frogs, leading to clustering behavior.
Frogs reduce their domain of danger by clustering close to other frogs in Hamilton's selfish herd theory. Adapted from [?].
The Boids (bird-oids) simulations [?] further demonstrated that flocking behavior in birds can be reproduced by specifying 3 simple rules for each agent’s behavior: maintaining (1) reasonable separation between itself and its neighbors, (2) alignment with neighbors’ heading angles, and (3) cohesion with its neighbors. Most importantly, just as hypothesized for animal groups, no group-level control mechanism is needed to produce flocking behavior – it is sufficient to simply specify agent-level behavior and rewards.
Emergence of structure as opposed to chaos depends on the environment
A simple model for collective behavior introduced in 1995 by Vicsek, et al. [?] uncovers a phase transition from random chaotic movement to the emergence of flocking exclusively through altering environmental parameters. The Vicsek model holds for any self-propelled particles (animals, bacteria, etc.) and assumes simply that, at each timestep, each particle will align its heading angle with its neighbors’ headings.
Specifically, each particle $i$ has a position $\xbf_i(t)$ and velocity $\vbf_i(t)$, where the vector $\vbf_i(t)$ is defined by velocity value $v$ (assumed global to all particles) and heading angle $\theta_i(t)$ at time $t$. At each timestep, each particle adjusts its heading angle and position as follows:
\begin{align*}
\theta_i(t+1) &= \langle \theta(t) \rangle_{r,i} + \eta_i(t) \cr
\xbf_i(t+1) &= \xbf_i(t) + \vbf_i(t) \Delta t
\end{align*}
where $\langle \theta(t) \rangle_{r,i}$ is the average heading of all neighbors $\theta_j(t)$ where particle $j$ is within some radius $r$ of particle $i$, and $\eta_i(t) \sim \textrm{Uni}(-\eta, \eta)$ is a randomly chosen noise/temperature factor with amplitude parameter $\eta$.
$\eta$ can be interpreted as noise in the communication channel between particles, or as the temperature parameter in an Ising-style model.
The key result from Vicsek et al. is that noise values lower than some critical value $\eta_c$ (and choices of particle density $\rho$ greater than critical density $\rho_c$) will induce a phase transition from complete disorder to an ordered state of synchronized collective transport.
The Vicsek model predicts a phase transition from a disordered state (left) to an ordered state with bulk flow (right) as environmental parameters (density and noise) change. Adapted from [?].
Notably, this phase transition is induced by altering environmental parameters only ($\eta, \rho$), while the incentives/behavior of the particles remains unchanged. More sophisticated models proposed in later work (e.g., [?], [?]) further bolstered these conclusions.
In particular, Cucker & Smale (2007) [?] proved that, as long as certain inter-agent communication conditions hold, the system will always converge to the ordered state of collective transport.
In this post, we set the scene for understanding emergence in a variety of systems with and without intelligent agents. We saw that:
Complexity is an intrinsic property of systems, even simple ones with no agents.
Complex emergent behavior can arise from a collection of purely self-interested intelligent agents.
The environment (e.g., number density of agents, noise in the system) determines whether structure or chaos emerges.
Designing for emergence and the separation of environmental impact/feedback from individual agent rewards is a major departure from the typical discourse on agentic systems and reinforcement learning, and I look forward to delving into this further in the next post.
