\( \renewcommand{\vec}[1]{\mathbf{#1}} \newcommand{\avec}{\vec{a}} \newcommand{\bvec}{\vec{b}} \newcommand{\cvec}{\vec{c}} \newcommand{\dvec}{\vec{d}} \newcommand{\evec}{\vec{e}} \newcommand{\fvec}{\vec{f}} \newcommand{\gvec}{\vec{g}} \newcommand{\hvec}{\vec{h}} \newcommand{\ivec}{\vec{i}} \newcommand{\jvec}{\vec{j}} \newcommand{\kvec}{\vec{k}} \newcommand{\lvec}{\vec{l}} \newcommand{\mvec}{\vec{m}} \newcommand{\nvec}{\vec{n}} \newcommand{\ovec}{\vec{o}} \newcommand{\pvec}{\vec{p}} \newcommand{\qvec}{\vec{q}} \newcommand{\rvec}{\vec{r}} \newcommand{\svec}{\vec{s}} \newcommand{\tvec}{\vec{t}} \newcommand{\uvec}{\vec{u}} \newcommand{\vvec}{\vec{v}} \newcommand{\wvec}{\vec{w}} \newcommand{\xvec}{\vec{x}} \newcommand{\yvec}{\vec{y}} \newcommand{\zvec}{\vec{z}} \newcommand{\abf}{\mathbf{a}} \newcommand{\bbbf}{\mathbf{b}} \newcommand{\cbf}{\mathbf{c}} \newcommand{\dbf}{\mathbf{d}} \newcommand{\ebf}{\mathbf{e}} \newcommand{\fbf}{\mathbf{f}} \newcommand{\gbf}{\mathbf{g}} \newcommand{\hbf}{\mathbf{h}} \newcommand{\ibf}{\mathbf{i}} \newcommand{\jbf}{\mathbf{j}} \newcommand{\kbf}{\mathbf{k}} \newcommand{\lbf}{\mathbf{l}} \newcommand{\mbf}{\mathbf{m}} \newcommand{\nbf}{\mathbf{n}} \newcommand{\obf}{\mathbf{o}} \newcommand{\pbf}{\mathbf{p}} \newcommand{\qbf}{\mathbf{q}} \newcommand{\rbf}{\mathbf{r}} \newcommand{\sbf}{\mathbf{s}} \newcommand{\tbf}{\mathbf{t}} \newcommand{\ubf}{\mathbf{u}} \newcommand{\vbf}{\mathbf{v}} \newcommand{\wbf}{\mathbf{w}} \newcommand{\xbf}{\mathbf{x}} \newcommand{\ybf}{\mathbf{y}} \newcommand{\zbf}{\mathbf{z}} \newcommand{\alphabf}{\boldsymbol{\alpha}} \newcommand{\Betabf}{\boldsymbol{\Beta}} \newcommand{\betabf}{\boldsymbol{\beta}} \newcommand{\Gammabf}{\boldsymbol{\Gamma}} \newcommand{\gammabf}{\boldsymbol{\gamma}} \newcommand{\Deltabf}{\boldsymbol{\Delta}} \newcommand{\deltabf}{\boldsymbol{\delta}} \newcommand{\epsilontbf}{\boldsymbol{\epsilon}} \newcommand{\varepsilontbf}{\boldsymbol{\varepsilon}} \newcommand{\zetabf}{\boldsymbol{\zeta}} \newcommand{\etabf}{\boldsymbol{\eta}} \newcommand{\Thetabf}{\boldsymbol{\Theta}} \newcommand{\thetabf}{\boldsymbol{\theta}} \newcommand{\iotabf}{\boldsymbol{\iota}} \newcommand{\kappabf}{\boldsymbol{\kappa}} \newcommand{\Lambdabf}{\boldsymbol{\Lambda}} \newcommand{\lambdabf}{\boldsymbol{\lambda}} \newcommand{\mubf}{\boldsymbol{\mu}} \newcommand{\nubf}{\boldsymbol{\nu}} \newcommand{\Xibf}{\boldsymbol{\Xi}} \newcommand{\xibf}{\boldsymbol{\xi}} \newcommand{\Pibf}{\boldsymbol{\Pi}} \newcommand{\pibf}{\boldsymbol{\pi}} \newcommand{\rhobf}{\boldsymbol{\rho}} \newcommand{\Sigmabf}{\boldsymbol{\Sigma}} \newcommand{\sigmabf}{\boldsymbol{\sigma}} \newcommand{\taubf}{\boldsymbol{\tau}} \newcommand{\Upsilonbf}{\boldsymbol{\Upsilon}} \newcommand{\upsilonbf}{\boldsymbol{\upsilon}} \newcommand{\Phibf}{\boldsymbol{\Phi}} \newcommand{\phibf}{\boldsymbol{\phi}} \newcommand{\varphibf}{\boldsymbol{\varphi}} \newcommand{\chibf}{\boldsymbol{\chi}} \newcommand{\Psibf}{\boldsymbol{\Psi}} \newcommand{\psibf}{\boldsymbol{\psi}} \newcommand{\Omegabf}{\boldsymbol{\Omega}} \newcommand{\omegabf}{\boldsymbol{\omega}} \newcommand{\Abf}{\mathbf{A}} \newcommand{\Bbf}{\mathbf{B}} \newcommand{\Cbf}{\mathbf{C}} \newcommand{\Dbf}{\mathbf{D}} \newcommand{\Ebf}{\mathbf{E}} \newcommand{\Fbf}{\mathbf{F}} \newcommand{\Gbf}{\mathbf{G}} \newcommand{\Hbf}{\mathbf{H}} \newcommand{\Ibf}{\mathbf{I}} \newcommand{\Jbf}{\mathbf{J}} \newcommand{\Kbf}{\mathbf{K}} \newcommand{\Lbf}{\mathbf{L}} \newcommand{\Mbf}{\mathbf{M}} \newcommand{\Nbf}{\mathbf{N}} \newcommand{\Obf}{\mathbf{O}} \newcommand{\Pbf}{\mathbf{P}} \newcommand{\Qbf}{\mathbf{Q}} \newcommand{\Rbf}{\mathbf{R}} \newcommand{\Sbf}{\mathbf{S}} \newcommand{\Tbf}{\mathbf{T}} \newcommand{\Ubf}{\mathbf{U}} \newcommand{\Vbf}{\mathbf{V}} \newcommand{\Wbf}{\mathbf{W}} \newcommand{\Xbf}{\mathbf{X}} \newcommand{\Ybf}{\mathbf{Y}} \newcommand{\Zbf}{\mathbf{Z}} % mathsf letters \newcommand{\asf}{\mathsf{a}} \newcommand{\bsf}{\mathsf{b}} \newcommand{\csf}{\mathsf{c}} \newcommand{\dsf}{\mathsf{d}} \newcommand{\esf}{\mathsf{e}} \newcommand{\fsf}{\mathsf{f}} \newcommand{\gsf}{\mathsf{g}} \newcommand{\hsf}{\mathsf{h}} \newcommand{\isf}{\mathsf{i}} \newcommand{\jsf}{\mathsf{j}} \newcommand{\ksf}{\mathsf{k}} \newcommand{\lsf}{\mathsf{l}} \newcommand{\msf}{\mathsf{m}} \newcommand{\nsf}{\mathsf{n}} \newcommand{\osf}{\mathsf{o}} \newcommand{\psf}{\mathsf{p}} \newcommand{\qsf}{\mathsf{q}} \newcommand{\rsf}{\mathsf{r}} \newcommand{\ssf}{\mathsf{s}} \newcommand{\tsf}{\mathsf{t}} \newcommand{\usf}{\mathsf{u}} \newcommand{\vsf}{\mathsf{v}} \newcommand{\wsf}{\mathsf{w}} \newcommand{\xsf}{\mathsf{x}} \newcommand{\ysf}{\mathsf{y}} \newcommand{\zsf}{\mathsf{z}} \newcommand{\Asf}{\mathsf{A}} \newcommand{\Bsf}{\mathsf{B}} \newcommand{\Csf}{\mathsf{C}} \newcommand{\Dsf}{\mathsf{D}} \newcommand{\Esf}{\mathsf{E}} \newcommand{\Fsf}{\mathsf{F}} \newcommand{\Gsf}{\mathsf{G}} \newcommand{\Hsf}{\mathsf{H}} \newcommand{\Isf}{\mathsf{I}} \newcommand{\Jsf}{\mathsf{J}} \newcommand{\Ksf}{\mathsf{K}} \newcommand{\Lsf}{\mathsf{L}} \newcommand{\Msf}{\mathsf{M}} \newcommand{\Nsf}{\mathsf{N}} \newcommand{\Osf}{\mathsf{O}} \newcommand{\Psf}{\mathsf{P}} \newcommand{\Qsf}{\mathsf{Q}} \newcommand{\Rsf}{\mathsf{R}} \newcommand{\Ssf}{\mathsf{S}} \newcommand{\Tsf}{\mathsf{T}} \newcommand{\Usf}{\mathsf{U}} \newcommand{\Vsf}{\mathsf{V}} \newcommand{\Wsf}{\mathsf{W}} \newcommand{\Xsf}{\mathsf{X}} \newcommand{\Ysf}{\mathsf{Y}} \newcommand{\Zsf}{\mathsf{Z}} % mathbb letters \newcommand{\Abb}{\mathbb{A}} \newcommand{\Cbb}{\mathbb{C}} \newcommand{\Dbb}{\mathbb{D}} \newcommand{\Ebb}{\mathbb{E}} \newcommand{\Fbb}{\mathbb{F}} \newcommand{\Gbb}{\mathbb{G}} \newcommand{\Hbb}{\mathbb{H}} \newcommand{\Ibb}{\mathbb{I}} \newcommand{\Jbb}{\mathbb{J}} \newcommand{\Kbb}{\mathbb{K}} \newcommand{\Lbb}{\mathbb{L}} \newcommand{\Mbb}{\mathbb{M}} \newcommand{\Nbb}{\mathbb{N}} \newcommand{\Obb}{\mathbb{O}} \newcommand{\Pbb}{\mathbb{P}} \newcommand{\Qbb}{\mathbb{Q}} \newcommand{\Rbb}{\mathbb{R}} \newcommand{\Sbb}{\mathbb{S}} \newcommand{\Tbb}{\mathbb{T}} \newcommand{\Ubb}{\mathbb{U}} \newcommand{\Vbb}{\mathbb{V}} \newcommand{\Wbb}{\mathbb{W}} \newcommand{\Xbb}{\mathbb{X}} \newcommand{\Ybb}{\mathbb{Y}} \newcommand{\Zbb}{\mathbb{Z}} % mathcal letters \newcommand{\Acal}{\mathcal{A}} \newcommand{\Bcal}{\mathcal{B}} \newcommand{\Ccal}{\mathcal{C}} \newcommand{\Dcal}{\mathcal{D}} \newcommand{\Ecal}{\mathcal{E}} \newcommand{\Fcal}{\mathcal{F}} \newcommand{\Gcal}{\mathcal{G}} \newcommand{\Hcal}{\mathcal{H}} \newcommand{\Ical}{\mathcal{I}} \newcommand{\Jcal}{\mathcal{J}} \newcommand{\Kcal}{\mathcal{K}} \newcommand{\Lcal}{\mathcal{L}} \newcommand{\Mcal}{\mathcal{M}} \newcommand{\Ncal}{\mathcal{N}} \newcommand{\Ocal}{\mathcal{O}} \newcommand{\Pcal}{\mathcal{P}} \newcommand{\Qcal}{\mathcal{Q}} \newcommand{\Rcal}{\mathcal{R}} \newcommand{\Scal}{\mathcal{S}} \newcommand{\Tcal}{\mathcal{T}} \newcommand{\Ucal}{\mathcal{U}} \newcommand{\Vcal}{\mathcal{V}} \newcommand{\Wcal}{\mathcal{W}} \newcommand{\Xcal}{\mathcal{X}} \newcommand{\Ycal}{\mathcal{Y}} \newcommand{\Zcal}{\mathcal{Z}} % shorthands \newcommand{\constant}{\textnormal{constant}} % topology \newcommand{\scomp}[1]{#1^{\mathsf c}} % set complement \newcommand{\closure}[1]{\overline{#1}} \newcommand{\conv}{*} \newcommand{\circconv}{\circledast} \DeclareMathOperator{\affinehull}{aff} \DeclareMathOperator{\convexhull}{conv} \DeclareMathOperator{\conichull}{conic} \newcommand{\pdiv}[3][]{\operatorname{D}_{#1}\left(#2\:\vert\vert\:#3 \right)} \newcommand{\kldiv}[2]{\pdiv[\text{KL}]{#1}{#2}} \newcommand{\bigo}[1]{\mathcal{O}\left( #1 \right)} \newcommand{\littleo}[1]{o\left(#1 \right)} \newcommand{\LHS}{\operatorname{LHS}} \newcommand{\RHS}{\operatorname{RHS}} \newcommand{\reals}{\mathbb{R}} \newcommand{\posreals}{\reals^+} \newcommand{\nonnegreals}{\reals^{\geq 0}} \newcommand{\naturals}{\mathbb{N}} \newcommand{\integers}{\mathbb{Z}} \newcommand{\complex}{\mathbb{C}} % operators \newcommand{\textif}{\text{if}\ } \newcommand{\textelse}{\text{else}\ } %\DeclareMathOperator* is used for defining operators that have limits typeset beneath them instead of to the right (at least when in a display) \DeclareMathOperator*{\maximizeop}{maximize} \DeclareMathOperator*{\minimizeop}{minimize} \newcommand{\minimize}[1]{\minimizeop\limits_{#1}\ } \newcommand{\maximize}[1]{\maximizeop\limits_{#1}\ } \newcommand{\subjectto}{\text{subject to}\ } \DeclareMathOperator*{\argmax}{\arg\!\max} \DeclareMathOperator*{\argmin}{\arg\!\min} \DeclareMathOperator*{\argsup}{\arg\!\sup} \DeclareMathOperator*{\arginf}{\arg\!\inf} % use \command* to get autoscaling with paired delimiter commands % the unstarred version takes an optional argument that can be \big, \Big, etc % WARNING: Mathjax has a bug where there is an extra s in DeclarePairedDelimiter % See: https://github.com/mathjax/MathJax/issues/2758 \DeclarePairedDelimiters{\ceil}{\lceil}{\rceil} % requires mathtools package \DeclarePairedDelimiters{\floor}{\lfloor}{\rfloor} % requires mathtools package \newcommand{\inner}[3][]{\langle #2, #3 \rangle_{#1}} \DeclarePairedDelimiters{\abs}{\lvert}{\rvert}% \newcommand{\norm}[2][]{\lVert #2 \rVert_{#1}} \DeclarePairedDelimiters{\parens}{(}{)} \DeclarePairedDelimiters{\braks}{[}{]} \newcommand{\image}[1]{\operatorname{img}\parens*{#1}} \newcommand{\kernel}[1]{\operatorname{ker}\parens*{#1}} \newcommand{\diameter}[1]{\operatorname{diam}\parens*{#1}} \newcommand{\radius}[1]{\operatorname{radius}\parens*{#1}} \newcommand{\vecmat}[1]{\operatorname{vec}\parens*{#1}} \newcommand{\diag}[1]{\operatorname{diag}\parens*{#1}} \newcommand{\trace}[1]{\operatorname{tr}\parens*{#1}} % note: span is a latex primitive \newcommand{\vspan}[1]{\operatorname{span}\parens*{#1}} \newcommand{\rank}[1]{\operatorname{rank}\parens*{#1}} \newcommand{\sign}[1]{\operatorname{sign}\parens*{#1}} \newcommand{\tdef}{\triangleq} % middle bar, for use in delimited expressions like sets \newcommand{\mmid}{\ \middle| \ } %\newcommand{\set}[2][]{ % \ifx\\#1\\ % \{ #2 \} % \else % \{ #1 \mid #2 \} % \fi %} \newcommand{\onevec}{\mathbf{1}} \newcommand{\neginfty}{{-\infty}} % Positive semidefinite, positive definite cones \newcommand{\sym}[1]{\Sbb^{#1}} \newcommand{\psd}[1]{\Sbb^{#1}_+} \newcommand{\pd}[1]{\Sbb^{#1}_{++}} % up/down arrows for limits \newcommand{\dto}{\downarrow} \newcommand{\uto}{\uparrow} \newcommand{\T}{\top} \newcommand{\transpose}[1]{{#1}^\top} \newcommand{\inverse}{^{-1}} % unit ball \newcommand{\ball}{\mathbb{B}} % n-sphere \newcommand{\sphere}[1]{S^{#1 - 1}} % Manifolds \newcommand{\mnf}[1]{\manifold{#1}} %\newcommand{\manifold}[1]{\mathcal{\MakeUppercase{#1}}} \newcommand{\manifold}[1]{\mathcal{#1}} \newcommand{\Mmanifold}{\manifold{M}} \newcommand{\tangent}{\mathrm{T}} % Tangent space \newcommand{\cotangent}{\mathrm{T}^*} % Cotangent space \newcommand{\scalarfields}[1]{\mathfrak{F}(#1)} \newcommand{\vectorfields}[1]{\mathfrak{X}(#1)} % Probability % Random variables %\newcommand{\rv}[1]{\mathbf{\MakeUppercase{#1}}} \newcommand{\rv}[1]{\mathbf{#1}} \newcommand{\arv}{\rv{A}} \newcommand{\brv}{\rv{B}} \newcommand{\crv}{\rv{C}} \newcommand{\drv}{\rv{D}} \newcommand{\erv}{\rv{E}} \newcommand{\frv}{\rv{F}} \newcommand{\grv}{\rv{G}} \newcommand{\hrv}{\rv{H}} \newcommand{\irv}{\rv{I}} \newcommand{\jrv}{\rv{J}} \newcommand{\krv}{\rv{K}} \newcommand{\lrv}{\rv{L}} \newcommand{\mrv}{\rv{M}} \newcommand{\nrv}{\rv{N}} \newcommand{\orv}{\rv{O}} \newcommand{\prv}{\rv{P}} \newcommand{\qrv}{\rv{Q}} \newcommand{\rrv}{\rv{R}} \newcommand{\srv}{\rv{S}} \newcommand{\trv}{\rv{T}} \newcommand{\urv}{\rv{U}} \newcommand{\vrv}{\rv{V}} \newcommand{\wrv}{\rv{W}} \newcommand{\xrv}{\rv{X}} \newcommand{\yrv}{\rv{Y}} \newcommand{\zrv}{\rv{Z}} \newcommand{\xspace}{\Xcal} \newcommand{\yspace}{\Ycal} \newcommand{\zspace}{\Zcal} \newcommand\given[1][]{\:#1\vert\:} \newcommand{\simplex}{\Delta} \newcommand{\ev}[2][]{\mathbb{E}_{#1}\left[ #2\right]} \newcommand{\var}[2][]{\operatorname{Var}_{#1} \braks{#2}} \newcommand{\cov}[2]{\operatorname{Cov}\parens*{#1, #2}} \newcommand{\prob}[2][]{\mathbb{P}_{#1}\left(#2\right)} \newcommand{\ind}[1]{\unicode{x1D7D9} _{\left[ #1\right]}} % https://stackoverflow.com/questions/43886576/how-to-do-a-bold-1-mathbbm1-in-mathjax \DeclareMathOperator{\Binom}{Binom} \newcommand{\Normal}[1]{\operatorname{Normal}\parens*{#1}} \newcommand{\Bernoulli}[1]{\operatorname{Bernoulli}\parens*{#1}} \newcommand{\Binomial}[1]{\operatorname{Binomial}\parens*{#1}} \newcommand{\stdnormal}{\Normal{0, 1}} \newcommand{\stdmvnormal}{\Normal(0, I)} % convergence symbols \newcommand{\distto}{\overset{d}{\longrightarrow}} \newcommand{\asto}{\overset{a.s.}{\longrightarrow}} \newcommand{\probto}{\overset{\Pbb}{\longrightarrow}} \newcommand{\lpto}[1]{\overset{L^{#1}}{\longrightarrow}} \newcommand{\weakto}{\overset{\mathrm{weak}}{\longrightarrow}} \newcommand{\normto}{\overset{\mathrm{norm}}{\longrightarrow}} % convex analysis \newcommand{\gph}[1]{\operatorname{gph}\parens*{#1}} \newcommand{\epigraph}[1]{\operatorname{epi}\parens*{#1}} \newcommand{\domain}[1]{\operatorname{dom}\parens*{#1}} \newcommand{\intr}[1]{\mathrm{int}\left( #1 \right)} \newcommand{\clos}[1]{\mathrm{cl}\left( #1 \right)} % groups \newcommand{\orth}[1]{\mathsf{O}(#1)} \newcommand{\sorth}[1]{\mathsf{SO}(#1)} \newcommand{\sunit}[1]{\mathsf{SU}(#1)} \newcommand{\unit}[1]{\mathsf{U}(#1)} % derivatives \newcommand{\dd}[1][]{\mathrm{d}{#1}} % differential \newcommand{\diff}{\textnormal{d}} \newcommand{\deriv}{\textnormal{D}} \newcommand{\grad}{\nabla} \renewcommand{\dfrac}[2]{\frac{\dd #1}{\dd #2}} \newcommand{\pfrac}[2]{\frac{\partial #1}{\partial #2}} % matrices \newcommand{\pmat}[1]{\begin{pmatrix} #1 \end{pmatrix}} \newcommand{\bmat}[1]{\begin{bmatrix} #1 \end{bmatrix}} \newcommand{\Bmat}[1]{\begin{Bmatrix} #1 \end{Bmatrix}} \)

Regulation and the Emergent Superintelligence Hypothesis (Emergence Part 2)

by Helen Qu

In part 1, we discussed the relationship between individual agent incentives, the environment, and emergent system-level phenomena.

In this post we’ll conclude our discussion on emergence by understanding regulation, the process of negative feedback from the environment, which we’ll see is the key to the emergence of collective intelligence. This framing suggests that regulated systems of intelligent agents is a very real yet underexplored path to superintelligence, one with distinctive implications for AI safety.

The role of regulation

The Vicsek and Cucker-Smale models from part 1 demonstrated that the emergent outcome of a collective system can be controlled entirely by altering environmental parameters, such as the number density of agents.

However, any realistic environment changes over time as a result of agents’ actions: people avoid overcrowded bars when the line gets too long, food sources that were once abundant are exploited and become depleted. These simple models overlook an agent’s ability to not only respond to but effect change on their environment.

The mechanism of regulation describes how agents’ imprints on the environment connect their local experiences to the global state of the system. This quote from Collective Animal Behavior (Sumpter, 2010) [?] explains this with the example of coffee availability:

At any time during the working day, I can get up from my desk, walk down to the cafeteria, and find a container full of hot coffee from which I can pour myself a cup. The fact that the coffee is there waiting for me is not a consequence of careful preparation for my arrival by the cafeteria staff. … Regulation of supply and demand does not require central planning by me or anyone else. I do not have to call down to the café in advance and ask them to switch on the percolator; the cafeteria owner does not have to know when the next boat of coffee beans is coming from South America; and the shipping agent does not need to check that new plants are already in the ground for the next year’s crop. Through a series of local economic interactions I am provided with a regular supply of coffee.

While this example illustrates regulation as a method of maintaining equilibrium, regulation can create a variety of interesting outcomes depending on the properties of the system.

A brief taxonomy of regulated systems

To see why, we’ll characterize regulated systems along two axes: the type of data saved by the environment, and the nature of the agents’ relationship with that environment.

Environment data type

Every regulated system has an environmental state that agents read from and write to. The clearest example of this is price in economics: items increase in price in response to higher demand, which in turn may quell further demand. Analogous to the overcrowded bar example from earlier, both of these scenarios have a scalar environmental state. Ant colonies, on the other hand, deposit pheromone trails to preserve information about the location and abundance of food and other resources. This structured environmental state is naturally much more high-dimensional and informative than a single scalar.

Agent-environment relationship

How does the environment change over time as a function of agent actions? The effect of agent actions can be volatile or persistent. Price is volatile, and pheromone trails persist only for a short while in the absence of continuous reinforcement. On the other hand, shared ledgers in human society, such as Wikipedia, persist in its current state until a user makes an edit. Notably, most persistent environments are inherently selective, e.g., pheromone trails are reinforced and persist longer if the food source is still present; the corpus of scientific literature historically only preserves peer-reviewed, reproducible publications. Selective environments curate, which means the environment itself starts taking on a role in the system’s behavior.

Regulation and externalized cognition

In systems where environments selectively store structured data, the consequences of regulation are surprisingly profound. Ant colonies are well known to achieve a kind of collective intelligence far surpassing that of its individual agents, so much so that running ant colony simulations (“ant colony optimization”) is a viable algorithm for solving NP-hard problems like the Traveling Salesman Problem. Furthermore, organisms like slime molds, despite lacking a central nervous system, leverage slime trails to search for food sources so effectively that they have been used to design transit networks that are often more optimized and robust than those designed by engineers.

This phenomenon is summarized by the concept of stigmergy [?] and the Extended Mind Hypothesis [?], which postulates that cognition doesn’t stop at the brain or skull but instead can involve entities outside the agent’s body, such as its environment.

The emergence path to superintelligence

This perspective leads me to believe that current discourse has fixated on a single God-like superintelligence and overlooked a more likely path: higher intelligence emerging from a regulated, multi-agent system. Empirical evidence of emergent task-solving abilities is everywhere in the LLM world (e.g., in-context learning), and recent work directly demonstrates the acquisition of higher intelligence capabilities in multi-agent LLM systems [?]. We are already at a critical point where individual agents undeniably demonstrate intelligence in many arenas, much more so than ants. We might not even notice when the system as a whole becomes smarter than us, because we’re too focused on the ants instead of the hive.

This path to superintelligence necessitates a reframing of the AI safety problem. The alignment of individual agents may be neither necessary nor sufficient for the alignment of the collective they form – a system of well-behaved agents can produce pathological emergent behavior, and a system of imperfectly aligned agents can be steered toward good outcomes by the right environment. Thus, the central question is not how to align individual models, but how to design environments and incentives to produce safe, human-aligned emergent outcomes.

Our perspective thus far has come from studying emergence in practice via examples from physics and collective animal behavior. In the next post, we’ll turn to game theory for a framework that can illuminate an alternative path forward for AI safety.

References

    back to all posts