helen qu

Mechanism Design for AI Safety (Emergence Part 3)

Thu, 04 Jun 2026 00:00:00 +0000

In previous posts, we’ve seen how complex emergent behavior can arise in multi-agent systems, and how distributed cognition can emerge from environmental feedback. This led me to conclude that superintelligence is arguably more likely to emerge from a multi-agent system as opposed to in a single God-like superintelligent model, what I called the emergent superintelligence hypothesis. Among other things, this has important implications for the AI safety landscape, namely placing more emphasis on designing for alignment of the collective as opposed to individual agents.

Regulation and the Emergent Superintelligence Hypothesis (Emergence Part 2)

Wed, 20 May 2026 00:00:00 +0000

In part 1, we discussed the relationship between individual agent incentives, the environment, and emergent system-level phenomena.

In this post we’ll conclude our discussion on emergence by understanding regulation, the process of negative feedback from the environment, which we’ll see is the key to the emergence of collective intelligence. This framing suggests that regulated systems of intelligent agents is a very real yet underexplored path to superintelligence, one with distinctive implications for AI safety.

An Emergence Perspective on Multi-Agent System Design (Part 1)

Thu, 16 Apr 2026 00:00:00 +0000

We observe emergent behavior in stunningly diverse parts of our world, ranging from collective animal behavior to economic systems. Emergence (as defined by Wikipedia) “occurs when a complex entity has properties or behaviors that its parts do not have on their own, and emerge only when they interact in a wider whole”. In other words, emergent (or system-level) outcomes are more than (and different from) the sum of their parts [?]. A wide range of system-level outcomes can emerge from a collection of agents following their own, often simple, reward/value functions.

RL's Deadly Triad Meets Optimization

Tue, 23 Dec 2025 00:00:00 +0000

\( \newcommand{\phit}{\phi_t} \newcommand{\phinext}{\phi_{t+1}} \newcommand{\enext}{\ebf_{s_{t+1}}} \newcommand{\et}{\ebf_{s_t}} \newcommand{\gradV}{\nabla_{\theta} V_{\theta}} \newcommand{\V}{V_{\theta}} \newcommand{\snext}{s_{t+1}} \newcommand{\tstar}{\theta^\star} \newcommand{\itmtx}{(\Ibf - \eta \Abf)} \newcommand{\AMC}{\Abf_{\text{MC}}} \newcommand{\Aon}{\Abf_{\text{on}}} \newcommand{\Atab}{\Abf_{\text{tab}}} \newcommand{\dpi}{d_{\pi}} \newcommand{\Ppi}{P_{\pi}} \newcommand{\Tpi}{\Tcal_{\pi}} \)

The unsavory combination of function approximation, off-policy learning, and bootstrapping, or the so-called “deadly triad” of reinforcement learning, has a status approaching that of folklore for its ability to induce unstable learning dynamics/divergence in even the simplest of systems. While likely a household name among RL enthusiasts, I’ve seen surprisingly little discussion on its origins or first principles. I made this post to shed light on the deadly triad from the perspective of traditional optimization theory, with the goal of demonstrating that the underlying principles are nothing more exotic than well-established convergence guarantees for dynamical systems.