the governance stack (+ why ai can’t govern)

19 views

values gap

in late july of this year, abt 1.4k frontier lab employees signed a statement called "pacing the frontier." the statement asks the u.s. government to support an international effort to develop the technical/governance tools needed to deliberately slow down ai development if we ever decide we need to. note that it asks that the option to pause exist later, and isn’t actually asking everyone to pause now. the official reasoning is more or less that ai progress could suddenly accelerate, no individual actor will slow down unilaterally if everyone else keeps going, so we need to coordinate, yada yada yada. as of september, the statement has 1,386 signatures. if you look closely, the statement is basically asking for machinery. in practice, deliberately slowing the entire frontier would require some combination of monitoring, verification protocols, inspectors, agreements between countries, and institutions capable of actually enforcing whatever gets agreed on. that’s an execution problem. the decision of when to pull that lever, how fast we should go, and who gets to bear which risks is fundamentally a values problem which verification tools will never be able to answer.

this i think highlights a massive gap in current ai governance. no amt of monitoring capacity can tell us how much risk society should accept. the letter is effectively a plea to build the execution machinery now, partly so we have the option to act while we figure out a values problem we haven't really even started on.

the 3 sub problems/layers for this

preference - what outcomes should count as good? should another $100 billion go to healthcare, research, housing, defense, or tax reduction? how much privacy is security worth? how should a small chance of catastrophe weigh against large ordinary benefits? a model can’t just infer these, because they are disagreements about the social objective.

the epistemic problem - given an objective, which actions achieve it? this is where expertise lives, and where people who share values can still disagree hard about model capabilities, timelines, cyber risk, or whether a given rule is technically enforceable.

execution - can the state actually do the thing? can it hire, procure, measure, coordinate across jurisdictions, and find out whether the program is working?

this stack looks a little like

a diagram of the governance stack showing citizen values flowing into social objectives, then causal beliefs and policy, implementation, and outcomes, with a feedback loop from outcomes to citizen values.

this decomposition is not original to me btw. it pulls together fritz scharpf and vivien schmidt’s work on democratic legitimacy, especially the distinction between citizen input, the processes of governance, and effective outputs; decades of public administration research, from pressman and wildavsky in the '70s to jennifer pahlka today, on why execution fails; and social choice theory, like arrow’s theorem, on what happens when you try to turn individual preferences into a collective one. schmidt’s version explicitly adds “throughput” to the older input/output framework, meaning the quality of the governance process sitting between public participation and policy results. the problem is that technologists building systems across all three layers rarely read these different fields together, so i’m hoping this distillation fills that gap for anyone interested. of course, the boundaries here are messy. in reality, it's a constant feedback loop of implementation limiting what’s feasible, experts shaping public beliefs, then citizens changing their minds once they see the actual consequences.

why certain good ideas for reforms aren’t enough

the value of breaking government down into these layers is that it tells us what a given reform can(‘t) fix, because it usually only covers one layer or another. a few examples of that here -

pay federal tech people more! this is a pretty reasonable idea. think about it - the stats in 2026 say that federal senior-level and scientific or professional positions top out at $228,000 in agencies with certified performance appraisal systems, and $209,600 in agencies without them. meanwhile, plenty of frontier lab research roles start above that before equity. openai currently lists research roles with base salary ranges starting at $250k or $295k depending on the job. so why would someone take a government role over a cushy oai salary? raising compensation significantly expands the pool of cracked candidates, and ppl have actually shown this works. ferraz and finan exploited salary discontinuities across brazilian municipalities and found that higher legislative pay increased political competition and improved observable legislator characteristics like education, prior profession, and political experience. the closer experiment for the technical layer is dal bó, finan and rossi, who randomized wage offers for mexican public-sector positions and found higher offers attracted more able applicants without evidence that they crowded out public-service motivation.

so this works!! the annoying part is that it tells us way less about what happens to people after they arrive, which is an execution problem. we can call the gap between someone's market compensation and what they accept to enter government the public-service tax. paying it down widens the funnel. procurement can still suck, data can still be inaccessible, and a program can still have no idea whether it’s succeeding. and ultimately, compensation is just one variable in a worker's utility function:

U(job)=f(compensation, mission, agency, resources, colleagues, speed, prestige, options)U(\text{job}) = f(\text{compensation},\ \text{mission},\ \text{agency},\ \text{resources},\ \text{colleagues},\ \text{speed},\ \text{prestige},\ \text{options})

a frontier researcher weighing government is also weighing compute, model access, execution speed and whether decisions have consequences.

expand caisi! (nist's center for ai standards and innovation). by may 2026, caisi had signed new pre-deployment evaluation agreements with google deepmind, microsoft and xai, building on its existing relationships with openai and anthropic. reporting around the announcement said it had already completed more than 40 evaluations, including evaluations of models that had not yet been publicly released. this is basically epistemic-layer capacity with some execution machinery attached. it can answer questions like "can this model do capability c" and "does safeguard s survive attack a." i think it is one of the clearest examples of actual operational technical capacity in u.s. ai governance right now. it still has nothing inherent to say about how much risk society should tolerate, and it wasn’t designed to answer that question anyway.

the pacing letter itself... as i mentioned, it’s asking for execution capacity in service of a preference/value decision that we have not actually made yet. each of these is pretty compelling within its own part of the problem. none does much to make the other problems disappear, which leaves us in an odd position where reform discourse feels simultaneously compelling and somehow still not enough.

what ai actually changes

ok so starting on layer 1, with the values, there are actually two problems hiding in here. first you need to know what people want. then you have to decide how millions of conflicting wants turn into some kind of social objective. we know democracy struggles with the first one purely as an information problem. citizens have complex views on thousands of issues, like taxes, healthcare, ai, climate change, whatever, and a presidential election forces all of that into a single vote. surveys help a little, but they are often expensive, lack depth, and struggle with conditional opinions like "i support x if y happens but absolutely not if z happens." so an obvious suggestion is to train a model to learn an approximation of each citizen's preferences and ask ppl only when the model is uncertain - and i was rather surprised to learn we already do something surprisingly close to this. google deepmind's "habermas machine" used an llm-mediated system to generate group statements from participants’ opinions and critiques. across 5,734 participants, people preferred its consensus statements to those written by human mediators and rated them as more informative, clearer and less biased. participants also sometimes shifted toward more shared positions after deliberating through the system.

polis comes at it differently. people submit statements, other people vote agree/disagree/pass, and the system clusters participants based on similarities in their voting patterns. that lets you see where different groups exist, what divides them, and which statements get agreement across groups. so preference bandwidth is becoming a solved-ish engineering problem. at minimum, it looks way more tractable than it did when democratic institutions were designed around occasionally asking millions of people to choose from a handful of names on a piece of paper. but that only handles the first half of layer 1.

suppose some godlike preference model gave us an extremely good estimate of every individual's utility,

U1(x),…,UN(x).U_1(x), \ldots, U_N(x).

we would still have to decide how those get turned into a social objective. you might write that abstractly as

W(x)=F(U1(x),…,UN(x))W(x) = F\big(U_1(x), \ldots, U_N(x)\big)

where different choices of FF could care about total welfare, give extra importance to the worst-off, care about preference intensity, or embody some completely different principle. there is no technical result sitting inside the preference data that tells us which FF society ought to use. social choice theory basically formalizes why aggregation rules have tradeoffs. arrow’s theorem is specifically about aggregating ordinal rankings rather than cardinal utility functions like the toy equation above, and shows that with at least three alternatives, no aggregation rule can simultaneously satisfy a familiar set of desirable conditions. the equation is just an intuitive way of seeing the broader issue here. once individual preferences have been measured, a normative choice is still left over. this becomes very concrete in systems like the habermas machine. the system designers have to decide what kinds of agreement the model should optimize for and how minority and majority views should enter that objective. the ai can make the machinery for eliciting and combining views ridiculously more sophisticated. someone still has to decide what the machinery is optimizing. so i’d split layer 1 mentally into preference elicitation and preference aggregation. ai looks extremely useful for the former, but the latter is still political...

meanwhile layer 2, once there is some objective, you have the problem of figuring out which actions are likely to achieve it. ai can synthesize giant literatures, model systems, search through policy options, forecast consequences, and eventually maybe do this much better than individual human experts. its claims still need to survive empirical checking, and this layer is going to stay full of disagreements because the world is annoying and causal inference is hard. there’s also a feedback loop with layer 1. if i think policy x will make me much richer, i may support it; if new evidence convinces me it will make me poorer, suddenly my expressed preference changes. values and empirical beliefs get mixed together constantly in real politics even if it’s analytically useful to separate them. that messiness doesn’t really hurt the stack imo. it is kind of the point. if we know which type of disagreement we are having, we can at least tell whether adding more compute, better polling, another economist, or a more competent civil service has any chance of solving it.

government has way weaker pre-deployment evaluation

hot take(?) but government has something pretty important to learn from ai safety. ai safety evaluation is built around this idea that you shouldn't infer how a system will behave purely from what its builder says about it. in the frontier-model world, this has led to stuff like testing systems before full deployment, deliberately searching for failure modes, and increasingly giving external evaluators access. there’s a tempting comparison to government programs here, although my original version of this argument was too strong. government absolutely does do pre-deployment testing in some contexts. federal digital-service guidance explicitly talks about prototyping and usability testing, and dhs has even published resources for testing government websites, forms and apps with actual users before launch. what is much weaker is the independent adversarial institution around it.

take the three ideas separately -

  • pre-deployment. frontier evaluators sometimes get access to systems before an irreversible release. a nice recent example is kimi k3, which the uk aisi and u.s. caisi jointly evaluated for cyber capabilities before its planned open-weight release. government services can be prototyped and piloted too, obviously. there just isn’t a standard expectation that a major new public-facing system gets subjected to an independent technical evaluation specifically designed to uncover catastrophic edge cases before it touches millions of ppl.

  • adversarial testing. ai red-teamers actively try to make a model misbehave. i want someone to do the equivalent to a government benefits portal. give a team an explicit mandate to find the cursed cases, like a claimant with an expired id, no fixed address, a name mismatch across databases, and a phone that can't get texts. normal usability testing can catch a bunch of this already. an adversarial team would be paid to keep going until they find the combinations nobody designing the happy path thought about.

  • third-party evaluation. ai evaluation increasingly involves institutes outside the model developer because self-assessment has obvious limits. government does have external oversight through gao, inspectors general, courts, congress, etc., so saying agencies literally grade themselves would be unfair. but the evidence act's specific capacity-assessment process does require agencies to assess their own capacity for statistics, evaluation, research and analysis. gao reviewed 23 agencies and found that 18 reported difficulty interpreting the guidance or carrying out those assessments. that is a pretty good example of why internal assessment alone can get weird.

tbf, there are some huge differences between ai and government systems. you can safely copy and attack an ai model a million times in a sandbox, while you can’t perfectly simulate a benefits system embedded in laws, databases, contractors, case workers, and actual humans. some ai-safety evals also have unusually operational targets like "can the model autonomously exploit this machine," while a government program can have multiple goals that people disagree about in the first place. and frontier labs now have a whole ecosystem of reputational pressure, voluntary agreements, researchers and emerging regulation pushing them toward outside evaluation. there isn’t an equivalent norm applied consistently to public programs. because of those differences, copying individual ai eval techniques literally seems kind of dumb. the institutional structure is the part i care about. my version would be small independent technical units with privileged access to a program before launch and an explicit mandate to break it. they would be organizationally separate enough from the program owner that finding a giant problem is considered a successful evaluation instead of an annoying obstacle to shipping. you could run one against a benefits portal, an immigration system, a procurement platform, whatever.

maybe the best way to describe this is pre-deployment red teaming for government. i have no idea if it would always be cheap once you actually build it lol, but finding that 8% of legitimate users get locked out before the system is national seems dramatically preferable to learning the same fact from headlines six months later.

why shouldn’t ai just govern?

it’s kind of tempting, at least among the sf tech bros, to look at all of this and ask: if ai can model preferences, forecast policy outcomes, and increasingly run bureaucratic processes, why not eventually just ask it to optimize government? i think the stack makes that framing kind of confused. at the preference layer, an ai system might eventually be able to maintain an insanely high-resolution picture of what millions of people want. that still leaves questions like whose preferences get standing, how rights constrain majority preferences, what happens to future generations who cannot answer a survey, and how competing preferences get aggregated.

then once some objective exists, ai could be enormously useful for the epistemic layer. maybe eventually you hand it all the administrative data and research literature and it gives vastly better forecasts than the humans currently writing policy memos. great. the claims still need to be inspectable and testable because policy failure has consequences outside the benchmark.

then there is execution, where ai could plausibly automate a huge fraction of bureaucracy. even there, authority and recourse have to live somewhere. if an automated system denies someone benefits, deports the wrong person, or makes some extremely weird procurement decision, saying "the model made the call" is not a governance structure. somebody or some institution has to own the decision and provide a way to contest it.

another structural concern here is that the freshest ai expertise lives inside frontier labs, so bringing that talent into government is obviously valuable. a state that imports a huge share of its expertise from the same few labs is also going to import some of their assumptions and blind spots along with the competence. that’s another reason technical expertise works better as an input into political decision-making than as the thing that defines the objective. ultimately, if we don’t want to live in some cursed dystopia, i don’t think the goal is to have ai discover society's objective function and then execute against it forever. there probably isn’t one sitting out there waiting to be discovered. the useful goal is to raise the bandwidth and competence of the whole stack while keeping the functions institutionally distinguishable enough that, when something fails, we can figure out what failed. did we choose an objective people reject? did we have the objective right and misunderstand the consequences of the policy? did we know what we wanted and how to achieve it, then completely butcher implementation?

those are very different failures, even though from the outside they can all just look like "government bad." frontier ai makes rethinking government urgent because it puts pressure on every one of these layers at the same time. weirdly, it also gives us some tools for improving them. preference modeling gives us much higher-bandwidth ways of hearing from people. better models could improve the epistemic machinery. ai safety has also spent the past few years building evaluation institutions around the assumption that important systems should be attacked before we trust them.

none of this tells us what society should want. which is, unfortunately, still the politics part.

references