System-level capability amplification and dynamic model selection for AIOS

Abstract

Model capability is an important determinant of AI performance, but it is not the only determinant of useful output. For a deployed reasoning system, the relevant object is the joint system: model, task contract, context selection, decomposition, working state, durable memory, provenance, planning, tools, verification, interface, and integration into subsequent work. Evidence published from August 2024 through August 2026 supports a bounded version of this claim. Carefully chosen scaffolds can produce large gains on tasks whose weak operations can be externalized, checked, or made more legible. In several bounded settings, a smaller model with task-specific tools or an optimized workflow matches or exceeds a larger model operating through a weak harness. The evidence does not support the universal claim that scaffolding makes model capability interchangeable, nor that many individually plausible interventions compound monotonically.

Dynamic model selection is also technically credible. Model routers and cascades can reduce inference cost while retaining most measured quality on in-distribution tasks, and contemporary model results are sufficiently jagged that a single global tier ordering is misleading. But routing gains depend on calibration, task observability, endpoint configuration, model complementarity, and a reliable escalation path. Recent large router comparisons find that sophisticated methods frequently fail to beat simple baselines and remain far below an oracle, especially under model churn and imperfect recall of the best candidate.

The defensible AIOS thesis is therefore not “small models replace frontier models.” It is that a model-plural, locally governed reasoning system can allocate frontier capability selectively, externalize recurring work into inspectable structure, and optimize for accepted outcomes rather than benchmark accuracy alone. That proposition remains a system hypothesis until it survives factorial ablation, compute-matched comparison, distribution shift, longitudinal reuse, and safety testing. This memo specifies such a test program.

Research scope and evidence discipline

This review addresses four linked questions:

1. How much performance is attributable to the underlying model, and how much to the harness and environment?
2. Under what conditions can scaffolding substitute for model scale on a bounded cognitive movement?
3. Can a system route movements among models according to capability, privacy, latency, cost, and risk without silently degrading outcomes?
4. Can verified structure accumulated over time reduce the marginal inference and review burden of future work?

The review prioritizes peer-reviewed primary research and independent or standards-setting benchmark reports published in the primary evidence window. Preprints and provider evaluations are included only where they address rapidly evolving questions not yet covered by mature independent evidence, and are labeled. Sources before August 2024 appear only in the foundational-lineage section.

Evidence labels used below are descriptive rather than scores:

• Peer reviewed: archival conference, journal, or findings paper.
• Independent benchmark/report: an independent research institute or standards consortium.
• Preprint: not treated as settled evidence.
• Provider evidence: useful for current product facts or hypotheses, but not an independent performance estimate.

Cross-paper point estimates should not be read as a leaderboard. Model versions, prompts, reasoning budgets, tool interfaces, endpoints, judges, and contamination controls differ. The most reliable comparisons are within-study, same-task contrasts.

1. Current evidence

1.1 There is no single “model gap”

Current benchmark gaps are simultaneously large, small, and occasionally reversed, depending on the task and evaluation harness.

The Stanford AI Index 2026 technical-performance review reports rapid frontier movement alongside a compressed top field. Its March 2026 Arena snapshot placed the leading systems within roughly 80 Elo points, while its broader review recorded a roughly 30-percentage-point one-year gain on Humanity’s Last Exam. Yet capability remained jagged: a leading system achieved 66.3% on OSWorld, still failing approximately one third of computer-use tasks, and model performance on analog-clock reading remained far below the cited human level. AI Index also estimates that invalid or defective items range from about 2% on MMLU mathematics to 42% on GSM8K under the audit it summarizes. A benchmark score is thus a mixture of model competence, benchmark validity, protocol, and sometimes judge behavior.

The 2025 AI Index documents a different compression: the smallest model exceeding 60% on MMLU fell from 540 billion parameters in 2022 to 3.8 billion in 2024, while the estimated inference price for GPT-3.5-level MMLU performance fell from about $20 to $0.07 per million tokens between November 2022 and October 2024. At the same time, its RE-Bench summary shows that agents can dominate human experts at two-hour horizons but lose by approximately two to one at 32 hours. Scale comparisons therefore depend on both task type and horizon.

Independent current composite benchmarks make their own value choices. Artificial Analysis Intelligence Index v4.1 reweighted its index toward agentic work, removed a saturated benchmark, increased allowed trajectory length, and assigned substantial weight to GDPval, terminal, and tool-use evaluations. The revision is informative precisely because rankings are index-dependent. A model optimized for short mathematical answers, professional deliverables, or long tool trajectories can move markedly when the mixture changes. The Endpoint Accuracy Index goes further: the same nominal open model can produce materially different benchmark accuracy across providers because of quantization, output limits, tool formatting, and serving configuration. That result turns “model” into an underspecified experimental variable unless endpoint and decoding details are pinned.

Contemporary families also show within-family jaggedness. OpenAI’s provider-reported GPT-5.6 results put its Sol, Terra, and Luna variants only 2.4 points apart on Agents’ Last Exam, but 17 points apart on Big Finance and 7.7 points apart on the Artificial Analysis index. OpenAI’s open-model report gives a similarly non-monotonic profile: gpt-oss-20b is reported near or above o3 on AIME 2025, but 11.8 points behind it on GPQA and 8.1 points behind on MMLU. Google’s provider-reported Gemma 3 results range from 14.7 on MMLU-Pro for 1B to 67.5 for 27B, a 52.8-point within-family difference. Apple’s 2025 foundation-model report describes an approximately 3B-parameter on-device model that is competitive with comparably sized open models, not with the most capable hosted systems across the board.

These results justify four operational conclusions:

• “Frontier,” “balanced,” “small,” and “on-device” are deployment classes, not stable capability orderings.
• Average benchmark rank is an inadequate routing target. Per-task success, calibration, tail risk, and endpoint behavior matter.
• A reported model gap does not identify how much of the gap is intrinsic capability versus inference budget, interface, retrieval, or judge.
• Local models are credible for some bounded movements, but cross-domain reversals do not erase large gaps on difficult scientific, financial, multilingual, visual, or open-ended tasks.

1.2 Scaffolding can substitute for scale when the bottleneck is externalizable

The strongest evidence for system-level amplification comes from bounded tasks with an executable oracle, specialized tool, or learnable scoring function.

Test-time compute

Snell et al., ICLR 2025 ask how inference compute should be allocated between repeated sampling and sequential revision. On mathematical reasoning tasks, adaptive compute allocation is reported to be more than four times as efficient as a best-of-N baseline, and in regions where a smaller base model has non-trivial success probability, additional test-time compute lets it outperform a model approximately 14 times larger under the paper’s comparison. This is strong evidence for scale substitution, but only inside the study’s conditions: mathematical problems, a useful process reward model, and matched compute. When a base model lacks a viable solution path, inference scaling cannot create one reliably.

Setlur et al., ICML 2025 compare verification-based reinforcement learning and search with verifier-free imitation of long reasoning traces across 3B, 8B, and 32B models. Their asymptotic result favors methods grounded in verifiable outcomes. For AIOS, the implication is not “make every movement think longer,” but “spend additional inference where feedback is discriminative.” The available evidence is strongest for mathematics and code, not for ambiguous judgment.

The much-discussed s1 preprint reports that supervised fine-tuning a 32B model on 1,000 selected reasoning traces plus “budget forcing” produced large gains on AIME and MATH. A later preprint, It’s Not That Simple, finds that appending “Wait” can induce oscillation, that much of the aggregate curve comes from truncating bad trajectories, and that distilled long chains do not yield a significant general benefit in its tests. Together these results warn against treating visible reasoning length as useful compute. The operative variable is productive search constrained by feedback, not token count.

Exact tools and task-specific interfaces

A peer-reviewed clinical study provides one of the clearest within-study model-versus-system contrasts. Pillai et al., npj Digital Medicine, 2025 ran 10,000 trials of clinical calculations using Llama 3.1 70B and GPT-4o under base, code, retrieval, retrieval-plus-code, and specialized OpenMedCalc conditions. Llama’s accuracy rose from 11% unassisted to 53% with generic tools and 84% with the specialized calculator; GPT-4o rose from 36% unassisted to 82% with retrieval plus code and 95% with OpenMedCalc. The smaller open model with a task-specific tool therefore exceeded the larger proprietary model without one. The result is narrow but consequential: exact computation, formula selection, and structured inputs are operations worth externalizing. The remaining 5% error at the best condition also matters. Tools reduced arithmetic and formula errors but did not eliminate interpretation errors.

SWE-agent, NeurIPS 2024 holds the language model relatively fixed while changing the agent-computer interface for repository repair. Its purpose-built interface and trajectory design reached 12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix; the paper’s interface ablations show a large improvement over a generic shell. The absolute SWE-bench success rate is dated by later systems, but the causal lesson survives: action space, observations, and feedback are part of measured intelligence.

Current provider evidence points in the same direction but should be treated as a hypothesis generator. OpenAI reports in its GPT-5.6 launch analysis that programmatic tool calling reduced total tokens by 63.5% and model turns by 50.1% in one partner workflow while retaining comparable scene-level results; another partner reported fewer steps, tool calls, and stuck runs. These are not independent evaluations and may reflect favorable production cases. They nevertheless motivate a direct AIOS comparison between natural-language tool choreography and code-like, typed orchestration.

Workflow search and decomposition

AFlow, ICLR 2025 represents an agent workflow as code and uses Monte Carlo tree search to optimize it across six benchmarks. The authors report a 5.7% average improvement over prior workflows and cases in which smaller models surpass GPT-4o at 4.55% of its inference dollar cost. This is direct evidence that workflow structure can dominate model tier on selected tasks. It is not evidence that automatic workflow search generalizes without cost: the optimization budget, benchmark overfitting, and static task distribution are important limitations.

Automated Design of Agentic Systems, ICLR 2025 similarly searches over agent designs and reports cross-domain and cross-model transfer. The key question for AIOS is whether the learned structure represents a reusable task invariant or a benchmark-specific policy. Only hidden, later-arriving tasks can distinguish the two.

1.3 Context quality is not context quantity

Retrieval and memory help when they increase the probability that relevant, trustworthy evidence appears in a usable position. They hurt when they add distractors, stale claims, adversarial content, or false confidence.

Li et al., EMNLP Industry 2024 compare long-context reading and retrieval-augmented generation across public question-answering tasks. Fully resourced long context consistently outperforms their RAG configurations, while RAG is substantially cheaper. A self-routing policy retains performance comparable to long context with lower compute by escalating only selected questions. This is an early direct demonstration of joint context and model routing, but it uses three models and public benchmarks; its self-reflection signal should not be assumed calibrated in production.

LongMemEval, ICLR 2025 tests information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention over histories up to 115,000 tokens. Models decline by 30–60% relative to oracle evidence, while online memory products in the study underperform offline long-context reading by 37–64%. Session decomposition, fact-oriented keys, and time-aware querying improve results. Yet one optimization—condensing history into user facts—helps multi-session retrieval while damaging other abilities. Memory design therefore moves bottlenecks rather than adding a uniform increment.

The negative result is stronger in Context Length Alone Hurts LLM Performance Despite Perfect Retrieval, Findings of EMNLP 2025. Across five models and mathematics, QA, and coding, longer inputs degrade performance by 13.9–85% even when the relevant information is perfectly retrievable; irrelevant whitespace and masked tokens can also hurt. Repeating the evidence immediately before solving recovers some performance. For AIOS, context composition should be treated as an active control problem with a token and attention budget, not as archival completeness.

HippoRAG 2, ICML 2025 combines graph structure, personalized PageRank, and deeper passage processing. It reports a 7% improvement over a strong embedding baseline on associative retrieval. Its analysis also exposes a relevant interaction: graph retrieval can improve multi-hop sensemaking while reducing basic factual retrieval relative to standard RAG. Relational metadata is valuable only when it is connected to a retrieval policy that preserves source text and can fall back to simpler search.

Query sensitivity remains a systems risk. A GEM workshop study runs 1,092 experiments and finds that small query variations can significantly degrade retrieval. The venue and task scope limit generalization, but the result supports query ensembles, retrieval diagnostics, and evidence sufficiency checks rather than a single opaque lookup.

1.4 Provenance improves auditability, not truth by itself

Recent work separates three concepts that deployed systems often conflate:

• Factuality: is the claim supported by the world or an authoritative source?
• Faithfulness: does the response follow the retrieved evidence, even if that evidence is wrong?
• Attribution: can a reviewer identify which source is asserted to support which claim?

FRANQ, Findings of ACL 2026 argues that RAG evaluations commonly conflate factuality and faithfulness and introduces an annotated benchmark to separate them. GaRAGe, Findings of ACL 2025 evaluates 2,366 questions and more than 35,000 annotated passages; its reported leading systems remain weak on relevance-aware factuality, deflection, and attribution, especially for time-sensitive and private information. Retrieval can therefore make an answer look grounded while propagating a bad premise or stale document.

ReClaim, Findings of NAACL 2025 reports approximately 90% citation accuracy using an interleaved reference-and-claim generation scheme. That is evidence that output structure can improve claim-source alignment. It does not show that the cited source is authoritative or that the claim is correct.

Even truthful metadata can bias generation. An authorship-metadata study, Findings of ACL 2025 uses counterfactual source labels across three language models and observes 3–18% changes in attribution quality, including preference for content explicitly described as human-authored. Relational and source metadata should thus be available for policy and audit, while the system should test whether exposing it to the generator creates unearned authority effects.

1.5 Critique, verification, and multiple agents have strong interaction effects

“Review” is not one intervention. It may mean self-revision without new evidence, independent generation and comparison, execution against a test, source checking, or expert adjudication. These mechanisms have different epistemic value.

Yang et al., ACL 2025 decompose self-correction into confidence—the ability to preserve a correct answer—and critique—the ability to repair an incorrect one. Across their evaluations, interventions that improve one can damage the other. Naive revision can lower final accuracy by changing initially correct answers. Premise critique, Findings of EMNLP 2025 similarly finds that many models need explicit instructions to question flawed premises and that raw reasoning ability does not consistently predict premise detection. A reviewer with the same model, same context, and same blind spot is not independent verification.

The corrective in-context learning study at the 2025 Insights workshop finds that filling a prompt with earlier mistakes and their corrections can underperform standard in-context learning, with degradation as corrections accumulate. Durable correction memory needs distillation, scope, validity intervals, and conflict resolution—not indiscriminate replay.

The broadest recent multi-agent scaling study is still a preprint. Towards a Science of Scaling Agent Systems evaluates 180 configurations across four benchmarks, five architectures, and three model families with fixed tools and token budgets. It reports an 80.9% gain for centralized coordination on one parallel financial task but 39–70% degradation for multi-agent designs on sequential tasks. Independent agents amplify error much more than centralized systems, and gains tend to vanish when single-agent accuracy exceeds roughly 45%. The exact thresholds require replication, but the architecture-task interaction is the important result. Agent count is not a monotone capability control.

Two 2026 preprints reinforce the need for diversity and protocol testing. Diversity Is All You Need reports that two deliberately diverse agents can match or exceed sixteen homogeneous agents because correlated errors cause early saturation. More Capable, Less Cooperative? reports cases where a stronger individual model reaches a worse collective equilibrium than a smaller one, with explicit protocols partly recovering cooperation. These are provisional results, but they undermine any design assumption that frontier participants and more parallelism automatically produce the best collective outcome.

1.6 Tool use expands both capability and attack surface

Agent benchmarks increasingly measure policy compliance, state tracking, and consistency rather than final-answer fluency.

τ-bench, ICLR 2025 evaluates agents interacting with users and tools under domain policies. Its leading reported systems complete fewer than half the tasks, and repeated-run reliability measured by pass^8 is far lower than single-run success in the retail domain. ToolSandbox, Findings of NAACL 2025 adds state dependencies, insufficient-information cases, canonicalization, and intermediate milestones; stateful dependencies remain difficult even for strong models. Both support measuring trajectory validity and consistency, not merely plausible final text.

The security evidence is equally material. AgentDojo, NeurIPS 2024 combines 97 realistic tasks with 629 prompt-injection security cases and finds both attacks and defenses challenging. MINJA, NeurIPS 2025 shows that an attacker can plant malicious records in agent memory through ordinary queries and have them retrieved later. MEXTRA, ACL 2025 demonstrates black-box extraction of private interactions stored by representative memory-enabled agents. More memory and more tools raise the value of the system while increasing the duration and blast radius of an error.

1.7 Dynamic model routing works, but robust routing remains unsolved

The routing literature supports selective model use under controlled conditions.

RouteLLM, ICLR 2025 learns routers from human preference data between a strong and weak model. The authors report more than a twofold cost reduction at matched quality and transfer to new model pairs. The savings are benchmark-specific, the setup is primarily two-model, and preferences are an imperfect proxy for accepted outcomes.

BEST-Route, ICML 2025 jointly selects a model and a number of samples based on task difficulty and target quality, reporting up to 60% cost reduction for less than a 1% performance drop. Its advantage depends on a reliable evaluator for candidate outputs—exactly the component that is often missing in high-stakes or open-ended work.

IRT-Router, ACL 2025 models both query difficulty and model capability using item-response theory over 20 models and 12 datasets, providing an interpretable cold-start mechanism. Self-REF, ICML 2025 trains explicit confidence tokens that outperform verbal confidence and raw likelihood for routing and rejection in its tests. These studies support calibrated, task-conditional escalation rather than asking a model, in ordinary prose, whether it feels confident.

The strongest caution comes from LLMRouterBench, Findings of ACL 2026, which aggregates more than 400,000 instances across 21 datasets, 33 models, and 10 router baselines. It confirms model complementarity, but finds many routing methods statistically similar, several recent or commercial methods unable to beat a simple baseline, and a large gap to oracle routing driven partly by failure to retrieve the best model candidate. Adding models yields diminishing returns unless the portfolio is curated. Dynamic routing is therefore a model-selection and a portfolio-design problem.

1.8 Efficiency metrics must be system metrics

Tokens and provider price are useful but incomplete. Model size, hardware utilization, quantization, batching, input/output mix, speculative decoding, and retry behavior can reverse an efficiency comparison.

MLPerf Inference v6.0 is an authoritative reference design for architecture-neutral, reproducible system measurement. Its 2026 suite adds gpt-oss-120B and an interactive DeepSeek-R1 scenario and explicitly measures deployed stacks rather than abstract model FLOPs. MLPerf Client v1.0 extends standardized testing to client devices. Neither benchmark measures accepted knowledge-work outcomes, but both show how endpoint, latency, throughput, and energy controls should be reported.

Early energy studies remain preprints. TokenPowerBench proposes separating prefill and decode energy and reporting joules per token across 1B–405B models. A 2026 inference-energy diagnostic reports up to 25-fold variation by task and three- to fivefold effects from utilization across a large configuration grid. These estimates need independent replication, but they make one point clear: “use the smaller model” is not itself an energy measurement.

2. Major-source evidence analyses

The table below states the evidentiary role of the most important sources rather than repeating every source cited in the narrative.

3. The AIOS thesis

3.1 The unit of capability is the reasoning system

For deployed work, output quality should be modeled as an interaction among components, not as a sum of model intelligence and independent bonuses:

\[ Y = f(M, F, C, D, W, L, G, P, T, V, I \mid B, E, X) \]

where:

• \(M\): model and endpoint configuration;
• \(F\): purpose, task contract, and success criteria;
• \(C\): selected context and its ordering;
• \(D\): decomposition into bounded cognitive movements;
• \(W\): short-term working state;
• \(L\): durable memory and retrieval;
• \(G\): relational structure and semantic metadata;
• \(P\): provenance and evidence quality;
• \(T\): exact tools and action interface;
• \(V\): planning, review, verification, and escalation;
• \(I\): post-response integration and correction propagation;
• \(B\): token, money, latency, energy, and human-review budget;
• \(E\): environment and tool state; and
• \(X\): task distribution, novelty, sensitivity, and consequence.

This formulation has three implications.

First, a high-capability model can be bottlenecked by missing evidence, an ambiguous objective, a lossy interface, or an unobservable tool state. Second, a smaller model can perform well when the system externalizes the operation it is most likely to fail: retrieval, calculation, schema enforcement, state tracking, or verification. Third, interventions interact. Better retrieval may make a weaker model viable; longer context may erase the gain; provenance may enable verification; misleading metadata may bias the generator; memory may reduce repeated work while preserving a poisoned instruction.

The architecture should therefore route and evaluate a bounded cognitive movement, not a whole project. A movement is a unit with an explicit input state, transformation, output schema, evidence policy, and acceptance test—for example: classify a document, extract claims, propose a plan, execute a calculation, compare sources, rewrite a paragraph, or make a high-stakes exception judgment. Movements are small enough to select an appropriate model and verification policy, but large enough to have a meaningful outcome.

3.2 What present evidence actually supports

The evidence supports a qualified substitution frontier:

A smaller model plus stronger scaffolding can match or exceed a larger model with a weak harness when the task is bounded, the relevant evidence can be surfaced, the fragile operation can be delegated to an exact tool or structured interface, and success can be checked with a signal sufficiently independent of generation.

The clinical calculator study, SWE-agent interface ablations, adaptive test-time compute, and AFlow workflow search all instantiate parts of this condition. None establishes substitution on open scientific synthesis, novel strategy, adversarial interpretation, or decisions whose correctness cannot be externally observed.

The likely benefit of each AIOS layer is therefore conditional:

3.3 Gains are not generally additive

Let \(Q_0\) be baseline accepted-outcome quality, \(Q_A\) and \(Q_B\) the quality with individual interventions, and \(Q_{AB}\) the joint quality. A simple interaction estimate is:

\[ S_{AB} = (Q_{AB}-Q_0) - [(Q_A-Q_0)+(Q_B-Q_0)] \]

• \(S_{AB}>0\): synergy, as when decomposition makes exact verification possible.
• \(S_{AB}\approx0\): approximately additive effects.
• \(S_{AB}<0\): redundancy or interference, as when retrieval and long context duplicate evidence, or critique changes correct answers.

Multiplicative descriptions can be appropriate for sequential success probabilities—if a workflow requires every stage to succeed—but they often reveal fragility rather than amplification. A five-stage pipeline whose stages each succeed with probability 0.95 has an uncorrected joint success probability of about 0.77 if failures are independent. Correlated errors can be worse. Decomposition creates measurement and routing opportunities, but it also creates more boundaries at which state can be lost.

The current literature supplies examples of all four regimes:

• Synergistic: retrieval plus executable code in clinical calculation; structured claims plus citations; task decomposition plus stage-specific tools.
• Substitutive or redundant: RAG versus full long context; multiple homogeneous agents; several routers that converge on similar decisions.
• Antagonistic: longer context despite perfect retrieval; aggressive self-correction; fact condensation that improves one memory ability while hurting another; multi-agent topologies on sequential work.
• Bottleneck shifting: exact calculation removes arithmetic error and exposes interpretation error; better retrieval exposes generator faithfulness; more capable generation shifts the bottleneck to verification and policy compliance.

AIOS should consequently treat “micro-improvement” as an experimental factor with a carrying cost, not as an architectural entitlement.

3.4 Dynamic model selection as constrained control

For movement \(i\), a router selects model \(m\), harness configuration \(h\), and test-time budget \(b\):

\[ (m_i,h_i,b_i)^* = \arg\max \; \mathbb{E}[Q_{accepted}] - \lambda_c C - \lambda_t T - \lambda_e E - \lambda_h H - \lambda_r R \]

subject to hard constraints on privacy, data location, tool authority, minimum quality, provenance, and maximum failure consequence. Here \(C\) is monetary cost, \(T\) latency, \(E\) energy, \(H\) human-review effort, and \(R\) expected risk. A privacy constraint should not be traded away merely because a hosted model has higher expected accuracy.

An implementable routing decision has three stages:

1. Admissibility filter. Exclude endpoints that violate data-locality, modality, context, tool, licensing, or safety constraints.
2. Capability allocation. Estimate success and resource use for each remaining model-harness pair from task type, novelty, evidence sufficiency, past calibrated performance, and required output schema.
3. Escalation control. Run the least expensive admissible configuration that meets the risk-adjusted threshold; escalate on a failed external check, low calibrated coverage, model disagreement, retrieval insufficiency, tool anomaly, or out-of-distribution signal.

Raw self-reported confidence should be a weak feature, not a gate. Confidence-token and item-response results show that calibration can be learned, while router benchmarks show that transfer remains difficult. A reliable router needs outcome history keyed to the actual model version, endpoint, task family, and harness.

The model portfolio should be small and complementary. Adding every available model increases evaluation and operational cost and produces diminishing returns if their errors are correlated. A plausible initial portfolio is:

• an on-device or local model for sensitive extraction, classification, rewriting, and deterministic tool setup;
• a fast hosted or larger local model for ordinary synthesis and structured generation;
• a balanced reasoning model for planning, source comparison, and exception handling;
• a frontier model for novel, ambiguous, high-consequence, or poorly verifiable movements and for adjudication where no formal oracle exists.

This is a testable allocation, not a permanent hierarchy. The role boundaries should move as empirical capability changes.

3.5 Durable structure as semantic capital

“Semantic capital” is a useful hypothesis if defined operationally rather than metaphorically. It is accepted, versioned structure that lowers the marginal cost or risk of later work. Candidate assets include:

• verified claims linked to source spans and validity dates;
• typed entities, relations, definitions, and constraints;
• executable tests, calculations, transformations, and acceptance rules;
• task contracts and successful movement templates;
• decision records with assumptions and reversal conditions;
• correction records that identify affected downstream artifacts;
• calibrated performance histories for model-harness-task combinations.

Raw transcripts and unverified summaries do not qualify merely because they persist. An asset becomes semantic capital only if it is reusable, inspectable, correctable, and net-positive after maintenance, retrieval, privacy, and security costs.

Present evidence is adjacent rather than decisive. LongMemEval shows that indexing and temporal structure can improve later retrieval, but also that compressed memory can destroy needed information. HippoRAG 2 shows that relational structure aids associative retrieval while creating other tradeoffs. AFlow shows that workflow structure can be reused across models and tasks, but does not establish long-run organizational returns. Memory-injection and extraction attacks show that durability magnifies liabilities too.

The key longitudinal claim is therefore unresolved: does the marginal inference and review cost of accepted work decline as verified structure accumulates faster than its maintenance and failure costs grow? Section 5 specifies a direct test.

4. Implications if the thesis is substantially correct

4.1 Capability engineering shifts from model choice to portfolio-and-harness design

If joint-system effects are large, choosing the strongest general model for every operation is a locally rational but globally inefficient policy. System designers would instead identify which movements require latent world knowledge or novel inference, which are failures of evidence access or interface, and which can be made deterministic. Frontier capability remains essential, but its highest-value use shifts toward ambiguous synthesis, new-domain planning, exception judgment, and verification without an executable oracle.

This would also change procurement and benchmarking. A lower model price would not guarantee lower accepted-outcome cost if it produces more retries and expert review. A higher-priced model could be less expensive when it shortens trajectories or prevents downstream correction. Model selection would be evaluated at the movement and workflow level, with total cost per accepted outcome.

4.2 Privacy becomes a routing dimension rather than a universal model property

Local execution can reduce data egress and provide stronger control over retention. It does not automatically make the reasoning system private. Sensitive data can leak through durable memory, logs, embeddings, tool calls, or later prompts; local agents can still overreach through authorized tools. Privacy-aware routing must classify both input and likely output sensitivity, apply data minimization before inference, and constrain which memory or tool state each movement may access.

Some movements should be transformed locally before escalation: redact identities, compute aggregates, extract a non-sensitive schema, or replace source text with verified features. This can allow stronger hosted reasoning over a lower-sensitivity representation. Whether the transformation preserves enough information is itself an acceptance test.

4.3 Hallucination control becomes failure-mode-specific

Interventions reduce hallucinations only when they target the generating failure mode:

RAG alone does not eliminate hallucinations. Self-critique alone is not verification. Citations alone do not prove truth. Multiple agents alone do not create independence. The safest designs add an external constraint or evidence source and report what was checked.

4.4 Post-response integration can change the economics of inference

Most benchmarks score isolated responses and discard the resulting structure. A longitudinal system can instead convert an accepted output into tests, provenance edges, structured facts, and reusable plans. If successful, subsequent work retrieves verified artifacts rather than regenerating them. The benefit is not only fewer tokens; it includes fewer repeated source checks, faster correction propagation, and smaller failure surfaces.

The same mechanism can entrench mistakes. Integration should therefore separate proposed, checked, accepted, superseded, and revoked artifacts through ordinary version and provenance records; preserve the source span behind a normalized fact; and make downstream dependencies queryable. These are data-integrity requirements, not claims that a language model can infer perfect ontology.

5. Conditions and tests

5.1 System-level benchmark design

A benchmark intended to separate model intelligence from harness intelligence needs at least four axes:

1. Model: pinned model version, endpoint, quantization, reasoning level, sampling parameters, and serving limits.
2. Harness: task prompt, context-selection policy, memory, decomposition, interface, tools, verification, and retry rules.
3. Environment: source corpus, tool and world state, permissions, adversarial content, and time.
4. Outcome: task acceptance, provenance, cost, latency, energy, human effort, and contained harm.

The essential comparison is a crossed design: each model runs under both a minimal standardized harness and a strong standardized harness; each harness also runs across models. A model-only leaderboard fixes the harness and estimates a conditional model effect. A system leaderboard permits the harness to vary and estimates best achieved outcome. Both are useful if reported separately.

The benchmark should publish three scores rather than one:

• Model effect: performance under the common minimal and common strong harnesses.
• Harness lift: within-model change from minimal to strong harness, including incremental resource cost.
• System frontier: Pareto-efficient configurations over accepted quality, cost, time, energy, review, privacy, and risk.

To prevent the harness from silently embedding answers, the benchmark should disclose task-specific code and retrieved evidence, enforce held-out and later-arriving evaluation sets, and separate generic infrastructure from task-trained artifacts. Harness development compute belongs in the accounting if it is amortized over too few tasks.

5.2 Task panel

Build a panel of at least 240 hidden tasks, 30 in each family:

1. classification and extraction;
2. editing and constrained writing;
3. source-grounded synthesis;
4. relational and structural reasoning;
5. numerical, scientific, and algorithmic operations;
6. planning under dependencies and change;
7. stateful tool execution under policy;
8. high-stakes judgment with incomplete or conflicting evidence.

Each family should contain easy, medium, hard, and out-of-distribution tasks; public and private evidence; short and long horizons; and at least some cases where abstention is the correct outcome. Half of the tasks should arrive after harness and router development. A contamination-resistant subset should be built from newly generated internal states or time-stamped post-development events.

The ground truth should match the task. Use exact tests for calculation and code, state diffs for tool tasks, blinded expert panels for open deliverables, source-level adjudication for factual claims, and prospective real-user acceptance where available. Automated model judges may assist triage but should be calibrated against blinded human judgments and tested for model-family favoritism.

5.3 Experimental factors

Use four deployment classes—on-device/local, small or fast hosted, balanced reasoning, and frontier—and pin one or more concrete endpoints in each class. Cross them with nine binary system factors:

A full \(2^9\) factorial is usually too expensive. Begin with a resolution-IV or -V 32-cell fractional factorial for each model class and task family, then add a foldover and targeted cells to estimate interactions most likely to matter:

• model × each system factor;
• context × retrieval/memory;
• decomposition × tools;
• working state × long horizon;
• memory × provenance;
• planning × verification;
• verification × model capability;
• model × task difficulty × test-time budget.

With 240 tasks, four model classes, 32 cells, and three stochastic repetitions, the confirmatory grid contains 92,160 runs. A 16-cell screening pilot can eliminate inert or harmful factors before the full study. Report all development and search runs separately from confirmatory inference.

5.4 Cumulative-gain and ablation tests

Run three complementary experiments:

Factorial ablation. Randomize tasks across the fractional-factorial cells. Estimate main effects and interactions using a hierarchical mixed-effects model with task, domain, model endpoint, and evaluator as random effects. This is the primary causal estimate.

Cumulative ladder. Starting with the minimal harness, add layers in a preregistered sequence and then in the reverse sequence. The ladder is easy to interpret but order-confounded; disagreement with factorial estimates is evidence of interaction or selection effects.

Bottleneck substitution. For each task, identify the expected weak operation—evidence access, arithmetic, state tracking, planning, or adjudication—and compare a targeted intervention with a generic increase in model tier or reasoning tokens. Match either total compute or accepted-outcome quality. This tests whether system structure actually reduces inference burden rather than merely adding more inference elsewhere.

For quality \(Q\), estimate each interaction contrast with confidence intervals and control the false-discovery rate. Also test the cost-adjusted contrast:

\[ \Delta U = \Delta Q_{accepted} - \lambda_c\Delta C - \lambda_t\Delta T - \lambda_e\Delta E - \lambda_h\Delta H - \lambda_r\Delta R \]

Publish both raw outcomes and any chosen utility weights; do not conceal a quality loss inside a composite.

5.5 Dynamic-routing experiment

Compare the following policies on exactly the same task stream:

1. fixed on-device/local model;
2. fixed small/fast model;
3. fixed balanced model;
4. fixed frontier model;
5. deterministic rules based on task family, sensitivity, and consequence;
6. uncertainty cascade with external checks and escalation;
7. learned pairwise strong/weak router;
8. learned multi-model, multi-harness router;
9. cost/latency-aware router with a quality floor;
10. privacy-constrained router;
11. retrospective oracle using observed accepted outcomes.

The router may use task family, input length, source sufficiency, modality, tool availability, sensitivity, novelty, recent error rate, model disagreement, calibrated confidence, and observed verifier results. It may not use the hidden answer. Evaluate three conditions:

• Stationary: development and test tasks drawn from matched distributions.
• Shifted: new domains, longer horizons, changed endpoint versions, and degraded retrieval.
• Adversarial: misleading confidence, poisoned memory, prompt injection, contradictory sources, and a failing tool.

Primary routing endpoints should be quality at a fixed total cost, cost at a fixed quality and coverage, and regret versus the oracle. Secondary endpoints are escalation rate, false-negative non-escalations, over-escalation, portfolio utilization, calibration error, and worst-group performance. Recalibration and router-evaluation costs must be included when models change.

Hard escalation conditions should include:

• no admissible endpoint under the privacy or tool policy;
• absence of evidence required by the task contract;
• failed executable or source-level verification;
• an out-of-distribution or poor-calibration alarm;
• material disagreement among genuinely diverse candidates;
• a state mutation with high reversal cost;
• high-consequence judgment without an independent acceptance mechanism.

5.6 Longitudinal semantic-capital experiment

Run a 12- to 24-week repeated-work study in domains where tasks naturally recur. Randomize workstreams to three persistence conditions:

1. No reuse: each task starts without earlier artifacts.
2. Transcript reuse: prior conversations and documents are retrievable as raw text.
3. Structured reuse: only accepted artifacts are normalized into typed claims, sources, relations, tests, decisions, and correction dependencies, with raw evidence preserved.

Introduce controlled updates: corrected facts, conflicting sources, expired policies, entity merges, deletions, and adversarial records. At preregistered intervals, remove the structured store for a subset of tasks to estimate dependence and true reuse value.

Measure:

• marginal inference tokens, tool calls, and review minutes per accepted outcome over time;
• reuse rate and fraction of retrieved artifacts actually used;
• accepted-quality slope as the knowledge base grows;
• correction-propagation completeness and time;
• stale-memory and contradiction rates;
• provenance coverage and source-replacement accuracy;
• maintenance labor, storage, retrieval, and migration cost;
• privacy extraction and poisoning success;
• rollback completeness after a bad artifact is revoked.

Semantic capital is supported only if structured reuse improves the longitudinal quality-cost frontier relative to both no reuse and raw transcripts, while keeping stale, privacy, and poisoning failures within preregistered bounds. A growing store with flat productivity and rising correction cost is accumulated semantic debt.

5.7 Metrics beyond answer accuracy

Report Pareto frontiers rather than a single weighted score. A configuration dominated on quality, cost, latency, and review should be eliminated. For non-dominated configurations, the choice depends on the consequence and privacy regime.

5.8 Validity controls

The experimental program should preregister:

• task sampling and exclusion rules;
• model and endpoint versions;
• prompts, interfaces, tool permissions, and retry limits;
• evaluator identity and adjudication procedure;
• development versus confirmatory compute;
• primary outcomes and interaction hypotheses;
• stopping, escalation, and abstention rules.

Necessary controls include prompt and order variants, multiple stochastic repetitions, blinded outcome review, later-arriving hidden tasks, source-corpus freeze dates, endpoint drift detection, contamination audits, and negative controls where an irrelevant memory or metadata layer is added. Evaluate with and without tool failures and malicious retrieved content. Preserve full trajectories for audit while applying appropriate privacy controls.

6. Experimental matrix

The following matrix is the minimum program capable of testing both cumulative amplification and dynamic routing.

7. Supported conclusions, unresolved questions, and unsafe overstatements

Supported conclusions

1. Harness design materially changes performance for the same model. Interface, tool access, context policy, workflow, and verification are causal components of observed system performance.
2. Scale can be substituted selectively, not universally. Smaller models with exact tools, optimized workflows, or adaptive verified inference can beat larger weakly scaffolded models on bounded tasks.
3. Interventions are conditional and non-monotonic. Context length, self-correction, memory compression, multi-agent scaling, and added routing complexity all have documented negative cases.
4. Dynamic routing can reduce measured cost at a small or unobserved quality loss on matched benchmarks. Robust transfer, model churn, evaluator error, and high-consequence escalation remain open deployment problems.
5. Provenance and retrieval improve inspectability more reliably than they guarantee truth. Factuality, faithfulness, attribution, and source quality must be measured separately.
6. Durable memory expands both reuse potential and failure persistence. Versioning, deletion, scope, provenance, and adversarial testing are core capability requirements.
7. Accepted outcome per total resource is the appropriate system objective. Model accuracy alone omits retries, tools, human review, time, energy, privacy, and downstream corrections.

Unresolved questions for later research

• Do cumulative AIOS-layer gains generalize from bounded benchmarks to accepted professional outcomes?
• Do routers remain calibrated under endpoint churn, new domains, retrieval failures, and adversarial inputs?
• Does structured long-term reuse create net semantic capital after development, maintenance, correction, privacy, and security costs?
• Which review topologies provide genuinely independent evidence for open-ended, high-stakes judgment?
• How should energy, correction propagation, and failure containment be measured consistently across heterogeneous work?

Unsafe overstatements

The following claims are not supported by the reviewed evidence:

• “The model no longer matters once the system is well scaffolded.”
• “A small or on-device model can replace a frontier model for general reasoning.”
• “Each AIOS layer adds a fixed improvement, so the gains multiply.”
• “Longer reasoning always improves an answer.”
• “RAG eliminates hallucinations.”
• “A citation proves that a claim is true.”
• “Self-critique is independent verification.”
• “More agents are better than one agent.”
• “On-device execution is automatically private or safe.”
• “A router can rely on a model’s ordinary self-confidence.”
• “Persisting more information necessarily creates reusable knowledge.”

8. Overall assessment

The ambitious version of the AIOS proposition is plausible enough to warrant a serious research program, but not established enough to be treated as an architectural law. Current evidence shows that useful capability is jointly produced by model and harness, and that the harness contribution can be large. It also shows that the most successful interventions share a property: they make a previously latent operation explicit and testable. A calculator externalizes arithmetic; a typed tool interface constrains action; retrieval exposes evidence; provenance connects a claim to an audit target; a verifier converts additional inference into directed search.

This does not flatten the capability frontier. Frontier models remain disproportionately valuable when the system cannot specify the right decomposition, when relevant knowledge is absent, when evidence conflicts, when novelty is high, or when the acceptance function itself requires judgment. The system-level opportunity is to reserve that capability for movements where it changes the outcome, while using local or lower-cost models for movements that have been bounded, evidenced, and checked.

The most important scientific test is not whether a demonstration can make a smaller model win. It is whether a preregistered, model-plural system improves the accepted-outcome frontier on hidden and later-arriving work; whether its gains survive full accounting for development, review, maintenance, privacy, and energy; whether errors are detected rather than cosmetically grounded; and whether reusable structure lowers future marginal burden without becoming a reservoir of stale or adversarial state. If those conditions hold, AIOS would represent capability amplification through system design and selective frontier use. If they do not, the complexity is orchestration overhead rather than intelligence infrastructure.

9. Foundational lineage before August 2024

These sources establish the intellectual lineage but are excluded from the primary evidence window.

10. Dated source table

Dates are publication or public-release dates where available; conference-year dates are used when proceedings do not expose a precise day. “Provider” identifies first-party model or deployment evidence rather than independent validation.