Cognitive Agency and AI Interaction — Research Brief

Full research memo: Cognitive Agency and AI Interaction

1. Domain question

What does current evidence establish about cognitive surrender, offloading, automation bias, overreliance, deskilling, metacognition, critical thinking, and human agency in AI-assisted reasoning; which interaction and architecture mechanisms keep people cognitively engaged while still extending cognition; and how should those findings support, complicate, bound, or test AIOS without treating adjacent research as validation of the integrated system?

2. Executive answer

The evidence supports a conditional account, not a general claim that AI makes people stop thinking. Confident AI advice can improve short-task reasoning when correct and pull performance below an unaided baseline when wrong. People often fail to calibrate reliance to correctness, and fluent explanations can increase persuasion without reliably improving discrimination. The focal “cognitive surrender” paper demonstrates this advice-following problem on Cognitive Reflection Test items, but does not directly measure an absence of thought, establish a new cognitive system, or show population-wide decline.

Assisted output, learning, durable skill, and agency are distinct outcomes. General answer delivery can improve work while AI is present yet weaken later unaided performance in some educational experiments. Structured tutors, hints, constructive note-taking, human-first ideation, diagnostic feedback, and workplace systems that distribute expertise can instead improve learning or extend capability under suitable conditions. Cognitive offloading is therefore neither inherently harmful nor inherently beneficial. The key variables are what is delegated, whether users perform the operations they need to learn or retain, whether errors are detectable, and whether consequential authority remains practical rather than ceremonial.

Short foreground responses, explicit choices, human judgment points, and non-blocking downstream work form a plausible AIOS interaction hypothesis, but no study evaluates the bundle. Brevity can focus attention or conceal assumptions. Choice can support control or create anchoring. Human review can catch error or become rubber-stamping. Background work can preserve flow or accumulate review debt. The strongest design inference is to require meaningful cognitive acts at the right points while keeping evidence, uncertainty, provenance, alternatives, reversal, and escalation accessible.

Architecture research sharpens the AIOS thesis without validating it. Context, harnesses, tools, output constraints, validation, and workflow materially affect model performance. Models are useful semantic proposers; deterministic mechanisms can enforce specified identities, scopes, policies, structures, and controlled state changes; people can retain accountable purpose and consequential commitment. These functions remain fallible at their boundaries. Local models and local-first systems already support selected bounded tasks and offline operation. If their capability envelope expands within a mature hybrid architecture, durable person-controlled knowledge could support a large share of ordinary reasoning, private cognition, regulated work, peer collaboration, and reduced dependence on centralized application layers and routine remote inference. Those are important conditional implications whose determining conditions must be measured longitudinally.

3. Essential findings

Finding 1 — AI advice moves reasoning in both directions without proving cognitive surrender

• Finding: Confident AI advice can improve or impair reasoning, but following it does not by itself demonstrate that internal deliberation stopped.
• Evidence: Shaw and Nave’s 2026 preprint reports three preregistered experiments totaling 1,372 participants and 9,593 trials on seven Cognitive Reflection Test items. Correct GPT-4o advice increased accuracy, confidently stated errors reduced it, and incentives or feedback did not eliminate the influence. The design measured answers, not the presence of thought.
• Relationship to AIOS: boundary condition — the paper identifies a real reliance hazard but cannot establish that AIOS’s interaction pattern prevents “surrender.”
• Implication: AIOS should evaluate error detection, initial views, evidence use, and reasoned revision rather than treating agreement as the cognitive outcome.
• Limits or counterevidence: The repeated task set is narrow and artificial; the paper is not peer reviewed; System 3 is theoretical; and no independent replication was identified.
• Exact descent: Full memo — “## 1. Assessment of ‘Thinking—Fast, Slow, and Artificial’”, especially “### What the experiments establish” and “### What remains interpretation or hypothesis”; SSRN manuscript; PsyArXiv version.

Finding 2 — Assisted performance, learning, skill, and agency must be measured separately

• Finding: Better work with AI present does not establish learning, durable expertise, independent performance, or meaningful agency.
• Evidence: Bastani et al.’s preregistered field RCT found that unrestricted GPT-4 improved practice but produced a later unaided penalty, which a guarded tutor reduced. Constructive note-taking preserved more learning than answer consumption in Kreijkes et al. Budzyń et al. observed lower unaided adenoma detection after clinical AI exposure, a noncausal deskilling signal.
• Relationship to AIOS: direct mechanism — AIOS can separately instrument assisted output, later unaided transfer, source use, correction behavior, and retained fallback capability.
• Implication: Evaluation should distinguish production, learning, and resilience. Safety-critical capabilities need periodic unaided practice and delayed tests.
• Limits or counterevidence: Educational effects depend on tutor design, task, delay, and use strategy; the clinical result is observational; current longitudinal evidence is sparse; and no reviewed study measures irreversible general cognitive decline.
• Exact descent: Full memo — “### 2.2 Learning, transfer, and possible deskilling” and “## 8. Unresolved and contradictory evidence”; Bastani et al.; Kreijkes et al.; Budzyń et al.00133-5).

Finding 3 — AI can extend cognition when interaction preserves productive work

• Finding: AI can improve learning, ideation, breadth, and knowledge-work performance when the system supports the cognitive operations relevant to the task.
• Evidence: Kestin et al.’s randomized crossover study found greater immediate learning with a structured AI tutor than in-class active learning. Brynjolfsson, Li, and Raymond found productivity gains concentrated among less-experienced support workers. Dell’Acqua et al. found improved ideation and cross-functional output alongside task-dependent capability boundaries.
• Relationship to AIOS: supporting evidence — the studies support augmentation through structured interaction and distributed expertise, but do not test local files, AIOS memory, or its recurring grammar.
• Implication: AIOS can plausibly extend cognition through relevant context, feedback, examples, and critique. Evaluation should include capability expansion and transfer, not only harm prevention.
• Limits or counterevidence: Human-AI teams often exceed unaided humans without exceeding the better of human or AI; benefits vary by expertise and task; tutor results do not generalize to unrestricted use; and some experts imitate lower-quality model behavior outside the model’s capability frontier.
• Exact descent: Full memo — “### 2.3 Evidence that AI can extend or improve cognition”; Kestin et al.; Brynjolfsson, Li, and Raymond; Dell’Acqua et al., “The Cybernetic Teammate”; Dell’Acqua et al., “Navigating the Jagged Technological Frontier”.

Finding 4 — Useful safeguards elicit a meaningful cognitive act

• Finding: Interaction mechanisms are most promising when they make the user generate, inspect, compare, diagnose, or justify something relevant to the task rather than merely add friction.
• Evidence: Human-first ideation improved later transfer in Wong and Qiu; decide-first interaction reduced some reliance effects in Keke et al.; source and contradiction cues aided error detection in Kim et al.; probability highlighting improved calibration in Spatharioti et al.; and partial explanations reduced overreliance but increased underreliance in De Jong et al.
• Relationship to AIOS: direct mechanism — explicit judgment points and mode choices can be implemented as task-relevant cognitive acts rather than approval rituals.
• Implication: Before consequential advice, AIOS can request a view, confidence, criteria, mode, or draft; afterward it can return discrepancies, evidence, alternatives, and unresolved decisions.
• Limits or counterevidence: Think-first designs can interrupt productive iteration; explanations can persuade users of wrong answers; partial information can reduce total accuracy; individual differences matter; and most studies measure immediate behavior rather than long-term agency.
• Exact descent: Full memo — “### 2.1 Acute reliance, confidence, and critical-thinking behavior”, “### Explicit choices”, and “### Human judgment points”; Wong and Qiu; Kim et al.; Spatharioti et al.; De Jong et al..

Finding 5 — Short foreground and non-blocking work are a coherent but untested bundle

• Finding: Compact responses and asynchronous downstream work plausibly preserve attention and flow only when evidence remains accessible and background work stays bounded, reversible, and subject to a real return judgment.
• Evidence: Kim and Ji’s small exploratory-search study supports progressive disclosure, while Melumad and Yun’s seven experiments show that synthesized answers can reduce constructive search and depth of learning. No located experiment separates response length from reasoning quality or evaluates the combined AIOS pattern.
• Relationship to AIOS: boundary condition — the research supports direct testing of the pattern, not a claim that brevity or asynchronous execution protects cognition.
• Implication: Foreground responses should preserve uncertainty, provenance, alternatives, and expansion paths. Background work should be provisional, inspectable, reversible, and return unresolved judgments.
• Limits or counterevidence: Terse answers can hide assumptions and increase perceived authority; complete background work can create sunk-cost pressure and review debt; visible process can itself overload attention. Response length and underlying reasoning budget must be manipulated independently.
• Exact descent: Full memo — “### Short foreground responses”, “### Non-blocking downstream work”, and “### The combined interaction pattern”; Kim and Ji; Melumad and Yun.

Finding 6 — Semantic judgment, exact effects, and consequential authority require different trust boundaries

• Finding: Models can propose meaning, deterministic mechanisms can enforce specified invariants and controlled state changes, and humans can retain accountable commitment, but none of these layers makes the others correct automatically.
• Evidence: Across AgentDojo’s 97 tasks and 629 security cases, pre-context tool filtering reduced targeted attack success from 57.69% to 6.84%. In τ-bench, agents completed fewer than half of policy-and-state tasks and were less reliable across repeated runs. Tam et al. found that strict formats could impair reasoning.
• Relationship to AIOS: direct mechanism — this evidence maps closely to AIOS’s proposed model–software–human division of labor while stopping short of validating the integrated architecture.
• Implication: Model outputs should be inspectable proposals; identity, scope, authorization, validation, and state change should be mediated outside the prompt; human decisions should precede hard-to-reverse effects.
• Limits or counterevidence: Deterministic code enforces only correctly specified policies within a completely mediated boundary; schema validity is not semantic truth; authenticated identity is not legitimate intent; human approval can be biased or unlawful; and open-ended exceptions may resist fixed rules.
• Exact descent: Full memo — “#### 1. Models make bounded semantic judgments…” and “### Model–software–human trust boundary”; AgentDojo; τ-bench; Tam et al..

Finding 7 — Scaffolds, contexts, and cognitive modes are task- and model-dependent

• Finding: Scaffolding can expose or suppress capability, but neither maximal context nor multiple named roles is universally superior.
• Evidence: RULER and Du et al. show degradation as context length and complexity increase. SWE-agent improved coding through a purpose-built interface; Agentless performed strongly with fixed stages. A 2026 fixed-model study of 35 harness releases found no significant resolve-rate trend but sharply rising token and tool use. Persona and multi-agent results expose role sensitivity and coordination overhead.
• Relationship to AIOS: complicating evidence — it supports mode-specific context and lean harnesses as hypotheses while rejecting role labels, added coordination, or more context as sufficient mechanisms.
• Implication: AIOS should distinguish modes through evidence, tools, permissions, and criteria, then ablate them. Independent review requires external evidence, deterministic checks, trained critics, different models, or humans—not only a new persona.
• Limits or counterevidence: Results are dominated by coding and synthetic benchmarks; stronger models can alter which scaffolds help; added restrictions may be justified by security even when they lower open-ended performance; and the integrated AIOS mode architecture is untested.
• Exact descent: Full memo — “### Evidence from adjacent architecture research”, “#### 3. Current scaffolding suppresses frontier-model reasoning”, and “#### 4. Thinking, writing, editing…”; RULER; Du et al.; SWE-agent; Agentless; Ben Sghaier et al., preprint; Kamoi et al..

Finding 8 — Local–frontier complementarity has large implications but remains an integrated-system thesis

• Finding: Local models, domain retrieval, and local-first state already support selected functions; whether their combination can satisfy a large share of ordinary reasoning and reduce centralized application dependence is a consequential open test.
• Evidence: PALMBench and SlimLM demonstrate bounded on-device work with device, quantization, latency, quality, and safety tradeoffs. BRIGHT shows difficult reasoning-intensive retrieval; Fensore et al. found domain RAG underperformed when retrieval precision was poor. ConLoc and Janus show feasible local-first correctness with synchronization, identity, invariants, or consensus when required.
• Relationship to AIOS: supporting evidence — components are feasible and relevant, while the combined personal or organizational intelligence environment has not been evaluated longitudinally.
• Implication: If local models and mature scaffolding cover routine work, person-controlled knowledge could support private cognition, continuity, regulated use, peer exchange, and access, with frontier escalation retained. Displaced application and inference demand could fall.
• Limits or counterevidence: “Ordinary needs” needs a measured workload distribution; local corpora can be stale, incomplete, or poisoned; local execution is not a full privacy or sovereignty guarantee; costs can shift to endpoints and coordination; centralized pooling can improve rapidly; and rebound use can offset reductions.
• Exact descent: Full memo — “#### 6. Local self-contained domain systems…”, “#### 7. The architecture may materially reduce…”, and “### Complementary local and frontier topology”; PALMBench; SlimLM; BRIGHT; Fensore et al.; Consistent Local-First Software; Janus.

4. How the evidence refines the AIOS account

1. It replaces “thinking versus surrender” with four measurable levels: assisted performance, reliance calibration, later unaided performance, and longitudinal independent capability.
2. It identifies the cognitive act—not friction itself—as the likely ingredient in human-first views, notes, source checks, comparisons, and diagnostic feedback.
3. It makes short foreground/non-blocking work falsifiable by separating visible length from reasoning quality and measuring review debt, reversibility, and return decisions.
4. It separates failure modes of model proposals, deterministic enforcement, and human commitment, preventing structure, authorization, or approval from standing in for semantic correctness.
5. It turns local-domain implications into workload and routing questions: what stays local, escalates, leaves the domain, disappears, shifts, or creates new load.

5. Architectural boundaries preserved

1. Person-controlled canonical files remain the durable ground rather than being replaced by model memory or centralized state; ownership and inspectability require their own cognitive evaluation.
2. Why–How–What and the Fractal Seed remain the recurring grammar because no located study directly tests them.
3. Human purpose and consequential authority remain accountable, informed, and practical powers rather than ceremonial approval.
4. Selective model pluralism remains preferable to either a local-only or remote-only rule.
5. Distinct cognitive modes remain available as separate contexts while their evidence, tools, permissions, and criteria are tested for value.

6. Where the evidence connects to AIOS

7. Relationships across research programs

1. Context and memory: Evidence that excess or compressed context can impair reasoning connects cognitive visibility to the memory, provenance, retrieval, and assembly program without equating visibility with model efficiency.
2. Workflow and modes: Mode-specific contexts connect to workflow research through a shared question: whether gains arise from evidence, permissions, independent feedback, or simply additional computation.
3. Authority and regulated action: Cognitive agency connects to identity, authorization, accountability, consent, and policy, while human purpose remains distinct from lawful intent.
4. Local-first infrastructure: Local-domain implications depend on synchronization, privacy, sovereignty, economics, and centralized efficiency, joining personal control to coordination and lifecycle cost.
5. Collaboration: Peer exchange connects to distributed-intelligence research through the conditions under which plurality reduces correlated error or instead adds overhead, shared-model bias, and diffuse responsibility.

8. Priority source set

1. 11 January 2026 manuscript; revised 10 February 2026 — preprint: Shaw and Nave support strong effects of correct and confidently wrong AI advice on short reasoning tasks, not a new cognitive system or absent thought. SSRN.
2. 2025 — peer-reviewed preregistered field RCT: Bastani et al. support improved assisted practice but worse later unaided performance under unrestricted answers, with a guarded tutor reducing the penalty. PNAS.
3. 28 October 2024 — peer-reviewed preregistered meta-analysis: Vaccaro, Almaatouq, and Malone support common augmentation but uncommon synergy above the better party. Nature Human Behaviour.
4. 2025 — peer-reviewed randomized crossover study: Kestin et al. support structured AI tutoring as one condition improving immediate learning, not universal superiority to teachers. Scientific Reports.
5. 2025 — peer-reviewed field study: Brynjolfsson, Li, and Raymond support productivity gains and compressed novice–expert differences in customer support. Quarterly Journal of Economics.
6. 2026 — peer-reviewed experiment: Wong and Qiu support improved later unaided transfer from human-first ideation in the tested task. Educational Psychology Review.
7. 2025 — peer-reviewed CHI study, partly vendor-authored: Kim et al. support sources and contradiction cues for error detection, while explanations can increase reliance on wrong answers. CHI.
8. 2024 — peer-reviewed NeurIPS security benchmark: AgentDojo can support task-scoped tool exposure as a prompt-injection mitigation under its benchmark, including its limitations when benign and malicious goals need the same tool. NeurIPS.
9. 2025 — peer-reviewed ICLR benchmark: τ-bench supports that valid calls and policy access do not ensure correct final state or repeated-run reliability. ICLR.
10. 2024 — peer-reviewed COLM benchmark: RULER supports that advertised context length does not equal effective use as complexity and length rise. COLM.
11. 2025 — peer-reviewed Findings of EMNLP study: Du et al. support performance loss from input length after ordinary retrieval-position confounds are controlled. ACL Anthology.
12. 20 July 2026 revision — preprint: Ben Sghaier et al. support fixed-model harness effects on efficiency and quality without significant average resolve-rate improvement, not a universal scaffolding law. arXiv.
13. 2025 — peer-reviewed ICLR benchmark: PALMBench supports selected on-device tasks and material quality, energy, latency, and quantization tradeoffs. ICLR.
14. 2025 — peer-reviewed ICLR retrieval benchmark: BRIGHT supports reasoning-intensive retrieval as both value source and bottleneck for domain systems. ICLR.
15. 2024 — peer-reviewed distributed-systems study: Köhler et al. support local-first invariant preservation while showing the need for explicit synchronization machinery. IEEE Transactions on Software Engineering.

9. Open research and design questions

1. When is “cognitive surrender” a useful risk label, and when does it overstate what advice-following evidence can establish about thought?
2. Which objective should foreground interaction optimize for a given task—production, learning, calibrated judgment, or a person-selected mode—and when is an added cognitive act worth the effort?
3. Which operations require judgment before commitment, which may proceed provisionally, and what evidence must precede canonical integration?
4. Which mechanisms justify distinct cognitive modes as stable architecture, and which modes should remain task-selected configurations?
5. Under what measured conditions do local domains strengthen cognition, agency, and continuity without creating review debt or hidden dependence?