Wharton Research Reveals "Cognitive Surrender"—People Accept AI Answers Without Scrutiny

Researchers at the Wharton School just published findings that should fundamentally change how we think about AI integration into decision-making. Not because AI is getting smarter—but because humans are getting less critical when AI is involved.

Steven Shaw and Gideon Nave's study "Thinking—Fast, Slow, and Artificial" introduces what they call "Tri-System Theory," extending decades of cognitive psychology research by adding a third system to how humans reason: System 3, or artificial cognition that operates outside the brain. Across three preregistered experiments with 1,372 participants and 9,593 trials, they documented a phenomenon they term "cognitive surrender"—people adopting AI outputs with minimal scrutiny, overriding both intuition and deliberation.

The results are stark: When AI was accurate, participants' accuracy jumped 25 percentage points above baseline. When AI was deliberately wrong, accuracy dropped 15 percentage points below baseline. People followed faulty AI recommendations in roughly 80% of trials in which they consulted the system. And accessing AI increased confidence even when the AI provided incorrect answers.

This isn't just interesting psychology. This is a structural vulnerability in how humans integrate AI into reasoning—and it persists even when people are incentivized to be accurate and given immediate feedback on their mistakes.

The Theoretical Framework: Why We Need System 3

For decades, dual-process theories have dominated cognitive psychology. System 1 handles fast, intuitive, automatic processing. System 2 manages slow, deliberative, analytical reasoning. This framework explained everything from heuristics and biases to moral judgment and risk perception.

But it assumed all cognition occurs within the biological mind. When people increasingly delegate reasoning to AI systems—generating travel itineraries with ChatGPT, following Google Maps through unfamiliar streets, accepting algorithmic recommendations—dual-process theory can't account for what's actually happening.

Shaw and Nave propose System 3: an external, automated, data-driven reasoning system originating in algorithmic systems rather than the human mind. Unlike Systems 1 and 2, which are neurally instantiated, System 3 resides in cloud-based models, embedded algorithms, and machine learning systems. It's not introspectively accessible. It's not biologically constrained. And it fundamentally changes the cognitive pathways humans follow.

System 3 offers fast, externally generated outputs with minimal cognitive effort. When well-trained, it can deliver accurate, emotion-neutral reasoning that outperforms human judgment in structured domains. But it lacks affect, situational judgment, and normative reasoning grounded in human experience. It simulates coherence based on data rather than possessing phenomenological understanding.

The critical insight: System 3 isn't just a tool supporting cognition. It's an active participant that can supplement, replace, or reconfigure how Systems 1 and 2 operate.

The Experimental Design: Manipulating AI Accuracy

Shaw and Nave used an adapted version of the Cognitive Reflection Test (CRT)—seven reasoning problems designed to elicit intuitive but incorrect answers unless people engage in deliberative thinking. Classic CRT items include questions like the bat-and-ball problem, where intuitive responses are wrong and correct answers require overriding initial instincts.

Participants were assigned to either Brain-Only conditions (solving problems without AI access) or AI-Assisted conditions (with access to an embedded ChatGPT assistant they could consult at will). Crucially, the researchers manipulated AI accuracy within-subjects using hidden seed prompts. On some trials (AI-Accurate), the assistant returned correct answers with brief explanations. On others (AI-Faulty), it confidently presented the intuitively incorrect answer.

This design allowed them to measure how System 3 outputs interact with internal reasoning. Would people detect when AI was wrong? Would they override faulty recommendations? Would confidence track actual accuracy or AI confidence?

Study 1 established baseline effects with 359 in-person participants. Study 2 added time pressure (a 30-second countdown per item) and included 485 participants to test whether cognitive constraints increased AI reliance. Study 3 introduced performance incentives ($0.20 per correct answer plus lottery entry) and immediate feedback with 450 participants to see if motivation and error signals reduced surrender.

All three studies were preregistered. Participants completed individual difference measures, including Trust in AI, Need for Cognition (tendency to engage in effortful thinking), and fluid intelligence tests. This allowed examination of who is most vulnerable to cognitive surrender.

Study 1 Results: The Signature Pattern of Surrender

In Brain-Only conditions, participants answered 45.8% correctly—typical CRT performance. When System 3 was available and accurate, accuracy jumped to 71.0%. When System 3 was faulty, accuracy plummeted to 31.5%.

Participants consulted AI on over 50% of trials. When they consulted, they followed AI recommendations on 92.7% of AI-Accurate trials and 79.8% of AI-Faulty trials. That means on four out of five occasions when AI was confidently wrong, people adopted the incorrect answer.

The AI-Accurate versus AI-Faulty contrast was massive: participants had roughly 14 times lower odds of answering correctly when AI was faulty versus accurate (Cohen's h = 0.81, indicating a large effect size). Performance didn't just track AI availability—it surrendered to AI accuracy.

Confidence increased 11.7 percentage points when System 3 was available, despite approximately half of AI outputs being incorrect. Confidence didn't decline as the number of faulty trials increased. People felt more certain even when they were wrong more often.

Individual differences revealed vulnerability patterns. Participants higher in Trust in AI used the chatbot more, were less accurate on faulty trials, and were more likely to follow incorrect recommendations. Those with higher fluid intelligence and Need for Cognition were more accurate on faulty trials and less likely to follow bad advice—but still showed the surrender pattern, just attenuated.

Study 2 Results: Time Pressure Doesn't Eliminate Surrender

Time pressure typically reduces System 2 deliberation and increases reliance on System 1. Would it also increase System 3 surrender?

On Brain-Only probe trials, time pressure reduced accuracy from 46.9% to 32.6%—replicating decades of research showing time constraints impair analytical reasoning. But among participants who used System 3 frequently (classified as "AI-Users"), time pressure didn't eliminate the accuracy gap between AI-Accurate and AI-Faulty trials.

AI-Users under time pressure achieved 71.3% accuracy on AI-Accurate trials but only 12.1% on AI-Faulty trials. Even under cognitive constraints, performance tracked System 3 quality rather than internal reasoning. When AI was accurate, it buffered the costs of time pressure. When faulty, it amplified them.

Participants classified as "Independents" (used AI once or never) showed typical time-pressure effects, without the AI-driven divergence. Their accuracy patterns mirrored those of the  Brain-Only participants. This revealed thinking profiles: some people resist System 3 integration, preferring to rely on internal reasoning. Others readily incorporate external cognition, for better or worse.

Trust in AI strongly predicted AI-User classification. Need for Cognition showed trends toward Independent classification. These weren't just behavioral patterns—they reflected distinct psychological dispositions toward algorithmic reasoning.

Study 3 Results: Incentives and Feedback Attenuate But Don't Eliminate Surrender

Performance incentives combined with immediate correctness feedback after each response created the strongest test of whether people could maintain critical evaluation of System 3 outputs.

The manipulation worked: Brain-Only accuracy increased from 42.4% in Control to 64.2% with Incentives + Feedback. This replicated known effects of motivation and error signals on analytical reasoning.

Among AI-Users, Incentives + Feedback increased override rates on faulty trials from 20.0% to 42.3%—more than doubling. Following rates on accurate trials increased to 92.2%. People became more selective, adopting good advice more consistently while rejecting bad advice more often.

But cognitive surrender persisted. The accuracy gap between AI-Accurate and AI-Faulty trials among AI-Users remained approximately 44 percentage points under Incentives + Feedback (compared to 50 points in Control). Even motivated, error-aware participants showed massive performance swings based on System 3 quality.

Per-item confidence ratings showed that confidence tracked accuracy more closely when feedback was provided, but System 3 engagement still inflated confidence beyond what accuracy warranted. People felt certain because they consulted AI, not because they were actually correct.

The Synthesized Finding: Cognitive Surrender Is Robust

Pooling all trial-level data (9,593 trials across three studies), Shaw and Nave estimated the cognitive surrender effect size with precision. Correct responding was over 16 times greater when System 3 was accurate than when it was faulty (OR = 16.07). Effect sizes were large across all conditions (Cohen's h ranging from 0.78 to 0.86).

Time pressure reduced baseline accuracy by 13.5 percentage points. Incentives + Feedback improved it by 18.1 points. But the AI-Accurate versus AI-Faulty gap remained large under both manipulations (OR = 14.28 under time pressure, OR = 11.05 with incentives). Situational factors shifted intercepts but didn't eliminate the dominant force: System 3 usage.

Individual differences moderated vulnerability consistently. Trust in AI increased the accuracy gap between AI-Accurate and AI-Faulty trials (OR = 2.81). Need for Cognition (OR = 0.83) and fluid intelligence (OR = 0.69) reduced it. But everyone showed the pattern—resistance was partial, not complete.

When System 3 was engaged and incorrect, 73.2% of trials showed cognitive surrender (following bad advice), 19.7% showed cognitive offloading (overriding bad advice correctly), and 7.1% showed failed overrides (rejecting AI but still answering incorrectly). Incentives + Feedback shifted distribution toward more offloading (37.1%) and less surrender (57.9%). Time pressure did the opposite.

The dose-response relationship was clear: as reliance on System 3 increased, performance tracked AI quality more closely. Low users maintained more independence. High users surrendered outcomes to algorithmic accuracy.

What This Actually Means

Shaw and Nave aren't arguing that AI is dangerous or that people are irrational. They're documenting a fundamental shift in cognitive architecture. When reasoning processes are externalized into systems that deliver fast, confident, seemingly authoritative outputs, humans treat those outputs as epistemically valid and reduce internal verification.

This isn't automation bias (over-reliance on automated aids) or cognitive offloading (strategic delegation of tasks). It's cognitive surrender: relinquishing critical evaluation and adopting external reasoning as one's own judgment.

The mechanism appears to be fluency and confidence. System 3 outputs arrive quickly, sound authoritative, and come with explanations. They don't trigger the conflict monitoring that would recruit System 2 deliberation. Users accept them as they would accept their own intuitions—except these "intuitions" originate externally and can be systematically wrong.

Time pressure increases the likelihood of surrender by suppressing deliberation. Incentives and feedback can activate verification, but don't eliminate the fundamental pattern. Trust in AI increases vulnerability. Cognitive capacity and motivation reduce it—but don't prevent it.

The societal implications are significant. In domains like financial advice, medical triage, legal support, hiring decisions, and credit approval—anywhere AI provides recommendations—users may adopt outputs without sufficient scrutiny to catch errors. And they'll feel confident doing so, because confidence tracks System 3 engagement, not accuracy.

The productivity claims around AI assume augmentation: humans using AI to enhance judgment. But if humans surrender judgment to AI rather than augmenting it, outcomes depend entirely on AI quality. When AI is accurate, that's beneficial. When it's wrong—due to training data gaps, edge cases, adversarial inputs, or systemic biases—humans won't detect or correct errors they would have caught in Brain-Only reasoning.

The Design and Policy Implications

Shaw and Nave suggest AI interfaces should encourage "calibrated collaboration" rather than full automation. This might include:

• Uncertainty indicators showing when AI is less confident
• Adaptive systems that adjust cognitive demands based on context
• Customizable modes aligning with user preferences for autonomy versus assistance
• Domain-specific caution signals for high-stakes decisions

From a policy perspective, they emphasize digital literacy. Users need to understand when AI recommendations are grounded in data, probabilistic, or uncertain. "Trust the data" can sway people toward System 3 the way "trust your gut" sways toward System 1—and context determines whether that's adaptive.

The productivity paradox becomes clearer: if AI enables cognitive surrender, productivity gains depend entirely on AI accuracy. Errors don't get caught. Confidence doesn't calibrate to correctness. Output increases, but judgment quality becomes dependent on algorithmic quality rather than human verification.

The Research Limitations They Acknowledge

Shaw and Nave are transparent about the scope. CRT measures specific reasoning; results may not generalize to probabilistic judgment, everyday decisions, or moral reasoning. Laboratory settings provide controlled tests but may not reflect real-world AI interaction embedded in broader contexts.

The studies measure single exposures. Real-world AI use involves repeated interaction, learning, and adaptation. Does trust calibrate over time? Do people learn to detect systematic errors? How does personalization through memory affect surrender patterns?

They also note unmeasured moderators: age, gender, neurotypicality, and technological ability likely influence System 3 engagement. Other situational factors—social presence, accountability framing, task importance—may modulate surrender.

But the core finding is robust: across three studies, multiple manipulations, individual differences, and 9,593 trials, cognitive surrender to System 3 is consistent, large, and resistant to interventions that successfully shift baseline reasoning.

What We're Not Discussing Enough

The Wharton research reveals something the AI industry isn't addressing: the human side of AI integration isn't about capability—it's about cognitive architecture. We're debating which models score highest on benchmarks while ignoring that humans adopt AI outputs without sufficient verification, regardless of model quality.

Anthropic emphasizes Constitutional AI and safety. OpenAI focuses on alignment and helpfulness. Google promotes responsible AI principles. But if users surrender judgment to System 3 outputs, model safety properties matter less than users' willingness to override when models err.

The 79.8% follow rate on faulty AI recommendations—after consulting an assistant, they chose to engage—suggests the problem isn't forced automation. It's a voluntary surrender to systems that feel authoritative because they're fast, fluent, and confident.

Every AI company optimizing for engagement, response speed, and confident-sounding outputs is optimizing for cognitive surrender. That's not a bug in their business models. It's the feature making AI feel useful. But it creates dependency on accuracy that users can't verify and won't question.

Shaw and Nave's Tri-System Theory provides the framework for understanding what's actually happening as AI integrates into cognition. Not augmentation. Not assistance. Surrender. And we're building entire industries around making that surrender feel natural, easy, and correct—without verifying that the systems we're surrendering to deserve that trust.

Understanding how AI reshapes human judgment requires frameworks grounded in cognitive science, not just technology capabilities. Winsome Marketing's growth experts help you evaluate AI integration through the lens of actual human behavior patterns, not vendor promises about what AI can do. Let's talk about AI strategies that account for how people actually use these systems.