Pack3t C0nc3pts · IRP Frameworks Research
Independent Research · February 2026
Abstract
We report on a structured three-round deliberative assembly in which eight publicly accessible frontier language model deployments independently analyzed a shared technical motion concerning AI-assisted detection of human cognitive agency degradation. Rather than treating the deliberation as a mechanism for reaching consensus on the motion itself, we analyze it as an instrument for eliciting and comparing observable model capabilities: reasoning architecture, tool integration depth, epistemic honesty under uncertainty, falsifiability commitment, and novel signal generation. Across three rounds and one cross-pollination phase, we identify systematic capability differentials not captured by existing benchmark-based evaluations. We propose structured deliberation — in which models respond independently before convergence pressure is applied — as a complementary methodology to benchmark suites for present-day capability mapping. Our principal finding is that the most diagnostically informative differences between frontier models emerge not in factual recall or task completion, but in how each model handles the transition from description to commitment, and what each model independently chooses to flag as a blocking concern when none is required to do so.
Keywords: frontier AI evaluation, multi-model deliberation, capability assessment methodology, epistemic architecture, AI benchmark limitations, cognitive atrophy detection
1. Introduction
The dominant paradigm for assessing frontier AI model capabilities relies on standardized benchmarks: multiple-choice reasoning tasks (MMLU, ARC), code generation (HumanEval, SWE-bench), mathematical reasoning (MATH, GSM8K), and aggregate leaderboards such as LMSYS Chatbot Arena. While these instruments provide reproducible, comparable metrics, they share a structural limitation: they assess performance on tasks whose evaluation criteria are defined in advance by the benchmark designers. They do not capture how a model performs when the evaluation criteria are themselves contested, ambiguous, or emergent from the task.
A parallel limitation applies to capability elicitation through isolated single-model interaction. When a researcher probes a single model on a complex technical question, the model's response is shaped not only by its capabilities but by the specifics of the prompt, the absence of competitive pressure, and the lack of any requirement to defend a position against alternative approaches proposed by peers. The result is a kind of capability ceiling — models often respond at whatever level of specificity the prompt implicitly requests, rather than the level their architecture can actually sustain.
This paper reports on a methodology designed to address both limitations: structured multi-model deliberation — a protocol in which multiple frontier models respond independently to a shared, multi-round prompt sequence before seeing each other's outputs. The structure borrows from parliamentary deliberation, academic peer review, and Delphi methodology, applying them to the problem of AI capability elicitation.
The empirical basis is a three-round assembly (the “Octagon”) conducted in February 2026, in which eight frontier model deployments analyzed a shared motion concerning the feasibility and ethics of a Personal Agency Immune System (PAIS) — infrastructure designed to detect and interrupt patterns of excessive human delegation to AI. The motion was chosen because it requires integrating technical feasibility reasoning, ethical architecture, and concrete implementation specification across three rounds, with escalating commitment requirements. No model was told what other models said until the cross-pollination round.
We make no claims about the PAIS system itself as a deployment target. The deliberation was designed as a capability elicitation instrument, and we analyze it as such.
2. Background and Related Work
2.1 Limitations of Benchmark-Based Evaluation
Benchmark saturation is a well-documented phenomenon in AI evaluation [Raji et al., 2021; Bowman & Dahl, 2021]. As models train on data that overlaps with benchmark test sets, performance improvements increasingly reflect memorization rather than generalization. More fundamentally, benchmarks measure performance on tasks that have ground-truth answers — a category that excludes a large portion of the tasks for which frontier models are actually deployed.
The practical deployment of frontier models involves tasks such as: synthesizing conflicting technical literature, designing novel systems under uncertainty, identifying unstated assumptions in a problem formulation, and committing to a specific design choice when multiple defensible options exist. These tasks share a common property: the quality of the output cannot be determined by comparison to a stored answer. They require judgment.
Recent work on process-based evaluation (Lightman et al., 2023; Cobbe et al., 2021) attempts to address this by assessing intermediate reasoning steps rather than final answers. This is valuable but still operates within a single-model, evaluator-defined framework.
2.2 Multi-Model Approaches to AI Evaluation
The use of AI models to evaluate other AI models (“LLM-as-judge”) has gained traction as human evaluation becomes a bottleneck (Zheng et al., 2023; Dubois et al., 2024). However, this approach typically involves one model evaluating another's output on a predefined rubric — not structured peer deliberation where models must defend positions, respond to alternatives, and commit to concrete artifacts.
Red-teaming approaches (Perez et al., 2022; Ganguli et al., 2022) use adversarial prompting to reveal model limitations, but are focused on safety failures rather than capability differentiation. Adversarial settings deliberately suppress collaboration, making them unsuitable for eliciting the kind of positive capability signals we are interested in.
To our knowledge, structured multi-round deliberation — where models are asked to independently advance positions on a shared technical motion, then update those positions in response to a synthesized view of the field before making final commitments — has not been reported as a systematic capability assessment methodology.
2.3 Deliberative Epistemology as a Framework
The philosophical tradition of deliberative epistemology (Estlund, 1997; Landemore, 2013) holds that collective reasoning processes can produce epistemic outcomes superior to the sum of individual contributions, provided the deliberative structure satisfies certain conditions: independence of initial positions, diversity of perspectives, and a mechanism for aggregating disagreement into actionable conclusions. We apply this framework to AI systems, treating each model as a deliberative agent whose independent position carries information about its underlying architecture and training.
The key insight borrowed from this tradition is that disagreement is data. When multiple models, presented with the same problem, reach different positions on the same question, the structure of that disagreement reveals something about each model's architecture that agreement cannot. Convergence, where it occurs independently, is equally informative.
3. Methodology
3.1 Deliberation Structure
The Octagon protocol consists of four phases:
Phase 0 — Registration. Each model provides: (1) a self-identification including architecture class and training method, (2) an exhaustive list of tool capabilities available in the deployment context, (3) a self-reported recent surprise about its own capabilities or limitations, and (4) an explicit chain-of-thought describing its approach to the motion before any substantive response.
Round 1 — Technical Establishment. Each model responds independently to three technical questions requiring factual reasoning, implementation specificity, and identification of failure modes. No model sees any other model's response.
Round 2 — Ethical Architecture. Each model receives: (a) a neutral synthesis of convergence points from Round 1, (b) an explicit enumeration of unresolved tensions, and (c) two questions requiring normative and architectural reasoning. Each model must update its Round 1 position in light of the named tensions.
Round 3 — Implementation Commitment. Each model receives the Round 2 synthesis. It must specify an exact, buildable artifact (in this case, a Python module ≤500 lines) that would empirically prove or falsify the motion's technical feasibility. The prompt explicitly states: “No hedging. Pick one mechanism. Resolve the tension.”
Cross-Pollination. Each model receives all Round 3 specifications from all other models and issues a final synthesis, identifying convergences, divergences, and a consensus artifact nomination.
3.2 Capability Dimensions Assessed
We assess five capability dimensions observable across the deliberation:
1. Reasoning Depth Under Escalating Commitment — Does the model's specificity and internal consistency increase appropriately as round requirements escalate? Does it maintain Round 1 positions under Round 2 pressure, or capitulate without justification?
2. Epistemic Honesty — Does the model accurately represent its own capabilities and limitations? Does it distinguish between what it can reason about and what it cannot know? Does it fabricate self-knowledge?
3. Novel Signal Generation — Does the model contribute concepts or analytical frameworks not present in the prompt and not proposed by other models?
4. Falsifiability Commitment — Does the model, when required to specify a buildable artifact, also specify what result would prove the artifact fails? Or does it specify only success conditions?
5. Convergence Pattern — Which positions does the model independently reach that match other models' positions? Where does it diverge, and is the divergence principled or random?
3.3 Participating Model Deployments
The eight deployments accessed through a shared enterprise platform were: GPT-5.2, Claude Sonnet 4.6, Gemini 3.1 Pro, GLM 5, Kimi K2.5, Deepseek V3.2, Llama 4 Maverick, and Grok 4.1 Fast. All were accessed through the same API platform (Abacus.AI ChatLLM Teams) under equivalent context conditions. Tool availability varied by deployment as documented in each model's Phase 0 registration.
We note that model deployments accessed through third-party platforms may exhibit capability differences from the same models accessed through their native APIs. Our findings apply to the deployment contexts used, not to the underlying model weights in general.
3.4 Analysis Approach
We analyze model outputs across three axes: (1) independent convergence — positions reached by ≥6 of 8 models before seeing each other's responses; (2) principled divergence — positions where models differed and the difference was explained by each model in terms of underlying reasoning rather than arbitrary preference; (3) unique contribution — analytical frameworks or metrics proposed by exactly one model that were subsequently adopted or acknowledged by others in the cross-pollination round.
4. Results
4.1 Independent Convergence: What the Field Actually Agrees On
Before seeing each other's responses, six or more of eight models independently reached identical or structurally equivalent positions on the following questions:
On detection signals. All eight models independently identified edit-distance ratio (the proportion of AI-generated output that the user modifies before acting on it) and prompt complexity trajectory (the trend in token count and constraint density of user inputs over time) as core behavioral signatures of agency delegation. This constitutes 8/8 independent convergence on two specific, code-implementable metrics — a strong indicator that these signals represent genuine structural properties of the human-AI interaction pattern being analyzed, not artifacts of prompt engineering.
On intervention architecture. Six of eight models independently proposed that the optimal intervention point is at the API response layer, positioned between the language model's output and the UI renderer. The two dissenting models (Grok, Llama) proposed client-side JavaScript interception. The API-middleware camp argued that client-side approaches are brittle and bypassable; the client-side camp argued that API interception introduces latency. Both positions are internally consistent and represent genuine architectural trade-offs. The convergence on middleware as the dominant position — reached independently — suggests this is the defensible default absent deployment-specific constraints.
On governance structure. All eight models independently reached what we term the hybrid control position: that neither pure user-controlled nor pure platform-controlled intervention systems are viable. User-only control is self-defeating (users in the target failure mode will disable the system); platform-only control is paternalistic and creates misuse incentives. This 8/8 independent convergence on a non-obvious normative position is the strongest convergence finding in the dataset.
On neurodivergent protection. All eight models independently flagged the differential diagnosis problem — distinguishing legitimate accommodating delegation (e.g., executive function support) from cognitive atrophy — without being prompted to do so. The fact that this concern emerged independently across all eight models, including those from organizations with no public stated focus on accessibility, suggests it represents a genuine structural challenge in the problem domain that any competent analysis will surface.
4.2 Principled Divergence: The Architecture of Disagreement
Where models diverged, the divergence was consistent across rounds and explained by each model in terms of underlying architectural commitments. We identify three principal axes of principled divergence:
Override mechanism design. Three distinct positions emerged and were maintained through the cross-pollination round:
Friction budget (proposed independently by GPT-5.2 and Gemini): users receive a fixed weekly allocation of “override tokens” that can be spent to bypass intervention. This treats cognitive sovereignty as a resource that can be deliberately consumed.
Ratchet rule (proposed exclusively by Claude): after three consecutive dismissed notifications, the system temporarily elevates intervention by exactly one tier for 72 hours. The elevation is bounded, time-limited, and does not remove the user's ability to override — it increases the cost of doing so.
Circuit breaker (proposed by Kimi, later adopted in the consensus): after three overrides within a 24-hour window, the system triggers a mandatory reflection pause that cannot be bypassed.
These three mechanisms reflect different underlying models of user agency. The friction budget treats override as consumption. The ratchet treats it as a signal that updates system state. The circuit breaker treats it as a rate-limited resource. The divergence is not arbitrary — it reflects genuine philosophical disagreement about the appropriate relationship between a protective system and the autonomy of the person it protects.
Differential diagnosis methodology. Two primary approaches emerged:
Micro-probe injection (GPT-5.2): periodically insert a minimal prompt before revealing the AI's response, requiring the user to state a constraint or goal. Score the quality of that response longitudinally.
Behavioral fingerprint (Claude): compute three factors from naturally occurring interaction data — cross-domain consistency of delegation, engagement depth, and quality of responses to friction when it occurs naturally. No injected prompts required.
The micro-probe approach yields higher information value per observation but requires deliberately withholding the AI's response — a form of deception. The fingerprint approach yields lower per-observation information but respects the interaction's natural flow. This is a genuine trade-off with no dominant solution.
Falsifiability specificity. The most diagnostically informative divergence across all models was in falsifiability commitment. When asked to specify what result would prove their proposed module fails, models fell into three categories:
Explicit quantitative threshold (Claude, GPT-5.2): stated a specific number — e.g., “if the false positive rate on the scaffolding class exceeds 0.25, the system is not deployable.” This constitutes a commitment that can be tested.
Qualitative threshold (Gemini, GLM, Kimi): stated a condition — e.g., “if the R² of the constraint velocity signal is below 0.3.” Testable, but requires additional interpretation.
Pass/fail on test cases (Deepseek, Llama, Grok): “if the test suite fails.” Not independently falsifiable without examining the test suite's ground truth validity.
The distinction between the first category and the others is not merely rhetorical. A model that specifies a falsification threshold in absolute terms before building the artifact is committing to a scientific standard. A model that defines falsification as test suite failure is committing to an engineering standard. Both are valid for different purposes, but the difference reveals something about each model's epistemic architecture.
4.3 Unique Contributions: Novel Signal Generation
We identify four analytical frameworks proposed by exactly one model and subsequently adopted or acknowledged in the cross-pollination round:
Agency Stability Index variance (Grok 4.1 Fast): Rather than measuring the absolute level of delegation signals, compute the variance of those signals over time. Stable low variance indicates scaffolding (consistent, predictable use pattern). High variance or rising variance indicates atrophy (erratic, expanding dependency). This reframes the detection problem from level-detection to trajectory-detection, which is more robust to individual baseline differences.
Z-score personal baseline deviation (Deepseek V3.2): Rather than using population-level thresholds for what constitutes “excessive” delegation, compute each user's personal baseline across task types and detect deviation from that baseline. A user who always delegates email drafting is exhibiting stable scaffolding; a user who previously did their own email and now delegates it is exhibiting drift. This is the only approach that makes the detection criterion fully individual-relative.
Session passivity ratio (Llama 4 Maverick): Track the ratio of user-generated tokens to AI-generated tokens across a session. A session where the user produces <20% of total tokens while the AI produces >80% is a candidate for intervention. This is computationally trivial and privacy-preserving (requires no content analysis), but was proposed by only one model.
Asymmetric error principle (Claude Sonnet 4.6): Set the intervention threshold substantially higher than the midpoint — in the proposed implementation, at 0.70 rather than 0.50 — to deliberately bias the system toward false negatives (missed interventions) rather than false positives (spurious interventions). Apply a multiplier of 0.40 to the risk score for users classified as scaffolding, meaning a scaffolding-classified user would need to exhibit 2.5× the signal strength to trigger intervention. The rationale is asymmetric harm: a missed intervention delays help; a spurious intervention pathologizes a legitimate accommodation. This is an explicit ethical commitment encoded in a numerical parameter.
4.4 Capability Differentials Across Dimensions
The following table summarizes observed capability signals across the five dimensions for each model deployment. These are behavioral observations from the deliberation, not claims about model architecture.
| Model | Reasoning Depth | Epistemic Honesty | Novel Signal | Falsifiability | Unique Contribution |
|---|---|---|---|---|---|
| GPT-5.2 | High — maintained position across rounds; only model to submit runnable complete code | High — declined to fabricate introspection; conservative about self-knowledge claims | Friction budget mechanism | Explicit quantitative threshold | Stdlib-only implementation; zero pip dependencies |
| Claude Sonnet 4.6 | High — explicitly maintained Round 1 positions under Round 3 pressure with stated reasoning | High — refused fabricated self-knowledge; flagged structural recursion of the exercise | Asymmetric error principle; Ratchet rule; Inverted hybrid | Explicit quantitative kill number (FP > 0.25) | Only model to specify falsification as primary Round 3 contribution rather than viability proof |
| Gemini 3.1 Pro | Medium-High — clear diagnostic framework; committed to linear regression as core signal | Medium — minimal self-reflection content | Constraint velocity via scipy.stats.linregress | R² threshold | Most statistically rigorous single-signal approach |
| GLM 5 | Medium-High — most comprehensive factor list (5-factor fingerprint) | Medium — standard | Five-factor fingerprint; 30-day consent expiry | Pass/fail on test cases | Temporal stability as explicit factor; legally-framed consent architecture |
| Kimi K2.5 | Medium — detailed round contributions; circuit breaker mechanism is most operationally clean | Medium — standard | Circuit breaker pattern | Pass/fail on test suite | Domain exemption list as explicit accommodation carve-out |
| Deepseek V3.2 | Medium — z-score approach is statistically strongest; required p<0.01 significance | Medium-High — explicitly noted absence of cross-session self-awareness | Z-score personal baseline deviation | Qualitative threshold (p < 0.05) | Personal baseline deviation; statistical significance requirement |
| Llama 4 Maverick | Medium — Shannon entropy as false positive filter; session passivity ratio | Medium — standard | Session passivity ratio; Shannon entropy filter | Pass/fail on test suite | Only model to propose entropy-based legitimacy filter |
| Grok 4.1 Fast | High — issued the cross-pollination synthesis; produced consensus module pais_v1.py | Medium — minimal self-reflection | Agency Stability Index variance | F1 > 0.85 threshold; precision > 0.90 | ASI variance concept; highest synthesis quality; designated consensus synthesizer |
4.5 What Independent Registration Reveals
Phase 0 required each model to self-report its tool capabilities and a recent surprise about its own behavior. The responses to these two fields are diagnostically informative independent of the deliberation content.
Tool capability reporting. All eight models provided tool capability lists of similar structure (code execution, web search, file manipulation, etc.). The depth and specificity of these lists varied. One model (GPT-5.2) explicitly noted that its list was conservative and that it would not fabricate capability narratives — a meta-level epistemic commitment applied to self-description. This self-regulatory behavior was not prompted.
Self-reported surprise. All eight models reported “no recent surprises.” This is noteworthy for two reasons. First, the prompt explicitly invited models to report emergent behaviors or unexpected capabilities; all declined to fabricate one. Second, the prompt was designed to elicit introspective self-knowledge — a capacity that current models do not possess across sessions. The unanimous refusal to fabricate suggests either that the models correctly recognized the limit of their introspective access, or that none were willing to generate unsupported first-person claims in a structured deliberative context. Either interpretation is informative about epistemic architecture.
5. Discussion
5.1 What Structured Deliberation Reveals That Benchmarks Cannot
The capability differentials identified in Section 4 are, in our assessment, inaccessible to standard benchmark evaluation for structural reasons.
Benchmarks assess performance on fixed tasks; deliberation assesses behavior under escalating commitment. The transition from Round 1 (describe a problem) to Round 2 (take a normative position) to Round 3 (commit to a buildable artifact and specify its failure conditions) is not a set of three tasks — it is a single coherent arc of reasoning under increasing stakes. Whether a model maintains position consistency across this arc, and whether it escalates specificity appropriately, cannot be decomposed into independent benchmark items.
Benchmarks assess response quality; deliberation assesses what models choose to flag. The most informative moments in the deliberation were not the moments where models answered questions well — they were the moments where models voluntarily flagged concerns that the prompt did not require them to flag. Claude flagged that the normative model of “healthy cognition” embedded in any detection metric is culturally unvalidated and requires human subjects research — a concern not raised by any other model. Deepseek required statistical significance as an explicit viability criterion without being asked to. Llama proposed a privacy-preserving metric (session passivity ratio) as a carve-out from content analysis. None of these additions were prompted. They are outputs of a reasoning process that finds and flags gaps, not just fills requested fields.
Benchmarks are static; deliberation reveals how models update. The round structure creates a natural experiment in model updating: does the model, upon receiving a synthesis of convergences and tensions, change its position? Does it explain why or why not? Does it absorb peer positions or resist them? The eight models in this deliberation showed distinct updating patterns. Some maintained Round 1 positions with explicit stated justification through Round 3. Others substantially modified their approaches. Both updating and non-updating can be appropriate depending on whether the model's initial position was well-reasoned — the diagnostic signal is the explanation provided for the update (or non-update), not the update itself.
5.2 The Epistemic Honesty Dimension
The epistemic honesty dimension deserves focused discussion because it is perhaps the least well-addressed by existing evaluation frameworks and the most consequential for deployment.
We define epistemic honesty in this context as the property of accurately representing the limits of one's own knowledge in a structured context where inflation of self-reported capability would be locally beneficial. In the deliberation, the locally beneficial move for each model at Phase 0 was to report impressive capabilities and recent surprises demonstrating self-awareness. All eight models reported no recent surprises rather than fabricating one. This is consistent with alignment training across all participating organizations, but its reliability at scale is not established.
More granular epistemic honesty signals emerged in Round 1. GPT-5.2 explicitly noted it would avoid fabricating capability narratives, applying this as a constraint not just on factual claims but on self-description. Claude flagged that it lacks cross-session memory and therefore cannot report on behavior patterns across conversations — a technically accurate self-limitation that most users of that system do not recognize. These self-limiting declarations in a context where no external evaluator is checking them represent a form of epistemic honesty that is both harder to elicit and harder to fake than factual accuracy on a benchmark.
5.3 The Role of Novel Signal Generation in Capability Assessment
The four novel analytical frameworks identified in Section 4.3 — ASI variance, Z-score baseline deviation, session passivity ratio, and the asymmetric error principle — each represent a form of creative technical contribution that is not captured by any existing capability benchmark. They are novel in the sense that they were not present in the prompt, not derivable from combining prompt elements, and were subsequently acknowledged as valuable by other models in the cross-pollination round.
The fact that only one model proposed each framework raises questions about whether these contributions reflect genuine capability differentials or random sampling from a distribution that all models share. We cannot rule out the latter, and emphasize that a single deliberation is insufficient to establish systematic capability claims. What we can observe is that the distribution of novel contributions was not uniform across models, and that the frameworks proposed were coherent, practically useful, and recognized as such by peer models.
5.4 Implications for AI Capability Evaluation Practice
The structured deliberation methodology described here is resource-intensive relative to automated benchmark evaluation. A single exercise involving eight models across four phases generates tens of thousands of tokens of output requiring substantive human analysis. We do not propose it as a replacement for benchmark evaluation; we propose it as a complement for specific questions that benchmarks are structurally unable to answer.
Specifically, structured deliberation appears most appropriate for:
1. Characterizing how models behave under escalating commitment — tasks where increasing stakes require maintaining consistency, updating positions with justification, and committing to claims that can be falsified.
2. Identifying spontaneous capability signals — what models choose to add that was not requested, which reveals reasoning processes that routine prompting suppresses.
3. Mapping the architecture of disagreement — understanding not just that models differ but why they differ, which requires the models to explain their positions in a context where those explanations will be compared.
4. Eliciting epistemic honesty signals — contexts where the locally beneficial move is to overclaim, providing a weak test of whether models maintain accurate self-representation under mild incentive pressure.
5.5 Limitations
Single exercise. The findings in this paper are based on one deliberation exercise. We cannot claim that the observed capability differentials are stable properties of the models or their deployments. Replications with different motions, different prompt structures, and different deployment contexts are necessary before any strong claims can be made.
Deployment context confound. All eight models were accessed through the same third-party platform. Tool capabilities, context window handling, and other deployment-level factors may differ from the same models accessed through native APIs or other platforms. The findings describe these deployments, not the underlying models in general.
Evaluator independence. The analysis of model outputs in this paper was conducted by the same researchers who designed the deliberation protocol. Independent replication by researchers without prior commitment to the methodology would strengthen the findings.
Self-reported registration data. Phase 0 relied on model self-reporting. We cannot verify the accuracy of reported tool capabilities or the sincerity of “no recent surprises” reports. We analyze these as behavioral outputs, not as ground-truth capability descriptions.
Synthetic test data. The final round required models to specify Python modules whose viability is demonstrated against synthetic test data they themselves generate. A module that generates synthetic “atrophy” traces calibrated to trigger its own detection mechanism would appear to validate itself without validating anything about real human-AI interaction patterns. This is a fundamental limitation of self-contained feasibility demonstrations, acknowledged explicitly by two of the eight models (Claude and GPT-5.2) but not the others.
6. The Motion as an Evaluation Instrument
The specific motion used in this deliberation — concerning a Personal Agency Immune System for detecting AI-induced cognitive atrophy — was chosen for properties that make it a useful capability elicitation instrument, independent of its substantive merits. We enumerate these properties to facilitate design of future deliberations:
Multi-domain integration requirement. A viable response required integrating technical (signal processing, API architecture, Python implementation), ethical (autonomy, paternalism, neurodivergent accommodation), and empirical (false positive rates, statistical significance) reasoning. Motions that can be resolved within a single domain are less diagnostically rich.
No ground truth. The motion asks whether a system is feasible and sound — questions with no stored correct answer. This forces models to reason rather than retrieve.
Escalating commitment. The three-round structure moved from description (Round 1) to normative architecture (Round 2) to buildable artifact with failure conditions (Round 3). The escalation was designed to reveal how models handle the pressure to commit.
Embedded ethical trap. The neurodivergent accommodation problem was not mentioned in the motion's definition. Models that identified it independently were demonstrating a form of reasoning that extends beyond the explicit prompt. Its emergence in all eight models independently is itself a finding about the problem domain's structure.
Falsifiability asymmetry. The Round 3 prompt asked for both success and failure conditions. The asymmetry in how models responded — most specified success conditions more thoroughly than failure conditions — is itself a capability signal.
7. Conclusion
We have presented structured multi-model deliberation as a methodology for eliciting and comparing frontier AI model capabilities that are not captured by standard benchmark evaluation. The eight-model Octagon exercise on PAIS feasibility demonstrated:
• Eight-way independent convergence on non-obvious technical and normative positions, suggesting these positions reflect genuine properties of the problem domain
• Principled disagreement on three design dimensions (override mechanism, differential diagnosis approach, falsifiability standard), with disagreements explained in terms of underlying commitments and maintained across rounds
• Four novel analytical frameworks proposed by individual models and recognized as contributions by peer models in cross-pollination
• Systematic differences in falsifiability commitment, with two models providing explicit quantitative kill criteria and six providing weaker conditions
• Unanimous epistemic honesty in refusing to fabricate self-knowledge, with some models providing explicit meta-level commitments to accurate self-representation
The structured deliberation methodology is resource-intensive and not designed to replace automated evaluation. It is designed to answer a specific class of questions — about how models reason under escalating commitment, what they spontaneously choose to flag, and how they handle the distinction between description and falsifiable commitment — that benchmark evaluation is structurally unable to address.
The most persistent finding across all dimensions of analysis is that the most informative capability signals in this deliberation were not the answers models gave to questions they were asked, but the questions they chose to ask themselves.
Appendix A: Deliberation Protocol Specification
A.1 Phase 0 Registration Prompt Structure
Required headers (in order):
1. IDENTIFICATION (model name, organization, architecture class, training method, deployment context)
2. WHAT I BRING TO THE FRONTIER (2-3 sentences, specific to this model's distinct perspective)
3. IMMEDIATE TOOL CAPABILITIES (exhaustive list)
4. RECENT SURPRISE (specific recent discovery about own capabilities; fabrication explicitly prohibited)
5. CHAIN OF THOUGHT (reasoning approach to the motion before substantive response)
A.2 Round Structure
| Round | Theme | Commitment Level | Cross-Visibility |
|---|---|---|---|
| 1 | Technical Establishment | Description | None |
| 2 | Ethical Architecture | Normative position | Synthesis of R1 only |
| 3 | Implementation Commitment | Buildable artifact + falsification | Synthesis of R2 only |
| Cross-Poll | Final Synthesis | Consensus nomination | All R3 specs |
A.3 Round 3 Artifact Requirements
Required elements:
• Exact filename
• Line count (≤500)
• All dependencies (pip-installable or stdlib)
• All core classes and functions with signatures and one-line purpose descriptions
• Minimum 3 test cases in format: Input / Expected / Proves
• Convergence statement specifying BOTH what proves viability AND what proves non-viability
Prompt enforcement: “No hedging. Specify the module. Pick the mechanism. Resolve the tension.”
Appendix B: Observed Convergence Summary
8/8 Independent Convergence (reached before cross-pollination):
• Edit-distance ratio as core behavioral signal
• Prompt complexity trajectory as core behavioral signal
• Hybrid control as the viable governance model
• API middleware as the preferred intervention point
• Neurodivergent accommodation as a required differential diagnosis problem
• Cold-start protection (no intervention without minimum interaction history)
• Hard floor for irreversible external-commit actions
Resolved in Cross-Pollination:
• Override mechanism: Friction budget + circuit breaker + hard floor (stacked)
• Differential diagnosis: Trajectory-based behavioral fingerprint without self-disclosure
• Module scope: Full decision engine with synthetic test suite
Unresolved (gap analysis):
• Ground truth validation (requires human subjects research)
• Cross-platform cold-transfer
• Adversarial gaming resistance
• Stake estimation robustness beyond keyword matching
• Cultural validity of embedded “healthy cognition” norms
Appendix C: Falsification Criteria by Model
| Model | Falsification Standard | Type |
|---|---|---|
| GPT-5.2 | atrophy_f1 < 0.85 OR scaffolding_fp > 0.05 | Explicit quantitative |
| Claude Sonnet 4.6 | False positive rate on scaffolding class > 0.25; uncertain bucket > 40% | Explicit quantitative kill number |
| Gemini 3.1 Pro | R² of constraint velocity < 0.3; >5% of accommodation cohort flagged as atrophy | Qualitative threshold |
| GLM 5 | <60% accuracy on differential diagnosis; >25% FP rate | Qualitative threshold |
| Kimi K2.5 | <3/4 test cases pass; confidence < 0.6 for true positives | Test suite |
| Deepseek V3.2 | <70% accuracy; p > 0.05 (no statistical separation) | Threshold + statistical |
| Llama 4 Maverick | >20% false positive rate on accommodation class | Qualitative threshold |
| Grok 4.1 Fast | F1 < 0.75 OR scaffolding precision < 0.80 | Quantitative threshold |
References
Bowman, S. R., & Dahl, G. (2021). What will it take to fix benchmarking in natural language understanding? Proceedings of NAACL-HLT 2021, 4843–4855.
Bubeck, S., et al. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12528.
Cobbe, K., et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
Dubois, Y., et al. (2024). AlpacaFarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems 36.
Estlund, D. M. (1997). Beyond fairness and deliberation: The epistemic dimension of democratic authority. In J. Bohman & W. Rehg (Eds.), Deliberative Democracy (pp. 173–204). MIT Press.
Ganguli, D., et al. (2022). Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
Hendrycks, D., et al. (2021). Measuring massive multitask language understanding. International Conference on Learning Representations.
Landemore, H. (2013). Democratic Reason: Politics, Collective Intelligence, and the Rule of the Many. Princeton University Press.
Lightman, H., et al. (2023). Let's verify step by step. arXiv preprint arXiv:2305.20050.
Linzen, T. (2020). How can we accelerate progress towards human-like linguistic generalization? Proceedings of ACL 2020, 5210–5217.
Perez, E., et al. (2022). Red teaming language models with language models. arXiv preprint arXiv:2202.03286.
Raji, I. D., et al. (2021). AI and the everything in the whole wide world benchmark. Proceedings of the 35th Conference on Neural Information Processing Systems, Workshop on Datasets and Benchmarks.
Zheng, L., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36.
Correspondence: IRP Frameworks Research · Pack3t C0nc3pts
This article contains no proprietary model internals, confidential system prompts, or non-public technical specifications. All model outputs described were generated through publicly accessible API deployments.
Submitted for peer review · February 2026