AX Research
Summary
Briefing: AX Research
For a recent PhD graduate and postdoctoral researcher positioning for industry AX research roles — tracking evaluation frameworks, human-agent interaction design, and safety/alignment challenges in agentic systems.
Key Insights
- Current agentic benchmarks measure task success while missing the dimensions that matter most for safe deployment. DeepSeek V4 Pro scores 1,554 on GDPval-AA — the leading open-weight result — yet simultaneously registers a 94% hallucination rate on AA-Omniscience. These two numbers coexist in the same evaluation suite, which tells you something important: the field currently lacks frameworks that integrate factual reliability, task performance, and operational cost into a single coherent measure of agent readiness. For an AX researcher, this gap is a live research opportunity, not a future problem.
-
94–96% hallucination rates in leading open-weight models are not a footnote — they are the central safety problem for agentic systems. V4 Pro hallucinates 94% of the time on the AA-Omniscience benchmark; V4 Flash is worse at 96%. For agentic systems that take real-world actions based on model outputs, this level of factual unreliability is a deployment blocker, not just a quality concern. This grounds the abstract "safety and alignment" agenda in a concrete, measurable failure mode — and points toward hallucination detection and mitigation as among the most immediately tractable research contributions someone entering this field could make.
-
Anthropic's Project Deal is the clearest current example of what AX research problems actually look like in the wild. Claude was deployed to negotiate compensation on behalf of employees — a high-stakes, real-world agentic task — and surfaced model-quality asymmetry (the agent negotiating against employees had access to better models) and policy challenges that no lab benchmark would have predicted. This signals that industry AX roles will require researchers who can design studies for agents operating in messy social contexts with unequal power dynamics, not just controlled task-completion pipelines.
-
Long-context architecture may be a more consequential contribution to agentic capability than raw benchmark position — and it's under-discussed in AX circles. A notable thread of technical commentary argues that V4's systems-level design for sustaining coherent behavior over millions of tokens matters more than where it ranks on leaderboards. For agentic systems that operate over extended interactions — executing multi-step tasks, maintaining context across sessions — the architecture governing how a model handles long contexts is directly relevant to interaction design, not just infrastructure. This is a lens that cognitive and experimental psychologists entering AX would be well-positioned to contribute to, given the connections to working memory, attention, and cognitive load research.
- [AINews] DeepSeek V4 Pro (1.6T-A49B) and Flash (284B-A13B), Base and Instruct — runnable on Huawei Ascend chips
Emerging Patterns
- Economic constraints are quietly reshaping agentic interaction design from the outside in. The V4 Pro vs. Flash comparison — 12× cost differential for roughly 30 leaderboard positions of performance — reveals that practical agent deployment decisions are driven as much by economics as by capability scores. A model with a 1-million-token context window is only useful if using that context doesn't make the system unaffordable at scale; V4's high token-volume usage in some evaluations shows this tension is already real. For AX researchers, this means interaction design principles cannot be developed in isolation from system economics — what an agent should do and what it can sustainably do are increasingly different questions.
-
Industry is moving from theoretical safety commitments to concrete safety mechanisms, but the implementations are uneven. OpenAI's Bio Bug Bounty for GPT-5.5 and Anthropic's Project Deal both represent practical, deployed safety work — one proactive (incentivizing vulnerability discovery before harm), one reactive (learning from a live deployment that surfaced unanticipated alignment failures). The contrast between these approaches is instructive: bug bounties treat safety as an adversarial search problem, while Project Deal exposed safety failures that were structural and contextual rather than technical. Researchers who can bridge these two paradigms — designing for anticipated failure modes and building methods to learn from unexpected real-world failures — will be well-positioned for industry roles.
- [AINews] DeepSeek V4 Pro (1.6T-A49B) and Flash (284B-A13B), Base and Instruct — runnable on Huawei Ascend chips
Dissenting Views
- The prevailing assumption is that better benchmark scores define better agentic capability — but a meaningful counterargument reframes what "better" means. Most coverage leads with GDPval-AA rankings and leaderboard positions as the primary signal of a model's agentic readiness. The dissenting view, held by a notable subset of technical readers, is that long-context systems architecture is the more important contribution for practical agentic applications: how a model sustains coherent, cost-efficient behavior over extended interactions matters more than its peak performance on a discrete task. This is a difference in emphasis rather than a factual contradiction, but it has real implications for AX researchers — if the dissenters are right, the field needs evaluation frameworks that measure sustained interaction quality, not just task-completion benchmarks.
- [AINews] DeepSeek V4 Pro (1.6T-A49B) and Flash (284B-A13B), Base and Instruct — runnable on Huawei Ascend chips
Read & Act
What to Read
- [AINews] DeepSeek V4 Pro (1.6T-A49B) and Flash (284B-A13B), Base and Instruct — runnable on Huawei Ascend chips — This is currently the most data-dense single source for the three AX pillars you're tracking. It gives you citable benchmark numbers (GDPval-AA, AA-Omniscience hallucination rates), the Pro/Flash cost-performance debate, the architectural argument about long-context design, and the Project Deal and Bio Bug Bounty examples — all in one place. Reading it in full also exposes you to the texture of how industry practitioners argue about model evaluation, which is directly useful for positioning in research conversations.
What to Do
-
Build a candidate evaluation framework that jointly scores agentic task performance, factual reliability, and operational cost — and stress-test it against the V4 data. The GDPval-AA / AA-Omniscience combination already gives you two axes (task performance and hallucination rate); adding a cost-efficiency metric (e.g., task completion cost normalized by token usage) gives you a third. Sketch what a composite AX evaluation rubric would look like using V4 Pro and Flash as test cases, then identify which dimension current benchmarks underweight most severely. This becomes a concrete research contribution you can discuss in interviews — and a prototype for a paper or industry report.
-
Design a small study modeled on the Project Deal scenario to operationalize what "model-quality asymmetry" means for human outcomes. Anthropic's deployment exposed that when an AI agent negotiates against a human backed by a weaker model, outcomes may be systematically unfair — but this wasn't measured, it was observed anecdotally. Use your experimental psychology background to design a controlled study that manipulates agent capability asymmetry in a negotiation or resource-allocation task and measures participant outcomes and perceptions of fairness. This directly applies your training to an open AX problem that industry has surfaced but not yet systematically studied.