AX Research

COMPLETED May 20, 2026
Summary

Briefing: AX Research

Purpose: I am a recent PhD graduate and postdoctoral researcher trying to break into industry research roles. While most Experimental and Cognitive Psychology researchers target User Experience (UX) roles, I am curious to learn more about an emerging area, which is Agentic Experience (AX). Because this is a new area, I want to keep up with trends and new ideas about how this work is currently being shaped and be up to date about the most pressing questions in the field. Follow academic and industry perspectives on core AX research questions and open problems. Follow academic and industry perspectives on: (1) evaluation frameworks and metrics for agent behavior/performance, (2) human-agent interaction design principles, and (3) safety and alignment challenges in agentic systems. Focus on insights that could position me for industry research roles.

Key Insights

  • The binding constraint in agent evaluation has shifted from task quantity to verifier quality — and this is where a cognitive scientist can make a distinctive contribution. The field has moved past "can agents complete tasks?" to the harder question of "how do we know what the agent actually accomplished, and what counts as meaningful progress?" A new benchmark, NanoGPT-Bench, makes this concrete: coding agents like Codex, Claude Code, and Autoresearch recover only 9.3% of human R&D progress, succeeding primarily at hyperparameter tuning while failing at algorithmic innovation. Researchers trained in construct validity, task decomposition, and measurement theory are unusually equipped to articulate what a well-designed verifier needs to capture — and to distinguish benchmark performance from genuine task capability.

  • [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash)

  • Multi-agent orchestration is rewriting the fundamental unit of human-agent interaction, and classical HCI frameworks are not built for it. The dominant deployment pattern is no longer one human interacting with one agent; Google's Antigravity 2.0 demo — an OS built in 12 hours using 93 parallel sub-agents, 15,000+ requests, and 2.6 billion tokens — is the concrete stress test of what this looks like at scale. Jeff Dean explicitly frames Gemini 3.5 Flash as an engine for sub-agents that "collaborate, run high-frequency iterative loops, and solve real-world problems at scale." The AX research questions this raises — how does a user build an accurate mental model of a fleet? how is trust calibrated across heterogeneous agents? what transparency is owed when no single agent is responsible for an outcome? — have no established answers, and psychologists who can formalize them will define what the field becomes.

  • [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash)

  • Internal rogue-agent safety is a live research priority at major labs, not a theoretical concern — and the methodology for studying it is underdeveloped. A researcher spent a month embedded at Anthropic specifically stress-testing systems designed to detect whether internally deployed agents could "go rogue," producing extensive appendices and transcripts. This signals that labs are treating internal agent control as a distinct safety surface, separate from model alignment or adversarial robustness. For an AX researcher, awareness of this specific program — and the institutional seriousness behind it — is the kind of field-specific knowledge that signals genuine engagement rather than surface familiarity in an industry research interview.

  • [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash)

  • Independent auditing methodology for agentic systems is essentially nonexistent, and the founding of Guidelight signals the field is beginning to reckon with that gap. The Anthropic embedded-researcher exercise allowed the company discretion over what could be redacted — meaning the safety assessment was self-administered by the entity being assessed. Steven Adler's announcement of Guidelight, a new AI safety standards organization, suggests the field is starting to recognize that internal evaluation is insufficient. For a researcher with a psychology background, this maps directly onto decades of work on the limits of self-report data and the methodological requirements for independent behavioral observation — a framing that translates naturally into research contributions on what rigorous third-party agent auditing would actually require.

  • [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash)

Emerging Patterns

  1. Benchmark headline scores and task-specific capability data are telling systematically different stories, and the gap matters for anyone designing evaluations. Gemini 3.5 Flash gained 70 Arena points and reached the top score in its price tier — strong competitive positioning signals. But on the same model: a 61% hallucination rate on an omniscience task setup, a 5.5x cost increase over the prior Flash generation, and the cross-model finding that agents recover only 9.3% of human R&D progress. Arena leaderboards measure relative competitive positioning; benchmarks like NanoGPT-Bench measure something closer to absolute task capability. The field has not resolved how to communicate this distinction to practitioners, and an AX researcher who can articulate what each metric type actually measures — and for whom — fills a genuine methodological gap.

  2. [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash)

  3. The formalization of agent architectures — defining agents as executable, inspectable, stateful, and governed — is creating the structural vocabulary that AX evaluation frameworks will need to build on. The shift from informal agent descriptions to these four properties isn't just taxonomic; it directly determines what can be measured and what kinds of verifiers are even possible. Threads at Google I/O emphasized that scaling agent benchmarks now depends less on adding tasks and more on improving verifier quality, with concrete benchmark work (BenchGuard, OSWorld-Verified, SWE-bench Verified) representing current attempts to operationalize this. For an AX researcher, the "governed" property is the most underdeveloped — it points toward questions about oversight, accountability, and user control that sit squarely in behavioral science territory.

  4. [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash)

Dissenting Views

  • The prevailing industry framing presents internal agent safety stress-testing as serious, rigorous safety research; an embedded researcher conducting that very work describes it differently. The official narrative — labs are actively stress-testing internal agent safety systems — is accurate as far as it goes. But the same researcher who spent a month doing this work at Anthropic explicitly characterized the activity as an "exercise rather than a formal audit," specifically because the company retained discretion to redact sensitive findings. This is a methodological disagreement, not a factual one: both accounts describe the same activity, but they disagree about what that activity proves. For an AX researcher, this tension is directly analogous to the distinction between self-administered and externally validated assessments — and naming it precisely is likely to signal methodological sophistication to industry research teams evaluating your fit.
  • [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash)

Read & Act

What to read

  • [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash) — Read the sections covering verifier quality and the NanoGPT-Bench findings together. The argument that benchmark scaling now depends on verifier quality over task quantity is compressed in the analysis above; reading the full source will surface the specific benchmarks (BenchGuard, OSWorld-Verified, SWE-bench Verified) and the reasoning behind this shift, which is precisely the kind of methodological framing you'll want to discuss fluently in an industry research interview.

  • [AINews] Google I/O 2026: Gemini 3.5 Flash, Omni (NanoBanana for Video), Spark (background agents), and Antigravity 2.0](https://www.latent.space/p/ainews-google-io-2026-gemini-35-flash) — Read the internal rogue-agent safety section and the "exercise vs. audit" caveat in sequence. The tension between them is the kind of nuanced safety discourse that separates researchers who understand the field's actual debates from those who've only read press releases — and it maps directly onto psychological methodology questions you already know how to reason about.

What to do

  • Draft a 1-page research position statement that translates "verifier quality" into behavioral science terms. Specifically: what does a well-designed verifier need to capture that current ones miss, and what concepts from your training — construct validity, behavioral coding reliability, interrater agreement — map onto those gaps? This document doesn't need to be published; its purpose is to give you a concrete, differentiated answer when an industry interviewer asks "what would you bring to our agent evaluation work that an ML engineer wouldn't?" Anchor it to the NanoGPT-Bench finding: if agents succeed at hyperparameter tuning but fail at algorithmic innovation, what does that distinction reveal about how we should define task success criteria in the first place?

  • Map the four formalized agent properties — executable, inspectable, stateful, governed — onto your existing research methods. Take "governed" as your focus, since it's the most underdeveloped and most directly relevant to behavioral science. Identify one or two methods from your PhD or postdoc work (e.g., protocol analysis, think-aloud studies, compliance paradigms) that could generate empirical evidence about what "governed" means from a user perspective — what oversight mechanisms users expect, what control they need, and how they respond when those expectations are violated. This exercise will help you articulate a concrete research agenda for AX rather than a general interest in the field.

Source Articles