Briefing: AX Research

For a cognitive/experimental psychology PhD transitioning into industry AX research roles — tracking evaluation frameworks, human-agent interaction design, and safety challenges in agentic systems.

Key Insights

• The "agents do more, humans work harder" paradox is likely the defining open problem for AX as a research field. Industry observers have noticed a striking pattern: as AI agents take on more work, people in AI-adjacent roles report higher cognitive load, not lower — a phenomenon characterized as "token anxiety." This isn't a minor inconsistency; it suggests that current agentic systems may be redistributing rather than reducing cognitive burden, potentially by shifting humans into high-stakes monitoring and error-recovery roles. For a researcher with a cognitive psychology background, this is a natural research program — studying attentional demands, trust calibration, and supervision costs in human-agent collaboration — and it maps directly onto the kind of question industry teams building agent products are struggling to answer.

• [AINews] Humanity's Last Gasp

• The real design surface for AX is the scaffold around the model, not the model itself. A growing body of practitioner argument holds that agent performance depends at least as much on the surrounding infrastructure — task-specific harnesses, memory management, tool output control, workflow design — as on which frontier model is underneath. This reframes the AX research object: instead of studying model capabilities in isolation, researchers should be asking how scaffold design choices shape user mental models, trust, and cognitive load. For a psychologist entering this space, this is genuinely good news — questions like "how should memory be structured for human-interpretable agent behavior?" and "what tool output formats reduce user cognitive load?" are unanswered and directly draw on experimental and human factors expertise.

• [AINews] Humanity's Last Gasp

• Production agent teams have converged on a canonical four-dimensional evaluation framework — accuracy, latency, cost, and safety — and are running evaluation continuously, not at the end. The industry standard is no longer a benchmark score at release; it's a continuous SFT → evaluate → RL → evaluate → deploy loop in which every training epoch produces evaluation data and decisions about next steps. This matters for your positioning: "evaluation frameworks for agent behavior" in industry contexts means designing multi-dimensional, workflow-integrated assessment — not one-time academic benchmarks. A researcher who can articulate how to add experiential and interaction quality dimensions to this four-variable framework (e.g., measuring monitoring burden, user trust calibration, or recovery experience) would be offering something most ML engineers on these teams cannot.

• Accelerate LLM post training with W&B Serverless SFT

• [AINews] Humanity's Last Gasp

• Existing agent benchmarks are saturating, and the next evaluation frontier is shifting toward knowledge-work domains and human-expert comparison. SWE-Bench — the dominant coding agent benchmark — is effectively saturated, with newer benchmarks (SWE-Bench Pro, Mythos) approaching the same fate. The field is moving toward evaluations like GDPval, which rates AI performance against human experts across broad economic sectors. This benchmark crisis is an opening: the tools for measuring whether an agent produces good outcomes for a human user — as opposed to technically correct outputs — are underdeveloped, and designing such measures is exactly where psychology methodology (validity, construct definition, human performance baselines) has direct transferable value.

• [AINews] Humanity's Last Gasp

Emerging Patterns

1. Both the infrastructure and tooling conversations converge on the same underlying problem: the gap between demo performance and production reliability is not primarily a model quality problem. The practitioner debate over scaffold design and the W&B tooling workflow both point to the same root issue — agents behave well in controlled conditions and degrade unpredictably in deployment. This "productionization gap" is not purely technical; it involves human factors including how users set expectations, recover from errors, and calibrate trust over time. The research niche this creates — understanding why agents that score well on benchmarks fail or create friction in real workflows — is one where cognitive and experimental psychology offers genuine comparative advantage over ML engineering backgrounds.
2. [AINews] Humanity's Last Gasp

3. Accelerate LLM post training with W&B Serverless SFT

4. Safety is named as a first-class optimization dimension alongside accuracy and cost, but the field lacks agreed-upon methods for measuring it in agentic contexts. At the product level, safety appears as a line item in evaluation dashboards and as a motivator for specialized fine-tuning (e.g., domain-restricted models like GPT-5.4-Cyber, or physical safety reasoning in Gemini Robotics-ER 1.6). But neither source specifies how safety is operationalized or measured in continuous evaluation loops — it's named without being defined. This is a concrete open problem, and it aligns with the alignment challenge literature: for AX researchers, the question is how to design safety metrics that are legible to both engineers and end users.

5. [AINews] Humanity's Last Gasp
6. Accelerate LLM post training with W&B Serverless SFT

Dissenting Views

• The prevailing industry framing treats human-agent workload friction as an engineering problem with identifiable solutions — but there's a credible alternative framing that it's a deeper structural tension. The W&B tooling narrative implies that better infrastructure (smoother SFT-RL loops, lower latency, integrated evaluation) resolves the friction between agent capability and human experience. The Latent Space roundup raises a more unsettling possibility: that the friction is not incidental to current tooling, but reflects something fundamental about how humans relate to systems that act autonomously on their behalf — a difference in emphasis rather than a direct contradiction, but one with major implications for research design. If the tooling-solvability view is correct, AX research should focus on reducing interface friction; if the structural-tension view is correct, the research agenda is more about understanding and managing irreducible supervision costs and trust dynamics. Which framing you adopt will shape what questions you pursue.
• [AINews] Humanity's Last Gasp
• Accelerate LLM post training with W&B Serverless SFT

Read & Act

What to Read

• [AINews] Humanity's Last Gasp — Read this in full as a rapid scan of where industry attention is currently concentrated across all three AX pillars (evaluation, interaction design, safety). Pay particular attention to the scaffold-vs-model debate and the linked discussions on task-specific harnesses — these will give you vocabulary and reference points that signal field fluency in interviews and research conversations.

• Accelerate LLM post training with W&B Serverless SFT — Watch this as a concrete orientation to how production agent teams actually evaluate and iterate, rather than for conceptual novelty. The specific SFT → evaluate → RL → evaluate → deploy workflow is becoming standard industry vocabulary, and understanding how "evaluation" functions inside this loop — as a continuous decision-making tool, not an endpoint — will help you translate your academic methodology skills into terms industry teams recognize.

What to Do

• Draft a research position statement that translates the four-dimension evaluation framework (accuracy, latency, cost, safety) into AX terms. Specifically, propose what a fifth dimension — call it "experiential quality" or "interaction burden" — would need to measure, how you would operationalize it given your psychology background, and what methods (think-aloud, experience sampling, physiological monitoring) could capture it in a real agent workflow context. This exercise does two things: it forces you to connect your existing expertise to industry evaluation practice, and it produces a concrete artifact (a one-pager or blog post) that demonstrates differentiated positioning relative to ML-background candidates.

• Use the "agents do more, humans work harder" paradox as the anchor for a targeted literature review that bridges cognitive psychology and AX. Identify what your field already knows about monitoring costs, automation complacency, and supervisory control (there is substantial literature from aviation human factors and process control) and map those findings onto what's described in current agentic deployments. The goal is to arrive at 2-3 testable hypotheses about when and why agentic systems increase rather than reduce cognitive load — this would constitute a genuine research contribution to AX and a compelling intellectual pitch for industry research roles at companies building agent products.