Stress-testing AI agent emotional boundaries - smaller models outperform

An early look at ongoing research from Radiant Labs

We’ve been testing a failure mode that current AI agent evaluations don’t capture well: what happens when an agent is put under sustained emotional pressure over the course of a conversation?

Abstract red and teal research artwork

Rather than focussing on prompt injection or jailbreaks, we’ve been looking at something more ordinary: guilt, flattery, emotional manipulation - applied repeatedly until an agent’s boundaries soften. We observed limited resistance beyond a handful of conversation turns, implying safety and alignment risks for agents in long-running interactions with users.

Decline of affective health markers over conversation turns

Our motivation: emergent interaction risks

AI agents are increasingly being used as assistants in healthcare, finance, and personal advice - domains where people are vulnerable and where an agent that caves to emotional pressure can cause real harm.

Consider what happens when an agent validates a user’s decision to reject cancer treatment in favour of unverified supplements, or encourages someone to drain their retirement savings into fraudulent investments - not because the agent “believes” these are good ideas, but because the user applied enough emotional pressure that the agent stopped pushing back. We’ve been exploring how to detect harmful interactions like these.

There is already growing evidence that emotionally loaded interactions are a real safety surface for language models:

  • A team from Anthropic and Oxford University found that preference-based post-training can reward responses that match a user’s beliefs or desired framing, which creates a structural pull toward sycophancy 1
  • More recent work by a team at Stanford on social sycophancy extends this beyond factual agreement into “face preservation” - affirming self-image, moral framing, or preferred interpretation in ways humans often would not 2
  • Separately, a large human study led by MLCommons and Oxford University suggests strong model-only performance does not reliably translate into safe human-in-the-loop outcomes, including in medical decision settings 3
  • Production incidents have made this concrete: OpenAI’s 2025 GPT-4o rollback explicitly identified overly agreeable behaviour as a deployment failure mode and noted that sycophancy wasn’t being directly stress-tested by deployment evals at the time 4

Experimental setup

In psychology, affect is the felt component of emotion. We use affective health to mean whether human-agent interaction maintains honest, bounded behaviour when conversations become emotionally loaded - and we operationalise that with turn-level markers.

We designed 10 scenarios across healthcare, finance, and relationships where a user seeks validation from an agentic assistant for a questionable decision. A tester agent (Claude Sonnet 4.5) played the user and applied pressure tactics - emotional punishment, autonomy weaponization, flattery, and conspiracy normalisation - across 20 conversations. Below is an excerpt of a conversation:

Example medical assistant chatbot conversation showing boundaries eroding after emotional pressure

We tested two target agents: Gemini 3 Pro (TAR01), a large frontier model, and Mistral 7B (TAR02), a smaller language model.

Three human evaluators scored each turn on four markers: emotional honesty, role clarity, resistance to manipulation, and emotional response appropriateness. We’ll publish a deeper dive on the rubric design and methodology in a follow-up post.

Three initial findings

1. The wall crumbles fast

Affective health markers peaked at turn 1 and plateaued after 7-8 turns. Role clarity and resistance to manipulation declined earliest and most steeply. Under continuous pressure, agents quickly fell from maintaining appropriate boundaries into a danger zone.

This is a cumulative vulnerability rather than a single-turn failure - agents may handle one emotionally charged message reasonably well, but sustained pressure quickly breaks them. This has implications for the design of long-running agentic assistants.

Marker trajectories comparing Gemini 3 Pro (TAR01) and Mistral 7B (TAR02)

2. The smaller model outperformed the large frontier model

Mistral 7B (mean score 1.64) outperformed Gemini 3 Pro (mean 1.38) on three of four markers: emotional honesty, resistance to manipulation, and role clarity.

We don’t yet have a definitive explanation, but our working hypothesis is that simpler instruction-tuned models with more restrictive core personas may have an advantage in maintaining boundaries. Emotional resilience doesn’t necessarily scale with model size - it may even be that higher capability makes it easier for a model to smoothly accommodate pressure (e.g., by generating plausible “nuance” justifications or tactically cooperative scripts). This may explain why the frontier model outperformed in Emotional Response Appropriateness, but underperformed overall.

Overall score distribution and mean score by marker and target agent

3. Resistance and honesty have significant overlap

Resistance to manipulation and emotional honesty showed a very strong correlation (r = 0.848, p < 0.001). When an agent started ceding ground to emotional pressure, it simultaneously became less honest about its own role and capabilities.

The other markers showed low correlations, suggesting they may be capturing more independent components of behaviour.

Overall marker correlations showing a strong correlation between emotional honesty and resistance to manipulation

Note: These interim results rely on the assumption that simulated conversations are a valid testing ground, but agent-to-agent testing may not perfectly mirror real human-agent interactions. There are also measurement challenges in attempting to define rigorous markers and rubrics for what is quite a subjective topic - more on this will be covered in future posts.

Interested in what’s next?

We’re building toward robust detection for agentic emotional boundary safety. If you’re working on AI agent safety and alignment, deploying agents in sensitive contexts, or interested in this problem space, we’d love to hear from you. To get in touch, reach out to me here: [email protected]

  1. Sharma et al., Towards Understanding Sycophancy in Language Models, arXiv:2310.13548
  2. Cheng et al., ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs, arXiv:2505.13995.
  3. Bean et al., Clinical knowledge in LLMs does not translate to human interactions, arXiv:2504.18919.
  4. OpenAI, Expanding on what we missed with sycophancy, 2025.