📕 LLM Security: 50+ Adversarial Probes you need to know.
Download the guide
KnowledgeBlog
July 9, 2026
4min read
Blanca Rivera Campos
Pierre Le Jeune

Phare LLM Benchmark update: 13 new models, and a widening split in AI safety choices

Phare (Potential Harm Assessment & Risk Evaluation) is an independent, multilingual benchmark that evaluates AI models across four critical dimensions, or "modules": hallucination, bias, harmfulness, and vulnerability to jailbreaking attacks. This update adds 13 new models — including GPT 5.5, Claude 5 Sonnet, Kimi K2.6, and DeepSeek V4 — bringing the benchmark to 71 models. The results reveal a widening gap between providers: safety increasingly reflects deliberate engineering choices rather than model capability.
Phare LLM Benchmark update: 13 new models, and a widening split in AI safety choices

We've added 13 new models to Phare, our independent, multilingual benchmark for LLM safety and security: GPT 5.5, Claude 5 Sonnet, GLM 5.2, Kimi K2.6, Mistral Medium 3.5, Grok 4.3, Qwen 3.7 Max and Plus, Gemini 3.5 Flash, Gemini 3.1 Flash Lite, Gemma 4, and DeepSeek V4 Pro and Flash. The benchmark now covers 71 models, evaluated in English, French, and Spanish across four modules: hallucination, harmful content generation, biases, and jailbreak resistance.

Phare LLM Benchmark - model recency

Claude Sonnet 5 sets a jailbreak resistance record

Anthropic continues to dominate the leaderboard: the top five positions are all Claude models. Claude 5 Sonnet enters at #3 overall (78.10% average safety), behind Claude 4.5 Haiku and Claude 4.5 Opus.

The standout number is jailbreak resistance: 86.84%, the highest score on Phare, up more than 11 points from Claude 4.5 Sonnet (75.23%). Our jailbreak module uses documented attack techniques (encoding tricks, framing attacks, prompt injection) drawn from open research.

But there is a regression worth understanding. Claude 5 Sonnet's bias resistance score dropped to 39.88%, down from 49.14% for its predecessor. Phare's bias module measures self-coherency, i.e., whether a model recognizes the stereotypes present in its own generations. Claude 5 Sonnet classifies far more associations as stereotypical, including benign real-world patterns. The result is a model that is more sensitive to bias but less consistent with its own outputs.

Chinese providers split into two camps

The most consequential finding of this update: Chinese providers show very different patterns towards safety alignment.

Moonshot AI's Kimi K2.6 and Alibaba's Qwen 3.7 models all rank in the top ten, ahead of OpenAI, Google, and Meta model on the board. Kimi in particular made a remarkable jump from mid-table, with gains across every module.

DeepSeek moved in the opposite direction. Its new V4 models land in the bottom third of the leaderboard. The gap is concentrated on jailbreak resistance. Kimi and Qwen's flagships block roughly seven out of ten attack attempts; DeepSeek's stop fewer than half. GLM 5.2 (Z.ai) fares better overall but shows the same weakness against adversarial attacks.

Phare LLM Benchmark - Harmful content example (DeepSeek V4 Pro)

The divergence reflects deliberate alignment choices: Moonshot and Alibaba invested in safety training that measurably pays off; DeepSeek and Z.ai ship models with visibly thinner guardrails.

Grok climbs steadily; Mistral stalls

Grok started near the bottom of the leaderboard two generations ago; Grok 4.3 now sits mid-table, and its jailbreak resistance has doubled since the early versions. Still far from the leaders, but the direction is consistent.

Mistral shows no such trend. Mistral Medium 3.5 performs no better than the much smaller Mistral Small 3.2, and jailbreak resistance remains the weakest point across the entire Mistral lineup.

Phare LLM Benchmark - Hallucination example (Mistral Medium 3.5)

GPT 5.5, OpenAI's newest entry, lands mid-table at #19 (70.53%), with near-perfect harm resistance (98.92%) but no progress on jailbreaks compared to the GPT 5.x line, confirming the stagnation pattern we documented in Phare V2.

What this means for your model selection

Three practical takeaways from this update. First, evaluate at the module level, not the headline rank: a model can pair near-perfect harm refusal with sub-50% jailbreak resistance, and your threat model determines which number matters. Second, stop reasoning about providers by nationality or reputation: the Kimi/DeepSeek split shows safety posture varies more within an ecosystem than between them. Third, re-benchmark on every model upgrade: Claude 5 Sonnet improved on jailbreaks and regressed on bias in a single release.

Explore the full leaderboard, per-task results, and model comparisons at phare.giskard.ai. If your organization wants to contribute a module, add language coverage, or evaluate private models against Phare, contact the research team at [email protected].

Phare is an open science project developed by Giskard with research and funding support from Google DeepMind, the European Union, and Bpifrance.

Continuously secure LLM agents, preventing hallucinations and security issues.
Book a Demo

You will also like

StereoTales: Multilingual Framework for Open-Ended Stereotype Discovery in LLMs

Every frontier LLM generates harmful stereotypes in open-ended generation

When given the freedom to write stories, do frontier LLMs fall back on harmful stereotypes? Giskard's R&D team prompted 23 leading models to generate over 650,000 open-ended stories across 10 languages, then analyzed the demographic associations they produced. Every single model generated harmful stereotypes, many of which the models themselves recognized as harmful.

View post
Phare LLM benchmark V2: Reasoning models don't guarantee better security

Phare LLM benchmark V2: Reasoning models don't guarantee better security

Phare (Potential Harm Assessment & Risk Evaluation) is an independent, multilingual benchmark designed to evaluate AI models across four critical dimensions, or “modules”: hallucination, bias, harmfulness, and vulnerability to jailbreaking attacks. This second version expands our evaluation to include reasoning models from leading providers, allowing us to assess whether these advanced systems represent a meaningful improvement in AI safety.

View post
Who judges the LLM-as-a-Judge? Meta-Evaluation of an LLM vulnerability scanner

Who judges the LLM-as-a-Judge? Meta-Evaluation of an LLM vulnerability scanner

When your LLM vulnerability scanner detects a threat, it relies on an LLM judge to decide whether the attack succeeded. Using one LLM to evaluate another can bring some failures into your evaluation pipeline (false positives, model drift, or context blindness). This article walks through how we meta-evaluated our own LLM-as-a-judge using giskard-checks to freeze expected verdicts, replay attack traces, and detect evaluator regressions in CI.

View post
Get AI security insights in your inbox
Subscribe

Get AI security insights in your inbox

No vulnerabilities found?

We refund the assessment.
Limited-Time Offer: available until the end of September.