.svg)
What is Custom LLM Judge Metric?
A Custom LLM Judge Metric uses a language model as an evaluator: you supply criteria and a grading prompt, and the judge model scores another system's outputs on subjective qualities that hard metrics miss--tone, helpfulness, policy fit, or domain correctness.
What this looks like in production
Teams score chatbot replies for empathy, RAG answers for groundedness style, or agent trajectories for plan quality when BLEU or exact match are the wrong yardstick.
What teams usually do about it
- Version judge prompts like code and calibrate against human labels.
- Watch for judge bias and position bias in pairwise setups.
- Combine judges with deterministic checks for tool use and schema validity.
Related Giskard articles
Score subjective quality with Giskard
Use Giskard checks and custom judges to mix deterministic gates with LLM-as-judge criteria in one suite. See Giskard OSS v3.
Further reading
Authority reference: Judging LLM-as-a-Judge (arXiv).
