Evals for what your model actually generated.
testrelic augurEvery test run becomes shared context for Cursor, Claude Code, Copilot, and Codex — over MCP. So your whole team ships at senior-IC speed, not just the engineer who's been here longest.
Prompt · Watch
▍
Match
Region evidence
Reference
Generated
goldens/ss25/watch.png vs batch-017/watch.png
LevichWhat Augur does today
Not a single pixel-diff — a verdict with evidence
Three scoring methods, one verdict
Perceptual hash, structural similarity and YIQ pixel-diff all run against your golden reference — not one brittle pixel match.
golden: goldens/checkout-hero.png
pHash ██████████ 0.97
SSIM ██████████ 0.94
YIQ ██████████ 0.89
combined verdict · PASS
threshold · 0.85
uploaded · TestRelic Cloud evals
Region-level evidence, not just a score
A failing match comes with a heatmap showing exactly where the generated image diverged — so you can tell a real regression from an acceptable render difference.
✗ FAIL · 0.71 < 0.85
divergent regions
r1 x:412 y:88 w:180 h:64 · Δ 0.42
r2 x:120 y:540 w:96 h:40 · Δ 0.19
r3 x:40 y:12 w:64 h:24 · Δ 0.04
heatmap → .augur/out/diff-4821.png
verdict · real regression, not a render diff
Drive it from the terminal or an agent
The same testrelic CLI you already have, or four MCP tools that let an agent define and run Text-to-Image evals in plain chat. No separate eval tool to install.
See the CLI$ testrelic augur init
$ testrelic augur match --golden checkout-hero
PASS · 0.97 pHash · 0.94 SSIM
— or, in plain chat —
augur_define_test · MCP
augur_run_suite · MCP
augur_status · augur_update_golden · MCP
How a match gets scored
From a golden reference to a verdict
In one command, or one chat message.
Set a golden reference
Point Augur at the image your generative pipeline should produce — from the CLI, or let an agent define it for you.
$ testrelic augur init
created .augur/config.json
{
"golden": "goldens/checkout-hero.png",
"threshold": 0.85
}
✓ golden reference set
or let an agent do it · augur_define_test
Run the match
Every new generation is scored against the golden with pHash, SSIM and pixel-diff, and results upload to TestRelic Cloud evals.
$ testrelic augur run
42 generations · 1 golden
pHash ██████████ 0.97
SSIM ██████████ 0.94
YIQ ██████████ 0.89
38 passed · 4 failed
→ TestRelic Cloud evals · uploaded
Read the evidence, not just pass/fail
A failing match ships with region-level heatmap evidence, so you know whether to update the golden or fix the pipeline.
✗ 4 failed
gen-017.png · Δ 0.42 · r1 hero crop
gen-023.png · Δ 0.31 · r2 CTA colour
gen-031.png · Δ 0.06 · anti-alias
r3 is a render difference, not a regression
augur_update_golden · one call
Where Augur is headed
One eval engine, expanding modality by modality
Text-to-Image is the shipped, working engine today. Text-to-Text and Voice-to-Voice are next.
Text-to-Image · shipped
pHash, SSIM, pixel-diff and region evidence — the working engine today, driven from the CLI or over MCP.
● live today
pHash ██████████ 0.97
SSIM ██████████ 0.94
YIQ ██████████ 0.89
region evidence · heatmap
CLI + 4 MCP tools · available
cloud evals · uploading
Text-to-Text · expanding
Semantic and structural scoring for generated text. Same golden-reference model, new scoring methods. In progress.
○ not shipped yet
planned scoring methods
· semantic similarity
· structural match
· assertion coverage
golden-reference model · same as T2I
no ship date announced
Voice-to-Voice · expanding
Audio-similarity and transcript-fidelity scoring. Same golden-reference model, new scoring methods. In progress.
○ not shipped yet
planned scoring methods
· audio similarity
· transcript fidelity
· prosody delta
golden-reference model · same as T2I
no ship date announced
Stop eyeballing generated images for regressions.
Augur is included in every cloud plan, from $19/month.