# Making LLM judges worth trusting

IBY Tokyo · Product Manager Intern

May 2026 - Jul 2026

IBY is a 20-person AI startup in Tokyo. Its product, ProcMine, watches screen recordings of people doing back-office work and writes the process manual for them.

I spent the summer on the quality team, reporting to the CTO. In May, nobody could answer two simple questions about ProcMine. Is each stage any good? And what does it cost to run?

This is how we got to answers you can check, and the things that broke on the way.

**Where my work sat in ProcMine**

## A scoreboard for every stage

Before this, pipeline changes shipped on gut feel, and spend was a surprise at the end of the month.

I built ProcMine Eval, the dashboard the team checks after every CI run. Each run lands about 15 seconds after CI finishes, with a suite of 30 metrics across every stage of the pipeline. It has taken in 180+ runs since the end of May, and it became the bar the four engineers shipping through CI have to clear before a release. When a baseline drifted, the Product Lead got an audit report from me saying where and by how much.

The first version was a static site with the data baked in, so it went stale whenever nobody rebuilt it. I split the data from the build. A small server takes the CI webhook, verifies it, writes the results to S3 and refreshes only those files. The page fetches them when you open it, so it never rebuilds and it's never behind.

## The meter was lying

A dashboard is only as honest as the data under it. The cost numbers were the first to prove it, because they disagreed with themselves.

I pulled the same spend three ways from Langfuse: trace totals, daily metrics and raw observations. They didn't match. Our traces were charging a cost for calls that had failed with a 429 and returned nothing. About $1,875 of the $11,818 we'd tracked had never been spent.

**The same month of spend, reconciled**

After the fix, all three sources agree to the cent, in every window.

Then I built the cost view the team uses now, split by tenant and stage, where clicking any point opens the exact trace behind it. I also tried hard budgets per tenant through a LiteLLM gateway, which stops a tenant with a 429 the moment it runs over.

## Cheaper judges

ProcMine grades its own manuals with LLM judges, and they ran on top-tier models. Once spend was visible per model and per tenant, the argument about what the pipeline actually needed could be settled with numbers, and I won it. I made the judges answer pass or fail instead of giving a score, benchmarked cheaper models against Opus, moved some checks to open-source classifiers and rerouted the rest. The monthly bill fell 57%, from $9.0K to $3.9K, and it held into the second month. That's about $61K a year.

**Monthly LLM spend**

| | |
|---|---|
| Before | $9.0K |
| After | $3.9K |
| Month two | held |

Then we read what the cheaper judges were flagging. Three flags in four were wrong.

**What the cheaper judges flagged**

A judge that is wrong most of the time only looks cheap. Someone still pays to catch its mistakes, just later.

## Reading the manuals

My manager scoped me to the LLM judges. The deterministic gates and the generator needed buy-in from the whole company, so they stayed out. I started where error analysis says to start, by reading.

I pulled 22 manuals that had ground truth behind them and read each one like a trace. About 11 kinds of failure kept coming back, and the judges we had caught very few of them.

The worst was overfitting to the recording. The manual copies the one case it saw on screen and turns it into the rule.

**One line from a real manual**

SHIP-203 was the shipment in the video. Follow this manual with the next shipment and you are stuck.

Other manuals dropped the rule behind a decision, made up approval steps nobody had, leaked pipeline leftovers to the reader, or taught OCR mistakes as fact.

## Eleven judges you can prove wrong

I wrote one judge per failure type, 11 in all, in two tiers. The first tier checks a manual against ground truth. The second runs on any manual. Each judge has to name the step it flags and quote the exact line.

A judge that can't be proven wrong isn't measuring anything. A full sweep across three test themes cost about $7.

## Checking the checkers

Next I graded the judges. I built a small review tool that shows one finding at a time with its step inline. It has a second mode for noting failures the judges were never told about, and it translates the Japanese source to English in place. Then I graded 95 findings by hand against ground truth.

The grading caught something no metric would have. Every "empty branch" finding was made up.

**Why every branch looked empty**

The eval read one level too high, so the judges only ever saw blank branches. I fixed it along with five other bugs in the harness.

Tooling can lie to you politely. Reading its output is how you find out.

## Nine experiments

After the midterm, the ask was precision and cost at the same time. A bigger model everywhere would buy one and lose the other, so the answer was routing.

**How a manual gets judged now**

The three judges that escalate are the ones where a false flag hurts most: hallucination, PII and overfitting.

Two changes did most of the work. Making judges reason before they flag cut hallucination false positives from 90 to 24. Sending overfitting to Sonnet lifted its recall from 0.70 to 0.95.

**Hallucination false positives**

| | |
|---|---|
| Flag first | 90 |
| Reason, then flag | 24 |

Over nine experiments, F1 went from 0.39 to about 0.52.

**F1 against the hand-graded answers**

The hybrid is now the release gate in CI. It reports rather than blocks until the judges agree with people at least 0.70 of the time. The labelled set to measure that exists now.

## Where people pushed back

My first version of the judges was its own package, sitting beside the codebase. The code review pushed back, and the reviewer was right. I moved it under the shared judges registry and the runner the team already had, so there's one config and one phase with two engines inside. Shipping into a shared codebase, rather than next to it, was one of the better lessons of the summer.

Its first live run flagged a Chrome workflow that a manual described and the operator never did. All three Haiku votes agreed and the verdict was HOLD. It took 79 seconds and cost $0.23.

The data science team sent manuals organised by workflow stage (intake, then decision, then documentation) instead of by business process. Judges anchored to one process at a time couldn't map onto that.

**The same work, cut two ways**

One of their chapters mixed invoice approval, bank reconciliation, budgets and vendor payments.

So I built comparison tables that put the ground-truth process, our manual and theirs side by side, and sent the team and my manager a short note with concrete examples. A confusing blocker turned into a clear ask.

Later my audits and my manager's disagreed. A shared OCR cache had been overwritten by a later run, and the new run was clean, so it passed every existing check while hiding the original failure rates. I wrote it up without blame and proposed a check that compares the cache's model with the one the run says it used.

I also turned the audit into a shared skill anyone can run in about 15 minutes, with about ten times fewer S3 calls than before.

## Agents that ask first

The second half of the summer went to IBY's next product for NTT Data: agents that do back-office work with a person in the loop.

Before designing anything, I interviewed Japanese back-office staff across time zones and tested each hypothesis against usage data. That became the requirements and success metrics that scoped ProcMine's automation pilot for enterprise clients, NTT Data among them.

We started from what operators actually do. I mapped 1.45M recorded events across 18+ apps. The biggest task everywhere was checking one record against another before acting.

**Where the back-office work went, across 18+ apps**

| | |
|---|---|
| Checking records | 30 to 35% |
| Everything else | the rest |

Nearly every decision an operator made had one of six shapes. Those shapes became the checks the agent runs.

**The six questions behind almost every decision**

I designed the approval flow and built the prototype. The agent drives Excel and SmartHR by itself for the routine steps. The two steps it can't undo need an approval token as a typed argument. Only the approval gate can mint one, each token works once, and it's tied to the exact change the person saw. Call the step without one and it fails, and a test proves the bypass fails.

Discipline can slip. Types don't.

**What the person sees when the agent stops**

Gates never approve on their own. If nobody answers, the run holds.

In 55+ test cases it made 0 false accepts. We treated a false accept as a bug in the test harness rather than a bad run, so the harness was fed known-bad outputs before anything trusted it.

## What I'd keep

The most useful findings were the unglamorous ones: a serializer making up findings, a cache overwritten without a trace, two teams cutting the same manuals in different ways. None of them showed up in a summary. Each one took reading the actual data.

Both halves of the summer came down to one question. A judge decides whether a manual is good enough, and a gate decides whether an agent's action is. The work was making that decision something you can check.