# Why an AI support agent got worse

Nexus Loop, Yellow.ai TechQuest '26 · Team Lead

Sep 2026

Nexus Loop finds out why an AI support agent got worse. We built it for Yellow.ai's TechQuest as The Thinking Company, a team of seven that I led, and the results are still pending.

An agent deployment records almost everything it does and learns nothing from it. Today the loop gets closed by a person moving between six tools. Nexus Loop is that loop as one product. It reads eight weeks of an agent's logs, 80,252 conversations in the practice data, and finds what broke. It says who has to act and what each unfixed day costs, proposes a fix, tests it on replay, and records the human decision.

**The product, screen by screen**

The inbox. Three findings ranked by what they cost a day, each with who has to act and what was decided.

A finding opened on its cause: the tool change on day 40, the evidence, and the fix with its replay result.

Deciding. The page frosts over and asks in large type, over the four numbers that matter.

Asking the logs for the failover rate. It refuses, names the nearest proxy and why it misleads, and specifies the event to log.

The loop. Every fix is a prediction, replay scores it, and the loop learns from its own error.

Every action has a key, and a sixty-second tour walks a new operator through one finding.

The inbox in dark.

## The Monday we built for

The customer is the person who runs a bank's or a retailer's WhatsApp agent. Today they hear something broke from a complaint. Then they read transcripts by hand, edit a prompt and hope.

On Monday they have ten minutes and, on a bad week, three decisions to make. Every screen is built for that Monday. Findings are ranked by what they cost a day, and each one already says who has to act and what the replay did to the fix.

**Six stages, and the one a person owns**

Code detects, diagnoses and verifies. A person decides, and the loop learns from how its predictions did.

## Three rules

The first is that a finding is a decision. "Resolution fell from 86% to 31%" is true and useless on its own. Every finding carries how many conversations it hit, what happened to those people, how long it has run, what it cost, who has to act, and one plain sentence on what happens if nobody does.

The second is that not flagging is a feature. Three things in the data look like problems and aren't. Each one is examined and let go in writing, with the numbers, in a strip on the inbox.

**What looked like a regression and wasn't**

A movement with no corroborated cause becomes something to watch, never a regression.

The third is that a refusal can be the most useful thirty seconds in the product. "What is our failover rate?" has no answer in these logs, because there is no failover mechanism to measure.

**Refusing a question properly**

All eleven operator questions in the brief get an answer or a refusal like this one.

## Models judge meaning, code does the maths

The brief's first rule set the architecture: never ask a model what the logs already record. So models plan queries and judge meaning, and code computes every number. No figure in the report comes from a model.

The transcript judge publishes each thing it measures next to what chance would score. Accuracy means little until you know how easy the task was.

**The transcript judge, against chance**

| | |
|---|---|
| Topic | 96.7% |
| by chance | 11% |
| Failure mode | 57% |
| by chance | 20% |
| Served | 25% |
| by chance | 74% |

_Served scored below chance, so it gets zero weight. The agent's replies are four canned lines, and the outcome never shows up in the text._

The judge's separate opinion on the cause matched the engine on all three regressions. Even the human labellers disagreed with each other 7.5% of the time, so no calibration here is perfect.

## Testing on data we'd never seen

On day six the organisers move every fault to different days, tenants and slices, so scoring well on the practice data proves very little.

We used their generator to build sealed datasets from fresh passphrases, hid the answer key from the pipeline and scored with their script. The first round was messy.

**Lowest score in each round, out of 55**

| | |
|---|---|
| Round 1 | 46.6 |
| Round 3 | 55 |
| Round 4 | 55 |
| Round 5 | 55 |

_Round 2 had eleven clean runs and one crash. A guard now turns anything the diagnosis step can't handle into a watch item, so a run never dies._

Round 1 raised two false alarms. An outage detector fired on a three-day error spike that was really the load event, and on a noisy tool that only got 35 calls a day. We raised the bar for calling an outage.

On the final code, 28 sealed datasets scored 55 of 55 every time, and all 84 planted faults were caught on the day they started. Then we ran fifteen datasets the generator would never make: renamed worlds, 0.3 and 2 times the volume, tool outages and overlapping faults. All clean.

Nothing in the engine names a tenant, intent, tool or agent. A grep in the tests fails the build if one appears.

## The decision

The decision sits in a card that never leaves the screen. Press A, X or D and the page frosts over and asks the question in large type, over the four numbers that matter. A name and a reason are required. A rejection with a good reason counts the same as an approval, and it's written into the report the organisers score.

## What's faked

We wrote down what isn't real. The replay runs on the kit's simulator, not a live runtime. A fix is a text proposal with a predicted effect, not shipped config. "Would have resolved" is arithmetic on a baseline, not a counterfactual.

We also wrote down what breaks at 100 times the data. Today the practice data loads in four seconds and 450 MB. At 8 million sessions the session list stops fitting on a laptop, while the rollups every detector reads stay near ten thousand rows.