Case 05 · AI tooling
Teaching an agent to see like a designer
An agent that traces live product screens into prototypes, cut back to one agent and taught to stop instead of guessing.
Fictional client · NDA
The real product is under NDA, so this page shows none of its screens. The scenario, names and data are fictional. The decisions and the lessons are mine.
- 9 to 1agents, first pipeline to packaged agent
- 0.4 pxworst ink gap across four toolbar icons, down from 7 px
- 69failures written down as rules
Problem
New work on the platform starts from screens that already exist. A prototype of a change is only honest if everything around it matches the live product.
When a tracing agent can’t see the page, it guesses, and a guess looks finished. My first pipeline, in June, fell back to raw API calls when the browser failed, and I’d told it to say which mode it was in. The one-agent version in July fell back from browser to script to raw API calls. By its own later notes, it did that silently and produced low-fidelity guesses.
Constraints
- A page changes with the role. Traced as whoever was signed in, one list lacked a header button and showed different data from the role we needed.
- The target is our team’s in-house prototyping framework, with its own components and icons. Traced values are translated into the framework’s components and tokens, except colour, which stays exact.
- Context is finite. The agent re-read its whole manual every time it resumed, so every extra line cost something on every run.
Key decisions
-
01
One agent, and a hard stop when the browser fails
The first version was a pipeline of agents. The one I packaged in August is a single agent that stops when there is no browser, and says why, instead of degrading.
- 9 agentsJune. A pipeline, with a raw-API fallback.
- 2 agentsJuly. One agent with copy and sync modes, plus a capture helper.
- 1 agentAugust. Packaged. No browser, no run.
Agent definitions in the repository at each step. The fallback was cut, not tuned. -
02
Parity is measured, not judged
In late July, parity became a mechanical check, built on a mechanism borrowed from a teammate’s browser-testing pipeline. Probes name the elements, a unit-tested comparer reads 47 style properties on both sides, and a missing element blocks the run. It still missed toolbar icons I could see were wrong, so now each icon is sized by its ink, the mark the eye actually compares.
Before · one box size for all
7 pxworst gap (27%)
After · sized by the ink
0.4 pxworst gap
Generic shapes, not the product’s icons. Dashed: the app’s ink. Solid: the prototype’s glyph. Schematic; the gaps are measured. -
03
Check it renders before measuring it
Two checks run before any parity pass: does the page render, and does it work. Parity logged the missing table only as two size rows among 181.
- CaptureThe live page, in a real browser, as the right role.
- BuildWith the framework’s own components.
- Renders?Look at a screenshot before any number.
- Works?Every visible control does something.
- ParityProbes and a report. Full trace only.
Dashed: runs only on request, after both checks pass. -
04
A quick draft by default
One pass over the landing view and whatever opens from it. It ends with the app and the build side by side and one question: good enough to show, or keep going? A full trace states its cost first and waits for a yes.
Default
Quick draft
One pass. Aimed at minutes.
The landing view, plus the modals, drawers and tooltips it opens.At most two questions, role first; the rest are stated assumptions.On request
Full trace
Every state, measured. Hours.
Other routes, full probes and a parity report.One page: about 1.1 million tokens, about 40% of them the manual re-sent (agent‑reported).Dashed: the agent’s own figure, with no usage log behind it. -
05
Every failure becomes a rule
Every real failure gets written down with its cost and the rule that prevents it. The agent only reads the table when a run goes wrong, so it doesn’t weigh on every run.
Failure Consequence Rule A fallback chain: browser, script, raw API calls. It almost always ran the weakest step, silently. One mode, a real browser. No browser, hard stop. Hunted for overlays by clicking around. A popover and some tooltips shipped untraced. List overlays from code and page; reconcile the counts. Read one element, not what was inside it. A status colour was guessed from a screenshot. Take the deepest element that carries the visual. Three rows, paraphrased.
In the room
One full trace · as a system administrator
The agent traced a 29-column table page in full-trace mode, the default at the time. Its timed phases alone add up to 115 minutes, not counting a gate of 8 questions and 3 parity runs. The report listed 181 style defects and 2 structural rows, both hover states it couldn’t measure. The first screenshot I saw had no table: one CSS specificity bug had collapsed it to zero height. I told the agent we had been at this for hours, for a result short of the effort.
A report can be precise about a page that isn’t there.
AI layer
Where AI assisted
- Writing the code. I built the agent with an AI coding agent, including a comparison toolset of about 3,300 lines, plus 84 tests. The design calls are mine. The agent drafted most rules from our failed runs.
- Capturing pages, running probes and tracing each difference to its source.
- Keeping the failure log.
Where I overruled it
- Colour. In a trace the app’s exact colours win: "fidelity in everything, colour too" (translated).
Where I pushed back
- Setup chores. Asked to run scripts in my own terminal, I asked: "can’t you do this from here?" (translated). It could, for all but one step.
- Naming. I read a count of "safe" rows as things that were fine; the agent renamed that bucket "defect".
What I got wrong
- Starting with 9 agents. Two orchestrators (copy and sync) repeated the same setup. One agent with two modes replaced them.
- Full trace as the default. By its own account, the agent spent about 55 minutes on states nobody had asked for.
- Trusting the parity count. Its 181 style defects described a page with no table.
Outcome
- 9 to 1agents, first pipeline (June) to packaged agent (August); 2 from July in between
- 164 to 0blocking parity rows on the first tuned page, over 6 rounds. The same day I spotted wrong icons no probe measured; tightening the check brought back 16 defects.
- 69failures written down as rules. A teammate turned them into 57 general tracing rules, and today the tracer itself, every lesson included, is used across the design team: how it got there.
No traced page reached full parity with probes.
Not measured: time saved, quick-draft run time, whether the tests pass today, or use after August. Token figures come from the agent itself. There’s no usage log.
What I’d do next
- Log token use per run, so the stated cost is measured, not reported.
- Run the 84 tests in a clean copy, and time one quick draft.
- Show a stakeholder a quick draft beside the app, to learn what "good enough" means to them.