Case 03 · AI tooling
From my project to the whole design team
Two AI assistants I built for my own project: a page tracer that turns a live screen into a prototype, and a hand-off assistant that writes the documents engineering builds from. Both were reworked so the whole design team can use them.
Illustrative · NDA
The product and the framework are under NDA, so the names here are generic and the pictures are redrawn in the same fictional supermarket chain as the other cases. The tools, the decisions and the lessons are mine.
- Adoptedthe page tracer, now used across the design team
- Walkthroughsevery procedure in the hand-off opens the prototype at its step
- Mapsroles, rules and flows drawn as pictures you can drill into
Problem
Two jobs fill a designer’s week: rebuilding screens that already exist as prototypes, and writing a hand-off that engineering can actually follow. The first comes with every product that’s already live; the second with every feature, on a brand-new product too.
I built an AI assistant for each, inside my own project. Both worked for me. The harder question came next: would they work for a designer who isn’t me, on a project that isn’t mine?
The UI Inspector had already made that trip to the rest of the team. These two took more work, because each had my project’s details built in: its paths, roles, components and colours.
The page tracer
Point it at a page of the live product, and it rebuilds that page as a prototype using our framework’s stack: the same structure, content, states and behaviour, in the product’s own visual values.
Our prototypes never depend on a product’s code: each one is a copy in plain HTML, CSS and JavaScript, and the tracer is what makes that copy.
Most new work on a product that already exists starts from a screen that’s already there. Rebuilt by hand, that screen takes hours, and a guessed colour or spacing looks just as finished as the real one. The tracer measures instead of guessing, and builds with the framework’s own components, so the result is ready to change.
Draft first
One fast pass gives a complete prototype of the page, ready to share. It’s the default, and often all a feature needs.
Refine one region
Point at the part that’s off. It measures that region against the live page, fixes it and checks again.
Complete, with proof
On request, it measures every state and compares the prototype with the product, property by property. It says what that will cost before it starts, and waits for a yes.
It renders, it works
Two checks come before any measure of fidelity: the page draws, and its controls respond. Then it looks at the screenshot itself.
It stops instead of guessing
When it can’t see the page, it stops and says why, rather than inventing what it can’t measure.
Every failure, a rule
Each run that went wrong became a written rule it reads when something looks off: 69 so far.
How I taught it to stop guessing: teaching an agent to see like a designer
The hand-off assistant
It writes the hand-off from the feature’s knowledge base, the matrix, the logs and the meeting notes, and puts it inside the prototype, where people already are.
Every feature gets two documents. What’s New is a guide for stakeholders that opens with who can do what on each screen. Dev and QA is the spec engineering and QA build from: numbered requirements, the user stories under each topic, and the business rules. Both state the current rule, never its history.
It works the same on a brand-new product: everything comes from the knowledge base and the prototype, not from a live app.
Nobody opens a document in the middle of a demo. So the documents open the prototype.
Inside the prototype
A panel beside the prototype
The documents open in a docked panel next to the live prototype, never over it, so a rule is read against the screen it describes. It remembers the document and the place.
Walkthroughs you can click
Every procedure is a walkthrough. Each step opens the prototype where it happens, as that role and in the state it needs, with a Click here marker on what to press.
Made to be read
Every topic opens with an In short line. An outline says where you are, Find highlights in place, Back undoes every jump, and a filter shows only what’s still open.
Stories where they belong
Under each topic, the user stories it fulfils, found through its requirements, so a developer reads the rule and the story together.
Copy it anywhere
Copy a document as Markdown, or as rich text for a document editor, or open it in a tab of its own.
A check before it ships
One run checks the whole set: steps that open a state the prototype can’t stage, questions pinned to screens that don’t exist, links that land nowhere, a role spelled two ways.
Flows and rules as pictures
A workflow view
One switch draws every flow as a picture: a lane per role, each hand-off on its line, a rejection dashed on its way back. Each step in the picture opens the prototype there.
Maps first
Every document opens with maps drawn from the feature’s rules: the process across roles, the status lifecycle, who can do what, a decision table, and how work moves across the prototypes.
Drill down
A role or a status opens its own page inside the map: who it receives from and hands off to, its steps flow by flow, and its rules, each linked to the passage where it’s written.
Questions on the screen
Every open point that needs an answer from a room becomes a question, written to be read out loud. One file feeds two places that can never disagree: a pin on the screen and a board for the work item.
A pin on the control
The question sits on the button or field it’s about, and several on one control share a pin, with a count. A link to a question opens the screen with that question showing.
A card that stands on its own
An id that never changes, so it can be quoted in meetings; a kind (open, confirm or attention); an owner role, never a person; the context behind Show context; and a link to the passage in the spec.
A board for every work item
One page lists every question of a ticket or an epic, a tab per related prototype, opening on what’s still unanswered. Each card opens its screen and its source in the spec, and every pin links back to the board.
Answers that land everywhere
Typed on the pin or on the board, it’s the same answer. Export answers hands them to the step that applies each one to the matrix, the logs, the prototype and both documents.
Nothing is deleted
An answered question stays on the board, read-only, until the prototype shows the answer. Then it moves to History, and its pin goes away.
From my project to the team
Page tracer
- Jun · A pipeline9 agents trace a live page into a prototype.
- Aug · One agentCut back to one agent that stops instead of guessing.
- Sep · Its lessons travelMy failure log becomes tracing rules the whole team can read.
- Oct · For the teamI hand it over as one package, and every designer on the team gets it.
Hand-off assistant
- Jul · The first specOne agent writes the developer spec from the matrix, the prototype and the meeting notes.
- Aug · Questions on the screenOpen points become questions pinned to the control they’re about, and the stakeholder guide opens with who can do what.
- Sep · Flows you can clickEvery procedure becomes a walkthrough, and the answers flow back into every document.
- Oct · Any projectMaps and the workflow view arrive, and it’s rebuilt so another project can run it with its own components, roles and colours.
What made them portable
-
01
Project values out of the code
Paths, roles, components, wording and colours moved into one profile and one theme per project. The acceptance test fits in a sentence: another project’s profile and theme, no code changes.
-
02
The rules in one place
The rules moved out of the agent and into a skill it reads only when a task needs them. The agent just runs the steps, and every project reads the same rules.
-
03
Generic pieces
The panel, the walkthroughs, the pictures and the questions became generic components, with each project’s colours coming from its theme.
-
04
Handed over as one package
I bundled the tracer, its rules and its lessons into a single package, so it could reach the rest of the team as it is. Now every designer on the team has it, without installing anything.
What people changed
The documents got better when other people used them.
- A manager found the questions board overwhelming. It now shows less at once.
- Answered questions used to disappear. Now they stay on the board with their answer, and then move to its history.
- Not everyone reads a whole topic, so each one now opens with a single plain sentence.
- In September I put together a ten-minute talk for the other designers, from the matrix to the questions on the screen.
AI layer
Where AI assisted
- Writing the code. I built both tools with an AI coding agent. The rules and the design are mine.
- Drafting the documents from the matrix, the prototype and the meeting notes.
- Turning failed runs into written rules, for both tools.
Where I overruled it
- Change notes. The documents state the current rule, never its history.
- Warning marks. They are kept for real open points only.
- Guessing. When the tracer can’t see the page, it stops and says why.
Outcome
- 80+questions pinned to the screens they’re about, since August
- 110+walkthroughs in my hand-off documents, each step one click from the prototype
The page tracer is now used across the design team, and the hand-off assistant is ready for any project.
The idea, the design and the code are mine; teammates helped bring them to the team. There are no team-wide usage numbers yet.
What I got wrong
- Rules inside the agent. Every new rule made it longer and more tied to my project.
- Deleting answers. An answered question vanished, and its history with it. Now it’s kept.
Next, I want to see the hand-off assistant run on a second project, with that project’s own profile and theme.