Labs2026Side project · 12 min

The research tool that tells you what to measure before you test.

Testify is a usability-testing tool I designed and built for myself, then turned inside out: from a study builder, which helps you run a study you have already designed, into a study planner, which starts from what you want to learn and works out the success metrics, the session, the prototype fixes and the fair comparison before anyone sits down. The hard part of research at myKaarma was never running the session. It was deciding what "better" meant.

Role: Designer, researcher and builder, solo, with AI pair-building
Context: Testing B2B dealership tools with parts staff, advisors, technicians and BDC agents
Timeline: Phase 1 earlier in 2026 · phase 2 [confirm] · phase 3 October 16–18, 2026
Built with: React, then one self-contained HTML file and a small Cloudflare Worker · Gemini for planning and summaries · PostHog, Supabase, Google Sheets for results · Cloudflare for hosting, publishing and team sign-in

Plan a study Skip to the decisions

TTestify/ Parts UI · pre-approved vs recommendationsSample plan · the first real study

1 · Describe the study

Talk it through or type. What are you testing, what are you unsure about, what would make you confident?

🎙 Dictate · in the real tool

A · Current — filter tabs at the top; pre-approved line nested in a green concern card; inline DMS and part-not-found errors.

B · Proposed — status filters as chips; pre-approved shown as its own section with "Approved at write-up".

In Testify a model drafts this from your words and asks 3–5 questions first. Here, the plan is the real one from the Parts study.

The plan appears here: research questions, success metrics with targets and the activity that measures each, predictions, the things only you can fix, and the session.

01 · Plan

The problem

Off-the-shelf tools assume a world I don't work in.

Maze, UserTesting and the rest assume Figma prototypes, consumer participants and a researcher with time to design every study by hand. My work breaks all three. My prototypes are coded, with real states and real errors, and teams that test seriously have moved the same way; click-tracking on a Figma frame cannot see whether someone actually finished. My participants are parts-counter staff and service advisors who have minutes, not hours. And the part that took most of my time was never running the test; it was deciding what success meant, which activity would measure it, and whether the two versions were even a fair comparison. That is exactly where existing tools give no help. They start at "add a task".

PrototypesCoded, not Figma framesReal states, real errors, scored inside the prototype

ParticipantsBusy professionalsParts counter staff with minutes to spare. Every task has to earn its place

The researcherMe, part timeNo dedicated research function; the judgement has to be in the tool

Decide what "better" means before you look at the data. The habit I brought from conversion work, turned into software. The planner above is that sentence, running.

02 · Build

Phase 1

I built the study builder first. It was the wrong tool.

The first Testify was about a thousand lines of React: a working, Maze-style builder with six test types, a block editor, a participant mode joined by a short code, shared storage, and a results view with a first-click heat map. I reused the patterns participants already know so the tool would not become the thing being tested, and kept it to a single file so it could run anywhere. It fell short in three ways that all pointed the same direction: upstream.

Testify's studies home: the Parts UI study and the four steps. — Where it landed: one study, four steps. Phase 1 had only the last two.

Success was self-reported.A participant who gave up and clicked "Completed" looked like a success.

It could only run a study I had already designed.The thinking still happened in my head, at five o'clock.

It had no idea of versions.Comparing A with B meant two studies and a pair of eyes.

Try the first gap

Said done versus actually done.

Task: source the pre-approved part for this line.

Diagnosis · approved at write-upno parts · labor only

nothing to source

CV axle boot · awaiting decision44305-TBA-A01 · qty 1

Brake pads · pre-approved45022-SNA-A00 · qty 1

Try finishing the task, or just say you did.

How Testify reports it

Reached the right state0

Said done, but hadn't0

Wrong-line edits0

The tagged element is the real "done". "Said done, but hadn't" became its own group in the results, and one of the most useful signals in the tool.

02 · Build

Phase 2 · What I decided

Move the tool upstream.

The pivot was changing the tool's job. Instead of helping me run a study I had designed, Testify now helps me design it, starting from the goal. Each rule below is one piece of research judgement I had been carrying in my head, made explicit and handed to the tool.

Rule 1

Every metric names the activity that measures it.

You cannot leave the Plan step with a metric that nothing measures. Each success metric carries a pass direction and the activity that produces the number, and in phase 2 a target too ("at least 80% of lines classified correctly"). Anything unmeasured is flagged.

In phase 3 targets became optional. Comparing versions already says which is better; a target only answers whether the winner is good enough, and a junior designer shouldn't have to invent one to start.

This is the habit from conversion work: decide what "better" means before you see the data, so the data cannot talk you into a story afterwards.

ShippedResearch methodOwn practice

Success metrics: classification ≥80%, unprompted mention ≥60%, correct first click ≥70%, wrong-line edits ≤1, time to fix ≤60s, each tied to an activity. — The Parts plan: six metrics, each with a target and an activity. The unmeasured ones would be flagged.

Rule 2

Tasks finish when the prototype reaches the right state.

A prototype task is complete when the tagged element has actually been reached, not when the participant says they are done. You tried it above. Testify tags the prototypes itself: drop in an HTML file, it finds the buttons and lines each task refers to, marks them, and hands you back a tagged copy.

ShippedMeasurementPhase 1 failure

The Build step: eight activities in order, from Welcome and First impression through Parts tasks, Preference and Wrap-up. — Eight activities, set up from the plan. The prototype tasks score themselves.

Rule 3

Comprehension questions run on the first version only.

By the time a participant sees the second version they have already learned the concept, so asking "what is this?" again measures memory, not the design. Five-second impressions, classification and first-click run only on the first version a person sees; task performance runs on all of them. It is the kind of judgement a researcher carries in their head, and the kind that gets skipped at five o'clock.

ShippedExperimental designOwn practice

Predictions: each hypothesis with the metric that will support or refute it. — Predictions written before the session, so they can be marked supported or not afterwards.

Rule 4

Flag what differs besides the design.

The planner checks whether the versions differ in more than the thing being tested (different data, missing error states) and lists those as must-do fixes before testing, separated from optional ideas, with Done and Skip states and a count of what is left.

On the very first real plan this caught a flawed A/B before any participant's time was spent. That alone paid for the rebuild.

ShippedValidityParts study

Changes only you can make: must-do items such as Give both versions the same data and Seed a part-not-found error, plus optional ideas like a variant C. — Must-dos and maybes, kept apart. The must-dos gate the test.

Rule 5

Anything safe to automate is automated, with undo.

I wanted my time on judgement, not setup. Session setup, prototype tagging and pulling results happen on their own, every automatic action can be undone, and the end of every step has a next-step banner so there is never a question of what to do now.

Participants need no account, and since phase 3 everyone gets the same link, with the design order shuffled for each person. One Publish button puts the study online, and Update keeps the same link after every change; a single downloadable HTML file stays as the fallback. Answers go to PostHog, Supabase or Google Sheets, services I already control, and read keys never leave my browser.

ShippedAutomationOwn usage

Share a link: choose where results go, download the participant page, put it online, paste the address to get participant links. — Before phase 3: four steps to a remote study, including downloading a file and hosting it. Now it is one: Publish.

03 · Run

The Parts study

The first real plan caught the flaw before the test did.

I used Testify's first real plan on a live question at myKaarma: which version of the Parts screen works best for parts staff, and is the new pre-approved section understood as different from recommendations? Three research questions, each with the activity that measures it and a target decided in advance.

Research question

Measured by

Target

Can they tell pre-approved from recommendations?

Line-type classification on the first version seen

≥ 80% correct

Does recognising it change what they act on first?

Correct first click; unprompted mention after five seconds

≥ 70% first clicks · ≥ 60% mention

Which version is faster with fewer wrong edits?

Prototype tasks on every version, scored automatically

≤ 1 wrong-line edit · ease ≥ 5.5/7

The planning stage surfaced four things before a single session ran.

UnfairThe comparison wasn't fair yet.A showed errors but no recommendations; B the reverse. Both now get the same data.

Vague"Understood" needed a behaviour.Defined as sourcing pre-approved work right away, so it could be measured.

RiskTwo near-identical green chips."Approved at write-up" and "Approved". The plan added a task that asks how each line got approved.

IdeaA variant C.Neutral colour for pre-approved, to separate "the structure helps" from "the colour confuses".

Results: [add once sessions are done: which version won, what surprised you, what changed in the shipped Parts screen].

04 · Results

What exists now

A planner, not a timer.

A four-step tool in one self-contained file, eight activity types including self-scoring prototype tasks, remote sessions through one published link with results landing in PostHog, Supabase or Google Sheets, heatmaps and click paths for every prototype task, and an end-to-end test suite covering sessions, scoring, export and merging. I was also its first user, so I ran a heuristic audit against Nielsen's ten principles on my own work and fixed what it found.

✓Contrast raised from about 3.3:1 to about 5:1 on secondary text.

✓Focus stays put. It used to jump to the top after every click.

✓Undo on deletes, and a visible save status.

✓Jargon I had invented, renamed. ~~Counterbalance version order~~ → alternate which version people see first. ~~Blocks~~ → activities.

✓Must-dos split from ideas; next-step banners; auto setup with undo; export; checked hypotheses; drag-to-tag.

Before · 3.3:1Optional. Testify names it from your goal if you leave this empty.

After · 5:1Optional. Testify names it from your goal if you leave this empty.

The same helper text, before and after the audit. WCAG AA asks for 4.5:1 on body text.

Shipped as

One file

Plan, Build, Run, Results in about 470 KB, plus a small Cloudflare Worker for publishing and sign-in. No accounts for participants.

Activity types

Including prototype tasks that score themselves and line classification.

Before any session

1 unfair A/B

Caught by the planner on the first real study.

Own confusion

7 fixes

From the heuristic audit of my own tool.

Open

[Parts results]

Participants reached, which version won, time saved per study versus planning by hand.

Method: own use across two real studies; a heuristic audit against Nielsen's ten principles; an end-to-end test suite. Figures in brackets are open until the Parts study reports.

05 · Phase 3

Three days in October

From my planner to a tool someone else can run.

Over three days, October 16 to 18, Testify went from a planner only I could drive to a hosted tool with its own look, voice and participant flow. The bar I set: a junior designer, new to research, should be able to plan and run a fair study with it.

What I changed

Fewer questions, fairer versions.

✓Setting up. Hosted online, with storage so old studies are kept. One shared participant link instead of one per person. The code on GitHub, so I can work from the cloud or my PC, not only my Mac.

✓Fair prototypes. A current, a proposed and a neutral pre-approved version of the Parts prototype, all with the same realistic data and no developer-only alerts, with the differences made easier to see.

✓Simpler for a junior designer. An audit of the whole tool for someone new to research. Plain words, one question per screen, and questions that didn't help taken out. Targets made optional, and called targets, not goals. The tagging screen gone, and screenshots allowed when there is no prototype.

✓The flow. It starts with a chat: the study's name, what you want to learn, and whether you have designs. Prototypes come after the questions. A viewer flips between versions and takes screenshots, a Check step confirms PostHog is recording, and Next buttons say where they go.

✓Look and voice. A branding deck to find the voice, then my own mix of its directions: sharpened type, warmth and cream, and colours with one job each: pink only for the main action, violet for links and highlights, amber for what needs attention. Folds for long sections, and my gestalt rules: related things grouped, labels left, actions right.

✓Feedback and control. A Save button with a saved state, confirmation and Undo when marking something done, and Remove that looks like remove.

✓Participants. Answers required by default, with Optional per question. A Back button, zoom on pictures, and questions that say which version they're about. Tasks that finish only on the right action or "I'm stuck", a first click that ignores stray clicks, and a warning when an ad blocker stops the study. Confidence and ease questions stay off unless asked for, and the task card lights up when the next task starts, because a prototype that reloads looking the same reads as stuck.

✓Publishing. One Publish button gives a link participants can open, and Update republishes after a change with the same link. No more downloading a file and uploading it to a host.

✓Team sign-in. Coworkers sign in with their work email and get the team's connections automatically: no setup codes, no keys to paste. Studies stay private until shared, and each one can go to the whole team or to named people, as view or edit; viewers can duplicate to make their own.

✓Results that say what to do. A heatmap over the real prototype for each task, the most-clicked controls with wrong ones marked, and the common click paths. The AI summary reads clicks and answers together and comes back with what to change first, themes with word-for-word quotes, and what to look into next.

The look

Each colour has one job.

Warm Studio: Space Grotesk throughout, a cream page with a faint graph-paper grid, plum ink, crisp 6px corners and almost no shadow. This page wears it too.

PinkMain actions only. As a highlight it read as an error. #E0245E

VioletLinks, selection, focus and highlights. #6A44E8

LilacSmall soft surfaces: chat replies, selected tabs. #EADFF7

AmberNeeds attention, never an error. #A64B0C

ApricotSmall decoration, like the heading underline. #FF9E64

Cream and plumThe page and the ink. #FBF4EC · #3B1D3F

What I learned

The things I hadn't thought to ask for.

Most of these came up while pair-building with Claude, raised at the moment they mattered rather than in a list at the end.

MethodShuffle the design order.Otherwise one design is always seen first, and that alone changes how people react to it.

MethodTargets answer "is it good enough?"Comparing says which design is better. With 5–8 people a small gap can be chance, so a target shows whether the winner is worth acting on.

MethodGive a "Not sure" option.Without it, a lucky guess counts as a right answer.

MethodShow the AI the screens first.Follow-up questions then match what's on screen. Ask early whether designs exist, but let the goal lead, not the prototype.

FairnessOnly the thing you're testing should differ.Each difference is marked Intended or Fix, and the fixes happen before anyone takes the study.

FairnessAlerts and placeholders count too.An error message in one version, or a broken image, changes behaviour for reasons that have nothing to do with the design.

HostingA link beats a file.A downloaded participant page is a snapshot that has to be re-uploaded after every change. A published link updates in place.

HostingAnalytics has two kinds of key.The project key only sends data and is safe in a public page. The personal key reads results and stays private.

HostingAd blockers stop analytics, and some fake it.Some answer with an empty reply instead of failing, so a real check looks for real replies.

DesignGive each colour one job.Pink caught the eye, but as a highlight it read as an error. So pink became the action colour only, highlights went violet and warnings amber.

DesignA voice can be a short rule set.Fact first, then one line of why. Two sentences at most, no exclamation marks, never grade the designer.

DesignFocus shouldn't look like selection.A violet focus ring after zooming read as "chosen", so focus got its own style.

TeamSign-in is the setup.Copying a setup code between browsers was the step coworkers stumbled on. Signing in now carries the connections with it.

TeamAn empty browser mustn't win.A first sign-in from a fresh window would have saved a blank setup as the team's. Each connection now moves as a whole, and only fills gaps.

PrivacyRecord what a control says, not what was typed.Click labels come from the button or the field's name, so no participant's input ends up in the results.

AnalysisClicks say where, quotes say why.Either alone sends you guessing. Read together, they turn into a change you can make.

Habit

Check the versions differ only in what you're testing before anyone takes the study.

Habit

Try the study yourself in Preview, then once for real in a clean browser.

Habit

Publish once, then press Update after every change. The link stays the same.

Habit

Wait a few minutes before deciding data is missing.

Habit

Write down what future sessions should know, so the tool's context doesn't live only in my head.

Differently

What I'd change with hindsight.

I'd have started with the planner. Phase 1 was a competent copy of tools that already exist; the thing that was actually missing was upstream of all of them, and I could have seen that from my own notes if I had read them as research.

I'd have split the Plan page into steps sooner. It is long, and the heuristic audit confirmed what I already felt every time I scrolled it.

What it doesn't do yet: it won't find participants for you, so you bring your own. Heatmaps cover each task's first screen only. And the AI summary has still to be judged on a real study, not a sample one.

And I'd have shown it to another researcher a month earlier. Building the planner's rules forced me to make my own judgement explicit, and explicit judgement is the kind someone else can argue with.