The workflow I was changing

Before large language models, a conversational assistant worked by classification. Someone typed a message, an intent classifier trained on labeled examples scored it against every intent the assistant knew, and the top prediction decided what happened next.

The interesting part was the failure case. When two intents came back close together, say order_status at 58 percent and track_package at 55 percent, the assistant had no basis for choosing. Explicit disambiguation logic existed to catch those low confidence cases and resolve them.

The fix lived upstream, in the training data. Two intents scored close because their labeled examples overlapped, so someone had to find the overlap and reassign the examples that belonged elsewhere. That review workflow is what I redesigned, and for enterprise clients it ran as a standing operational routine.

What was at stake

By 2024 the industry was moving toward generative models, and HumanFirst had to move with it. The platform's future depended on being credible in that new world.

The pull ran in two directions at once. The revenue funding that move came from enterprise accounts sitting on years of labeled data, trained pipelines, and teams whose weekly routine ran straight through the intent workflow. Those teams had built real fluency in the existing patterns. A migration that asked them to relearn their own daily job was a migration they could refuse.

Before: the comparison happened in your head

Disambiguation was something you performed, and the interface gave it nowhere to live. You navigated to Intents, opened one, and read its labeled examples in a list.

Open username in the screen below. It holds a run of zip code utterances that were labeled into it by mistake, and nothing on the screen marks them as a problem. There is no signal that username and zip_code are colliding, no ranking of which intent they collide with most, and no score on any single example. A reviewer carried the whole comparison from memory, one intent at a time.

Three directions

I explored three shapes for the workflow before committing. All three were drawn against the same task: finding which examples inside zip_code belong to another intent, and moving them there. The rail below switches between them, each one running.

The bet

I gave disambiguation its own place in the product, with a mode, a pinned intent, and a surface shaped by the comparison it exists to support.

The core move is the two column compare. On the left sit the examples currently labeled into the pinned intent that look like they belong to the candidate intent. On the right sit the candidate's own examples. Moving an example across the divide reassigns it. The comparison the old workflow asked people to hold in memory now happens on screen.

• Disambiguate is a mode

  A mode banner, a pinned intent, and a close control mean the reviewer always knows which intent they are sharpening and how to leave.

• The conflict is ranked

  The left rail lists every intent overlapping the pinned one, ordered by how much they overlap, with a meter on each. The reviewer opens on the worst collision and works down.

• Every example is scored

  Each confusing example carries its own similarity percentage in a colored band, so confidence is legible per row and a reassignment is a judgment made with the evidence in view.

The migration lived in the details

The surface was a different shape, so every gesture inside it had to feel like something these teams already did. Moving an example between columns reuses the move behavior from the old labeling flow. Hover shows the direction a row will travel before the click commits it. Apply stays quiet until there is a change to apply, so a reviewer can open a conflict, read it, and leave with their data untouched.

Browsing a conflict had to be free, and writing to a client's labeled data had to be deliberate. That split is the whole trust model of the screen.On keeping people in control

What changed

These are the outcomes I can stand behind, and they are qualitative. I had no instrumentation on this workflow, so there is no before and after on task time or error rate to show here. The signal we had was the absence of the thing we were most afraid of. No account asked to stay on the previous surface, and the rollout passed without a wave of questions about where the workflow had gone.

What I would measure next

Given another cycle, I would instrument the decision itself. Four things I would put in place:

• Time from opening a conflict to applying the first reassignment, as a proxy for how much of the comparison the screen is carrying.
• Reassignments per session, and how many get reversed later, as a read on whether the similarity scores are calibrated to how people actually judge these examples.
• How many intents a reviewer touches per session, to check that the overlap ranking really does pull people to the worst collisions first.
• Classifier confidence spread after a cleanup session, which is the closest thing to an outcome measure for the work itself.

What I take from this

Evolving a live system is a migration problem before it is a design problem. The first question is what the people already inside the old surface can carry across. How good the new screen is comes second, and it only counts once they arrive.

You can make something dramatically better while keeping it the same. Familiar mechanics are the budget you spend a redesign against. Holding onto the move gesture, the labeling vocabulary, and the navigation model bought the room to change the entire shape of the task.

The best outcome in enterprise work is sometimes that nobody notices. A silent rollout reads as a quiet result on a portfolio page and as a loud one inside the company. This project is the reason I now design the migration path at the same time as the destination.