Designing what a support agent says when it doesn't know

Abstract mark for the support agent project

01 / problem

The welcome message was promising something the menu could no longer do

The client's internal platform had a support chat that told users it could answer questions about the product or help them report an issue.

Then the documentation library migrated to a new system and the connection broke. The menu that remained offered login support, feedback, feature requests and bug reports. None of them is a path to asking a product question.

So the greeting kept making a promise the interface had quietly stopped keeping. The message and the menu contradicted each other, and nobody had noticed, because the failure was silent.

A beta connector could technically answer product questions, but its raw output carried embedded video and unformatted source lists, which do not belong in a chat widget. Its matching was keyword-based and narrow while the input field accepted open-ended text. The interface was promising more flexibility than the retrieval layer could deliver.

02 / decision one

Treat it as a content-surfacing problem, not a reconnection ticket

The obvious framing was that a connector broke, so engineering should reconnect it. That work was already owned and already underway.

I argued the more useful gap was downstream: nobody had defined how migrated content should be retrieved, formatted, or presented once it returned, or what the agent should do when it could not find a confident answer. Reconnecting a pipe to an undefined interface restores a worse version of the same problem.

So I scoped and authored the PRD around the UX of content surfacing rather than the plumbing.

The option not taken: waiting for the connector to be restored and designing against whatever it returned. Lower risk, and it would have left every formatting and failure decision to be made by default.

03 / decision two

Specify two scenarios instead of one, because retrieval reliability was unknown

Keyword matching precision was still unclear when the spec had to be written. A single target design would either assume a retrieval quality the team could not guarantee, or lower its ambition to match a floor nobody had measured.

I wrote paired content scenarios: a best-case target for what great looks like, and a realistic MVP baseline. Same content, two display treatments, both specified.

Whichever way retrieval quality landed, the team had a design already agreed on and did not lose a cycle renegotiating scope.

The option not taken: designing the ideal and handing it over. Cleaner to present, and it would have stalled the moment engineering came back with a number.

04 / decision three

Specify the failure state before the success state

For an agent answering questions from a knowledge base, the most consequential moment is not a good answer. It is a confident wrong one.

The PRD defines behavior for zero results, for retrieval failure, and for unrecognized input, each with a next step rather than a dead end. It also requires low-confidence responses to be visually distinct, so an answer the agent is unsure of never presents as fact.

The research supported designing for repair rather than for coverage: shallow decision trees, an explicit escape hatch, disambiguation instead of "I don't understand," and a hybrid of conversational input with guided controls, so new users get direction and experienced users can just type.

The revised welcome screen came out of the same logic. It leads with content-help prompts and demotes feedback, bug and feature options to secondary weight, so the greeting and the menu finally say the same thing.

The option not taken: specifying the happy path and treating no-match as an edge case for later. Every shipped AI support agent that confidently invents an answer made that choice.

05 / outcome

What moved

The recommendations were taken for beta integration and the PRD is in active use by the team as the reference spec for the enhancement.

It has not shipped, and the reason is partnership logistics rather than the spec. An external vendor partnership introduced integration hurdles that sit outside the design. The document is still what the team is building against.

Alongside it I ran a platform adoption survey across the intern cohort, spanning ten teams.

06 / reflection

I was designing a better door to a room nobody enters

The survey undercut my own project, and that is the most useful thing it did.

I spent the summer specifying how documentation should surface in chat. Not one surveyed user had ever opened the documentation library. Half rarely touched the platform at all, and given a choice between it and a general-purpose AI agent, most reached for the general one, because a generic chat is what they already know how to use.

So retrieval was never the real constraint. A specialized platform needs explicit onboarding to its own tools, and the thing actually teaching people the platform was not documentation. It was live demos and office hours.

If I ran this again I would test whether users reach for platform support at all before specifying what it returns. The formatting work still holds. It just sits one layer below a positioning problem that no content pattern solves.