How to train an AI chatbot on your help center (and why the articles matter more than the bot)
By The Relay team · Published · 4 min read
The short answer
Most support bots aren't trained on your help center in the machine-learning sense. They look up your articles when a question arrives and answer from them. So the quality of the bot is the quality of the articles: one question per article, the answer in the first paragraph, and the words customers actually use in the title. Test it by asking the bot your last 20 real tickets.
What "training" actually means
When a vendor says a bot is trained on your help center, it usually means the bot reads your articles at the moment someone asks a question. Nothing is learned permanently, and nothing is memorised. That is good news: fixing a wrong answer means fixing an article, and the fix applies to the next question.
How a grounded answer is made
Relay's bot follows four steps, which are typical of this kind of tool.
- Retrieve. It matches the customer's question against your published articles. Matches in an article's title count for more than matches in its excerpt, and those count for more than matches in the body.
- Answer. With AI on, it sends the question, the last few turns and the matched articles to Claude, with instructions to answer only from those articles, keep it brief, and say it can't help when they don't cover the question. If keyword matching finds nothing, Claude is given up to 20 of your published articles to read, so a paraphrased question still has a chance.
- Cite. Articles that matched are attached as cards the customer can open. When the answer came from the wider set instead of a keyword match, there is nothing to cite.
- Offer the next step. The latest answer has "That helped" and "Talk to a human" buttons.
Only published articles are used. Drafts, internal notes and other customers' data are never sent to the model.
Five rules for articles a bot can use
1. One question per article
An article called "Billing and account settings" that covers eight topics gives the bot eight things to confuse. Split it: "How do I update my card?", "How do I download an invoice?", "How do I change my plan?".
2. Put the customer's words in the title
Titles carry the most weight in matching. Use the phrase a customer would type ("reset my password"), not the name of your feature ("Credential recovery"). Your last month of tickets is the best source of those phrases.
3. Answer in the first paragraph
The excerpt is weighted too, and a short answer first helps both the bot and a person skimming. Put steps and detail below it.
4. Be exact about names, plans and limits
Say "on the Team plan" or "in Settings → Billing", not "in your account". Avoid pronouns that need the previous paragraph to make sense, because the bot may see one article without the rest.
5. Say what you don't do
If you don't offer phone support or refunds after 30 days, write an article that says so. Otherwise the bot will reach for the nearest article, and a customer will hear something that isn't true. A clear "we don't" is better than silence.
Test it: the 20-ticket coverage check
- Pull your last 20 support requests.
- Ask the bot each one in the customer's own words, in the demo workspace or your own widget.
- Mark each answer correct, partly right, wrong, or handed off.
- For every wrong or handed-off answer, find out whether the article is missing, buried or badly titled, then fix that article.
- Run the 20 again. Repeat each month with a fresh 20.
Handoffs aren't failures. A bot that hands off a question you haven't written an article for is doing its job; the article is the gap. See how a bot should hand off and how to stop it making things up.
Keeping articles current
- Review the articles behind your most common tags every quarter.
- When the product changes, change the article the same day. A stale article gives wrong answers with a citation attached.
- Unpublish anything you can't keep accurate. An unpublished article can't be used by the bot.
Questions
Common questions
- Do I need to train the AI on my help center?
- Not in the technical sense. With Relay, you publish articles and the bot reads them when a question comes in. There is no separate training step; improving the answers means improving the articles.
- How many help center articles do I need?
- Enough to cover your most common questions. Start with the 20 questions you answer most, one article each, then add more as new questions repeat. Quality of titles and first paragraphs matters more than count.
- Can the bot answer questions the help center doesn't cover?
- It is told to answer only from your articles and to hand off when they don't cover the question, so a missing article results in a handoff to a person rather than a guess.
- Does Relay's bot show which articles it used?
- Yes, when the answer came from articles matched by keyword search, they appear as cards under the answer. When the answer came from the wider set of articles Claude was given, there is nothing to cite.
Sources and how we wrote this
Relay is made by the team that wrote this post. Figures for other tools come from their own public pages, listed under Sources, and are list prices in US dollars that can change. If something here is out of date, tell us on the feedback board and we'll correct it.