Skip to content
AI-SNS.IO — CHAT SUPPORTAI-SNS.IO — CHAT SUPPORT
Knowledge Base

Versioning Knowledge to Avoid Stale Answers

AI SNS CHAT EditorialPublished 8 min read

How to keep an AI support agent from quoting old prices, policies and hours: version your knowledge, test changes first, and roll back safely.

A customer asks your website chat about the return window, and the agent confidently quotes last year's policy. Nothing was "wrong" with the AI. It answered faithfully from a document nobody retired. This article shows how to treat support knowledge like a product release, with versions, owners, tests and a way back, so that stale answers become rare and easy to trace.

Why stale answers happen

An AI agent has no sense of time. If your knowledge base contains a 2023 shipping table and a 2025 shipping table, both are simply text it may retrieve. Staleness usually comes from a handful of everyday causes:

  • Old and new documents coexist. A new policy is uploaded, but the old PDF is never removed.
  • Copies drift apart. The same price appears in a help article, a spreadsheet and a Q&A entry, and only one gets updated.
  • Changes happen outside support. Marketing changes a promotion, or operations changes holiday hours, and nobody tells the person who maintains the agent.
  • Synced sources fall behind. A connected wiki is edited daily, but the agent's copy was read weeks ago.
  • Fixes are made live. Someone edits knowledge in a hurry, and the edit breaks another answer nobody rechecks.

The cost is not only a wrong answer. On WhatsApp or Messenger, customers treat a reply as a commitment. A quoted refund window or delivery date can become an argument your team has to settle by hand.

What "versioning" means for support knowledge

You do not need a software engineering department. Versioning here means three simple habits applied to every piece of knowledge:

  1. Every fact has one home. One document or entry is the authority for each topic.
  2. Every change is attributable and dated. You can tell who changed what and when.
  3. Every release can be undone. If a change makes answers worse, you can return to the last good state quickly.

A lightweight naming convention helps more than people expect. Put the topic, the scope and the effective date in the title of each source, for example "Returns policy - all regions - effective 1 March". When a new version arrives, the old one is removed or clearly marked as retired, not left beside it.

Keep one source of truth and retire the old one

Conflicting sources are the most common cause of stale answers, and the fix is organisational before it is technical.

Assign an owner per topic

Give each topic a named owner: pricing, shipping, returns, opening hours, product specifications, warranty. The owner is the person who hears first when something changes and who is accountable for the knowledge staying correct. A short shared list of topic, owner and source is enough.

Collapse duplicates

Search your sources for the facts that change most often, such as prices, fees, deadlines and phone numbers. Wherever a fact appears in several places, keep one authoritative copy and link or point everything else to it. Where that is not practical, list the duplicates beside the owner so they are updated together.

Retire, do not just add

Adding the new document is half the job. The release is not complete until the superseded version is deleted or removed from the agent's knowledge. If you worry about losing history, keep the old file in your own document system, outside the agent's sources.

A release routine: draft, test, publish, roll back

Treat a knowledge change the way a careful team treats a website change. The loop below works whether you change one answer or a hundred.

  1. 01Draft the change
  2. 02Test with real questions
  3. 03Publish to the live agent
  4. 04Watch and roll back if needed
A safe knowledge release loopIllustrative example

Draft away from the live agent

Prepare the new content first, ideally in the owner's own document or in a copy of the agent, rather than editing the live knowledge mid-conversation. A half-finished edit seen by real customers is a stale or contradictory answer waiting to happen.

Test with a fixed question set

Keep a list of ten to thirty questions that matter most: the ones customers ask daily, the ones tied to money or deadlines, and the ones that went wrong before. Ask them after every change and compare the answers with what you expect. Include some phrased the way customers actually type them, with typos and short fragments, since that is what arrives on chat.

Publish deliberately

Make the change at a quiet time, tell the team what changed, and note the date. If several topics change together, such as a pricing update, release them as one batch so the agent is not briefly half old and half new.

Keep a way back

Before a large change, make sure you can return to the previous state. A rollback you have never practised is a hope, not a plan. Decide beforehand who may roll back and what triggers it, for instance "two or more wrong answers on the test list".

Handle time-sensitive content differently

Not all knowledge ages at the same speed. Sorting your content by how quickly it expires tells you how often to review it.

Content typeTypical change speedSuggested review
Brand voice, greeting, toneRarelyTwice a year
Product descriptions, how-to guidesOccasionallyAt each product update
Return, shipping and warranty policyA few times a yearAt each policy change, plus a quarterly check
Prices, fees, plansOftenEvery price change, with a named owner
Promotions, holiday hours, campaignsShort-livedSet an end date when you publish

The last row deserves special care. Promotions are the classic source of stale answers, because the offer ends but the document does not. When you add a time-limited item, add the removal task to someone's calendar on the same day. If the end date is written in the content itself, the agent also has a chance of mentioning the deadline rather than promising an expired deal.

Detect stale answers after launch

Even with a good routine, some drift gets through. Build in signals that surface it early.

  • Read real conversations. Sample recent chats on each channel, focusing on topics you changed recently.
  • Use customer ratings. Thumbs-down ratings on answers about a topic you recently updated suggest the update did not land, or the old content is still there.
  • Watch repeated questions. A topic that suddenly attracts many questions or many handoffs to humans often means the answer changed in the real world and the agent has not caught up.
  • Listen to your agents. When a human takes over and corrects the bot, that correction is a free report of stale knowledge. Capture it.
  • Re-run the test list on a schedule. A monthly pass catches slow drift in synced sources before customers do.
Superseded document still loaded
5
Duplicate fact left unchanged
4
Expired promotion
3
Synced source not refreshed
2
Example: where one team found stale answers in a monthly review (hypothetical counts)Illustrative example

The point of a tally like this is not the numbers but the pattern. If most problems come from superseded documents, your retirement step needs work. If they come from synced sources, shorten the refresh routine.

Make changes traceable

When an answer is wrong, the first question is "where did that come from, and who changed it last?" Without a record, the investigation turns into guesswork and blame. With one, it takes a minute.

Good traceability means you can see who added each source, who edited it last, and when. It also means the people allowed to change knowledge are a deliberate, small group. Customers' questions are answered by whatever is loaded, so permission to edit knowledge is permission to change what your brand says.

Pair that record with a short change note, written in your own tracker or channel: what changed, why, and which test questions you ran. Six months later, that note is what lets you explain a policy answer to a colleague or a customer.

Doing this in AI SNS CHAT

AI SNS CHAT does not present a document-by-document diff view, so versioning is a discipline built from a few features that are there. Here is a practical set-up.

Organise sources under Data sources. In the sidebar, Data sources holds Files, Text snippets, Website, Q&A, Notion, Social, Tickets and Suggestions. Use the naming convention above, with topic, scope and effective date, so a reviewer can see at a glance what is current. Connected knowledge tools under Channels, such as Confluence, GitBook, Guru, Zendesk Guide, Google Drive and others, keep the authoritative copy in the system your team already edits. Notion is re-read on demand, so trigger a re-read after a significant edit. Wiki connectors have a per-sync limit of 1.5 million characters and 400 pages, and truncation is reported to you, so check that message rather than assuming everything was read.

Stage changes in a copy. Duplicate an agent to copy its persona, model and training data without its conversations. Load and test the new knowledge in the copy, then apply the same change to the live agent. Your plan's agent allowance applies: Free includes one agent, Hobby two, Standard three, and extra agents are available as an add-on, so check /pricing before relying on a permanent staging copy.

Test in Playground. Run your fixed question set in Playground before and after each change. Playground A/B model compare lets you see how two models answer the same question, which is useful when a change affects wording.

Keep a way back. Agents support restore points for rollback, up to 40 per bot, and deleted agents wait in Trash for 30 days. Nightly downloadable backups keep the last 7 snapshots. Before a large knowledge change, confirm a recent restore point exists and know who may use it.

Find stale answers from real chats. Open Activity, then Chat logs, to filter conversations, review thumbs up and down ratings, and flag or escalate problem chats. When a person takes over a conversation, you can edit the reply into the ideal answer to train the agent, which turns a live correction into lasting knowledge. Analytics, including Topics and Question history, show what customers keep asking. Advanced analytics need the Pro plan.

Know who changed what. Under Workspace settings, then Activity, owners and admins see every create, edit, delete or upload, with name, email, feature and time. Each data source also shows who added it and who last edited it. Roles such as Owner, Member, Support and Viewer let you keep editing rights with the topic owners only.

For the full list of capabilities, see /features.

Frequently asked questions

Does AI SNS CHAT keep a version history of each document?

It does not show per-document diffs. Version control comes from the practices in this article: dated source names, a staging copy made with Duplicate, restore points for rollback, nightly backups and the workspace activity log that records who changed what and when.

How often should I review the knowledge base?

Match the review to how fast content ages. Prices and promotions need a check at every change, policies need a quarterly pass, and tone or brand content can wait for a twice-yearly review. A monthly re-run of your test questions catches slow drift in synced sources.

A connected wiki is edited daily. How do I stop the agent falling behind?

Treat the sync as a release step. After significant edits, refresh the source, check whether any truncation was reported, and re-run your test questions. Notion in particular is re-read on demand, so make that refresh part of whoever publishes the change.

Can I let the agent search the web so it never goes stale?

Only partly. Answering from the web when the knowledge base has no answer is available on LINE WORKS, defaults to Microsoft and Apple support sites and cites the page it used. For other channels, keep your own knowledge accurate, or use the Tavily action if you need real-time web search for an agent.

What should I do the moment I discover a wrong answer in production?

Correct the source first so the error stops repeating, then retest the related questions in Playground. If a recent change caused it and the damage is wider than one topic, use a restore point to return to the last good state, and record the cause in your change note.

#knowledge base#version control#accuracy#AI support

Put this into practice with AI SNS CHAT

Build an AI agent trained on your own content and connect it to your website, LINE, WhatsApp and more. Free to start, no credit card required.