A practical routine for keeping chatbot answers accurate: audit sources, assign owners, fix gaps from real chats, and re-sync content as it changes.
Most support bots do not fail on launch day. They fail three months later, when the return policy has changed, a product has been renamed, and the knowledge base still says the old thing with total confidence. This article gives you a lightweight maintenance routine: what to review, who owns it, how to turn real conversations into fixes, and how to keep changes safe and reversible.
Why knowledge bases go stale
A knowledge base is a snapshot of what your business believed on the day someone uploaded it. Your business keeps moving, and the snapshot does not.
Staleness usually comes from a handful of predictable sources:
- Policy changes. Shipping times, refund windows, opening hours, pricing and warranty terms.
- Product changes. New plans, retired items, renamed features, updated screenshots and steps.
- Duplicated content. The same answer lives in a PDF, a website page and a pasted snippet, and only one copy gets updated.
- Seasonal information. Holiday hours, campaign terms and temporary delivery delays that nobody removes afterwards.
- Unowned sources. A file was uploaded by someone who has since changed roles, and nobody feels responsible for it.
The danger is not that the bot says "I don't know". It is that it answers fluently from outdated material, so the customer trusts the wrong answer on website chat, WhatsApp or Messenger.
Start with an inventory, not a rewrite
Before you fix anything, list what the agent actually learns from. Many teams are surprised by the result.
For each source, record four things:
- What it is: file, text snippet, website crawl, Q&A pair, or a connected document tool.
- What it covers: returns, pricing, onboarding, troubleshooting and so on.
- Who owns it: a named person or team, not "support".
- How it changes: rarely, monthly, or whenever the business changes something.
Then group the sources by how fast they go out of date. A company history PDF can be reviewed once a year. Delivery times and promotions may need a check every week.
Assign owners and a review rhythm
A maintenance routine survives only if it is small enough to actually happen. Aim for a short, regular cadence rather than a large annual project.
A simple rhythm
| Cadence | What to do | Who |
|---|---|---|
| Weekly (15 minutes) | Read recent low-rated or escalated chats and fix the cause | Support lead |
| Monthly | Review the most-asked topics and confirm the answers are still right | Topic owners |
| On change | Update the source the same day a policy, price or product changes | Whoever makes the change |
| Quarterly | Remove duplicates, expired campaigns and unused sources | Knowledge base owner |
The "on change" row matters most. Add one line to your launch or policy-change checklist: "Update the AI agent's knowledge." That single habit prevents more wrong answers than any audit.
Let real conversations tell you what to fix
Guessing which content is weak is slow. Your chat history already shows it.
Look for these signals:
- Thumbs-down ratings on specific answers.
- Escalations where the bot handed a customer to a person because it could not answer.
- Repeated questions on the same topic, which suggest the answer is missing, buried or unclear.
- Corrections by your team. When a human has to rewrite the bot's answer, that is a free lesson.
Each signal leads to one of three fixes:
- Gap: the answer does not exist anywhere. Add it.
- Conflict: two sources disagree. Pick one and remove the other.
- Clarity: the answer exists but is vague or hidden in a long document. Add a short, direct Q&A entry.
- 01Review flagged chats
- 02Find the cause: gap, conflict or clarity
- 03Fix the source
- 04Test with the same question
Close the loop by re-asking the original question after the fix. If the answer is now correct, you are done. If not, the fix did not land where the agent looks.
Keep sources tidy and within limits
Bigger is not better. A knowledge base stuffed with every document you own gives the agent more chances to retrieve the wrong passage.
Practical hygiene rules:
- Prefer short, single-topic sources. One page per policy is easier to update than one 80-page handbook.
- Date your content. A line such as "Valid from 1 March" in the text helps your team spot expired material.
- Remove temporary content on a schedule. When you add a campaign, write its end date in your review calendar.
- Mind storage and size limits. Each plan has a storage allowance, so pruning old files is not just tidiness, it frees room for current material.
- Watch large document syncs. When you pull in a big wiki, a single sync can be truncated if it is very large, so check that the pages you care about actually arrived.
Here is a hypothetical example of how a quarterly clean-up might look. These numbers are illustrative only.
Fewer, better sources are easier to own, easier to review and less likely to contradict each other.
Change safely: test, track and roll back
Every edit to a knowledge base is a small change to how your bot behaves. Treat it that way.
Test before customers see it. Ask the agent the questions your edit should affect, plus two or three neighbouring questions to check you did not break something nearby. Use the exact wording customers use, including typos and short messages typical of WhatsApp or Messenger.
Know who changed what. When an answer suddenly goes wrong, the first question is "what changed and who changed it?" Keeping a record of additions and edits turns a mystery into a two-minute check.
Keep a way back. Before a large change, such as replacing a main document or rewriting a persona's instructions, save a restore point or a backup so you can undo it quickly if answers get worse.
Handle personal data carefully. Do not paste customer names, order details or private records into shared knowledge. Knowledge should describe your policies and products, not individual people.
Doing this in AI SNS CHAT
AI SNS CHAT gives you the pieces for this routine in a few places in the dashboard.
Data sources. Open Data sources in the sidebar to see what each agent learns from: Files, Text snippets, Website, Q&A, Notion, Social, Tickets and Suggestions. Each source shows who added it and who last edited it, which makes ownership visible. For documentation tools, connectors under Channels cover Confluence, GitBook, ReadMe, Guru, esa.io, DocBase, Qiita Team, Zendesk Guide, Google Drive, OneDrive, Dropbox, Box, an Obsidian vault via Git, SharePoint, and Notion, which can be re-read on demand. A wiki connector syncs up to 1.5M characters and 400 pages at a time, and if a sync is truncated, you are told. See /connections for the full list.
Finding what to fix. Conversations from every channel except email appear under Activity → Chat logs, where you can filter, flag, escalate, resolve or export chats and review thumbs up and down ratings. Email conversations are in Inbox. Under Analytics, the Topics and Question history views show what customers ask most, which tells you where accuracy matters most. Advanced analytics need the Pro plan.
Fixing from a real chat. In both Inbox and Chat logs you can take over a conversation, which pauses the bot, and edit a reply into the ideal answer to train the agent. That turns a human correction directly into improved knowledge. Add a Q&A entry for a question the documents answer poorly.
Testing. Use Playground to ask the agent your test questions before the change reaches customers. Playground A/B model compare lets you check how different models handle the same question.
Tracking and undoing. Workspace settings → Activity, available to owners and admins, records every create, edit, delete or upload with name, email, feature and time. Agents support restore points (up to 40 per bot) for rollback, Duplicate for safely experimenting on a copy, and a 30-day Trash for deleted agents. Nightly downloadable backups keep the last 7 snapshots.
Sharing the work. Roles (Owner, Co-owner, Member, Support, Viewer) with per-member permission overrides let you give knowledge editing to the right people and read-only access to others. If you use Co-Agent memory notes, remember they are off by default and notes wait for approval before they are used.
Plan limits apply to storage (from 30 MB on Free up to 3 GB on Corporate, with storage add-ons available). Check /pricing and /features for current details, or start with /signup.
A one-page checklist
Copy this into your team's notes:
- Every source has a named owner.
- Policy and product changes trigger a same-day knowledge update.
- Low-rated and escalated chats are reviewed weekly.
- Duplicates and expired content are removed quarterly.
- Large changes are tested first and have a way back.
- Personal data stays out of shared knowledge.
Conclusion
Keeping a knowledge base current is less about effort and more about habit. Name owners, review a few real conversations each week, update at the moment things change, and keep every edit testable and reversible. Do that, and your agent stays as accurate in month twelve as it was on launch day.
Frequently asked questions
How often should I review my AI agent's knowledge base?
Do a short weekly check of low-rated and escalated chats, a monthly review of your most-asked topics, and a quarterly clean-up of duplicates and expired content. Also update the relevant source on the same day any policy, price or product changes.
What is the fastest way to fix a wrong answer?
Find the chat in Chat logs (or Inbox for email), take over the conversation, and edit the reply into the ideal answer so the agent learns from it. Then ask the same question in Playground to confirm the fix worked.
Can I see who changed a knowledge source?
Yes. Each data source shows who added it and who last edited it, and the workspace Activity log under Workspace settings records every create, edit, delete or upload with name, email, feature and time. Owners and admins can open it.
What if an update makes answers worse?
Agents support restore points, up to 40 per bot, so you can roll back. You can also test changes on a duplicated agent first, and nightly backups keep the last 7 snapshots for download.
Does the agent automatically pick up changes in my documentation tool?
Connected documentation sources are synced from the connector, and Notion can be re-read on demand. After a large documentation change, re-sync the source and check that the pages you care about arrived, since very large wiki syncs can be truncated.


