Learn how to sample chats, score them with a simple rubric, and turn findings into fixes for your website chat, WhatsApp and Messenger support.
Most teams only look at their chats when a customer complains. By then the same weak answer may have gone out dozens of times. A regular quality review replaces that firefighting with a short, repeatable routine: sample, score, fix, and check again. This article shows you how to set one up for website chat, WhatsApp and Messenger, whether replies come from an AI agent, a human, or both.
Why a review routine beats ad hoc checking
Spot checks feel productive, but they are biased. You tend to open the chats that someone flagged, or the ones that happen to be at the top of the list. A routine gives you three things that spot checks cannot:
- A fair picture. Random sampling shows what a typical customer experiences, not just the dramatic cases.
- A shared standard. When everyone scores against the same rubric, "good" stops being a matter of opinion.
- A trend. One review tells you little. Four reviews in a row tell you whether your changes are working.
Quality review is also the main way an AI agent improves. An agent rarely fails everywhere at once. It fails on specific topics where the knowledge is missing, outdated or contradictory, and only a review will find them.
Decide what "good" means before you read a single chat
Reviewers who start reading without criteria drift toward personal taste. Write down four or five dimensions first. A workable starting set:
| Dimension | The question to ask |
|---|---|
| Accuracy | Was every factual claim correct for your business today? |
| Resolution | Did the customer get what they came for, or a clear next step? |
| Tone | Did it sound like your brand, in the customer's language? |
| Handoff | When a human was needed, was the transition timely and clear? |
| Safety | Did it avoid promises, guesses and private data it should not share? |
Score each dimension on a small scale, such as 0 (fail), 1 (acceptable), 2 (good). Keep the scale coarse. Finer scales create arguments without creating insight.
Build a sample that represents your real traffic
You do not need to read everything. You need to read the right mix. Combine two kinds of samples each week.
Random sample
Pick a fixed number of conversations at random across channels. For example, if a shop receives 300 chats a month, reading 20 to 30 of them is a reasonable weekly habit. The number is illustrative; choose one your team can sustain without skipping weeks.
Targeted sample
Add conversations that are statistically more likely to hide problems:
- Chats the customer rated negatively
- Chats that ended in a handoff to a human
- Chats flagged or escalated by your team
- Long conversations where the customer repeated themselves
- Chats in languages or channels you rarely look at, such as WhatsApp at weekends
Keep the two samples separate in your notes. The random sample estimates your overall quality; the targeted sample finds the fixable defects.
- 01Sample chats
- 02Score against rubric
- 03Fix the cause
- 04Re-check next week
Score fast, then diagnose slowly
Run the scoring pass at speed. Spend a couple of minutes per chat, mark each dimension, and add a one-line note only when something is wrong. The goal is to produce a list of failures, not to rewrite every answer.
Then switch to diagnosis. For each failure, ask where the root cause sits, because the fix depends on it:
- Missing knowledge. The answer was not in the sources, so the agent guessed or deflected. Add the content.
- Wrong or stale knowledge. The agent quoted an old policy. Update or remove the source.
- Conflicting knowledge. Two documents disagree. Keep one.
- Persona or guardrail gap. The facts were right but the tone or boundaries were off. Adjust the instructions.
- Process gap. The agent was right to hand off, but nobody picked it up quickly. Fix the rota or alerting.
Most teams find that a small number of root causes explain most failures. Fixing one stale document can clear a whole cluster.
The numbers above are a hypothetical illustration of how a tally might look, not a benchmark.
Turn findings into changes, and keep a log
A review that produces only observations changes nothing. End every session with a short action list, each item having an owner and a date:
- Add or correct a knowledge source
- Rewrite an instruction in the agent's persona
- Add a saved reply for humans to use on a recurring question
- Adjust shift coverage where handoffs sat waiting
Record what you changed and why. Next week, check the same topic in your sample. If the failure disappears, you have evidence the fix worked. If it persists, your diagnosis was wrong, and you can try the next likely cause.
A few habits keep the routine alive:
- Rotate reviewers. Fresh eyes catch what the usual reader has stopped noticing.
- Calibrate occasionally. Have two people score the same five chats and compare. Large gaps mean the rubric needs clearer wording.
- Share wins. Tell the team when a fix measurably improved answers. It keeps the review from feeling like an audit.
- Protect privacy. Limit who can read full conversations, and avoid copying customer details into review notes.
Handling different channels fairly
Website chat, WhatsApp and Messenger set different expectations. Website visitors often want a quick answer while browsing; WhatsApp and Messenger conversations tend to pause and resume over hours or days. When you score, judge each chat against the norms of its channel. A reply gap that is fine on WhatsApp may be a failure on a live website widget. Tag each reviewed chat by channel so you can see whether problems cluster in one place.
Doing this in AI SNS CHAT
AI SNS CHAT gives you the raw material and the levers for each step of the loop.
Find and sample chats. Open Activity → Chat logs. Conversations from every channel except email appear here, and you can filter them, export them, and select several at once. Customers can leave thumbs up or down ratings, so you can filter toward the chats they rated poorly for your targeted sample. Email conversations live separately in the Inbox.
Mark and track. In Chat logs you can flag, escalate, resolve or delete conversations in bulk. Flagging the failures from your review gives the team a shared list to work through.
Fix answers directly. When you open a conversation in Chat logs or the Inbox, you can edit a reply into the ideal answer to train the agent. That turns a review finding into an immediate correction rather than a note in a spreadsheet.
Fix knowledge. Under Data sources you can add Files, Text snippets, Website pages, Q&A pairs, Notion content and more, and see which sources feed an agent. Many documentation tools can also be connected under Channels, including Confluence, GitBook, Zendesk Guide, Google Drive and SharePoint. The workspace Activity log (Workspace settings → Activity, for owners and admins) shows who added or last edited each source, which helps when you need to trace a stale answer to its origin.
Adjust persona and test safely. Persona and guardrails are edited per agent. Before you publish a change, try it in Playground, and use the Playground A/B model compare if you want to see how two models answer the same question. Restore points let you roll an agent back if a change makes things worse.
Fix the handoff. Helpdesk → Scheduling sets shifts per person and timezone so handoffs go to whoever is on shift, and handoff alerts can reach Slack, Teams, LINE WORKS, WhatsApp, email or a webhook. Helpdesk needs the Standard plan or higher.
Look at the trend. Analytics includes Chats, Topics, Question history and Sentiment, which help you choose where to look next. Advanced analytics need the Pro plan. You can also ask Synthia, the built-in assistant, to walk you to the right screen. See /features for the full list and /pricing for plan details.
Conclusion
A good quality review is small, regular and honest. Sample randomly, add the risky chats, score against a simple rubric, find the root cause, make one clear change per problem, and check it next week. Do that for a month and your answers become more consistent, your team agrees on what good looks like, and complaints stop being your only source of feedback.
Frequently asked questions
How many chats should we review each week?
Choose a number your team can keep up every week. For example, 20 to 30 chats from a mix of random and targeted samples is a reasonable starting point for a small team. Consistency matters more than volume, so reduce the number rather than skip weeks.
Should humans or the AI agent be reviewed more strictly?
Use the same rubric for both. The customer experiences the whole conversation, so accuracy, tone and resolution matter regardless of who replied. The difference is in the fix: for the agent you change knowledge or instructions, for people you change training or saved replies.
Who should do the reviews?
Ideally someone who knows the product and the customers well, with the role rotated among two or three people. Occasionally have two reviewers score the same chats to check that your rubric is understood the same way.
Can I use customer ratings instead of reviewing chats myself?
Ratings are a useful filter for finding problem chats, but most customers never leave one. Use them to feed your targeted sample, and keep a random sample to see the quality of conversations nobody rated.


