Skip to content
AI-SNS.IO — CHAT SUPPORTAI-SNS.IO — CHAT SUPPORT
KPIs

Measuring AI Resolution Rate Correctly

AI SNS CHAT EditorialPublished 7 min read

Most AI resolution rates are inflated or misleading. Learn how to define, sample and report a resolution rate you can trust, using real chat log data.

Almost every team that deploys an AI support agent ends up quoting a "resolution rate", and almost every team calculates it differently. One person counts chats that never reached a human; another counts thumbs-up ratings; a third counts anything that ended without a complaint. The result is a number that looks good in a slide and tells you very little. This article shows you how to define resolution precisely, measure it with a method you can defend, and report it without fooling yourself.

Why the "obvious" resolution rate misleads

The simplest formula is: chats the AI handled without a human, divided by all chats. It is easy to compute, and it is wrong in at least four ways.

  • Silence is not success. A customer who gives up after an unhelpful answer looks identical to one who got what they needed. Both leave without escalating.
  • Deflection is not resolution. If the AI politely refuses, or points to a help page the customer never opens, the chat was deflected. The problem still exists.
  • The denominator is soft. Greetings, test messages, spam and accidental opens inflate the total, which pushes the rate down or up depending on how you filter them.
  • Channels behave differently. A website chat visitor who closes the tab is normal behaviour. A WhatsApp customer who stops replying may simply be waiting until tomorrow.

None of this means the simple number is useless. It is a good containment rate. It just should not be called resolution.

Define the terms before you count anything

Agree on definitions in writing first. A workable set looks like this.

Conversation (the denominator)

A conversation is a customer-initiated exchange with a real support need. Decide, up front, what you exclude: obvious tests from your own staff, empty sessions where the visitor never typed, and spam. Keep the exclusion rules stable from month to month, otherwise your trend line moves for reasons unrelated to quality.

Contained

The AI replied and no human took over or was alerted. This is what your platform can count automatically.

Resolved

The customer's question was answered correctly and completely, or the requested action was done. This needs a judgement, either from the customer or from a reviewer.

Escalated

The chat was passed to a human, either because the customer asked or because the AI could not continue. Escalation is not a failure by itself. A clean handoff on a billing dispute is a good outcome.

Unresolved or abandoned

The AI replied, nobody took over, and the evidence says the need was not met. This is the bucket the simple formula hides.

  1. 01All chats
  2. 02Remove tests and spam
  3. 03Split: contained vs escalated
  4. 04Review a sample of contained chats
From raw chats to a trustworthy resolution rateIllustrative example

Build the metric in layers

Instead of one number, report a small stack. Each layer answers a different question.

LayerQuestion it answersHow you get it
Containment rateHow often did the AI handle a chat alone?Count chats with no human takeover or escalation
Escalation rateHow often did people need a human?Count escalated or taken-over chats
Customer-rated satisfactionDid customers who bothered to rate feel helped?Thumbs up/down ratings
Verified resolution rateOf contained chats, how many were actually solved?Manual review of a random sample

The last row is the one that matters. It is a contained rate corrected by what a reviewer actually finds.

The formula

Verified resolution rate = containment rate × share of sampled contained chats judged "resolved".

Here is a purely hypothetical illustration. Suppose a shop receives 300 real chats in a month. The AI handles 210 of them without a human, so containment is 70%. You randomly review 30 of those 210 and judge 24 as properly resolved, which is 80%. Verified resolution is then 70% × 80% = 56% of all chats. The headline "70%" would have overstated reality by 14 points. These numbers are invented for the example; your own will differ.

Containment (hypothetical)
70
Verified resolution (hypothetical)
56
Escalated to a human (hypothetical)
30
Example: how a 70% containment rate becomes 56% verified resolutionIllustrative example

Sample properly, not conveniently

The review step is where most teams cut corners, so be deliberate.

  1. Pick randomly. Do not review only the chats you remember or the ones that were flagged. Random selection is what makes the sample representative.
  2. Review enough to be useful. A tiny sample gives a noisy number. If you can only review a handful of chats, treat the result as a rough signal, not a precise rate, and say so in your report.
  3. Stratify when volume allows. Sample separately by channel (website chat, WhatsApp, Messenger) and by topic. A good average can hide a channel that is failing.
  4. Use a simple rubric. For each chat, record: was the answer correct, was it complete, did the customer need anything further, and would a competent human have said the same thing.
  5. Have two people review a subset. If two reviewers disagree often, your rubric is too vague.

Use customer signals, but know their bias

Ratings and follow-up behaviour are valuable evidence, with caveats.

  • Thumbs up/down is cheap and immediate, but only a minority of customers use it, and unhappy or very happy people are over-represented. Use it as a trend and a source of chats to inspect, not as the resolution rate itself.
  • Repeat contact is a strong negative signal. If the same person returns within a day or two about the same issue, the first chat probably did not resolve it. Contacts make this easy to spot if you identify customers on WhatsApp or by email.
  • Sentiment shifts during a conversation. A chat that starts neutral and ends negative deserves review even if it was contained.
  • Time of last message. On WhatsApp and Messenger, a customer may reply hours later. Allow a reasonable waiting window before you classify a chat as abandoned.

Handle escalations fairly

An AI that escalates the right chats is doing its job. Do not punish a healthy handoff by lumping it with failures.

Split escalations into two groups:

  • Appropriate: refunds above a threshold, legal or safety questions, account security, anything your policy says needs a person.
  • Avoidable: the answer existed in your documentation but the AI did not find it, or the AI misunderstood a simple request.

Your goal is to shrink the avoidable group, not to drive escalation to zero. Chasing zero encourages the AI to bluff, which damages the verified resolution rate you are trying to improve.

Report it so people trust it

A resolution rate is only useful if the people reading it believe it. A few habits help.

  • State the definition on the dashboard or report itself, in one line.
  • Show containment and verified resolution side by side.
  • Report the sample size and the period.
  • Keep a short list of the top reasons for unresolved chats, with an example of each.
  • Track the number monthly, but avoid reacting to small movements. With modest volume, a few points of change can be noise.

The most valuable output is not the percentage. It is the list of knowledge gaps it points to: missing articles, outdated policies, ambiguous instructions.

Doing this in AI SNS CHAT

AI SNS CHAT gives you the raw material for each layer, though the verified resolution step remains a review process you run yourself. The platform does not produce a single "resolution rate" figure for you.

Find and review conversations. Every channel except email lives in Activity → Chat logs, covering your website chat widget, WhatsApp, Messenger and the rest. Email conversations are in the Inbox. In Chat logs you can filter, bulk select, flag, escalate, resolve or delete conversations, and export them. Export a month, remove tests, and draw your random sample from the result. Use flags to mark the chats you reviewed and found wrong.

Count escalations and takeovers. When a person takes over a chat, the bot pauses, so takeovers are visible in the logs. Helpdesk → Scheduling lets you set shifts per person and timezone, and handoffs go to whoever is on shift, with alerts to Slack, Teams, LINE WORKS, WhatsApp, email or a webhook. Helpdesk needs the Standard plan or higher. The customer-facing handoff message is written by the server from your rota, and it never invents a wait time.

Collect customer signals. Thumbs up/down ratings appear in Chat logs. Under Analytics you will find Chats, Topics, Question history and Sentiment; advanced analytics need the Pro plan. Topics and Question history are useful for spotting which subjects produce unresolved chats.

Close the gaps you find. When a reviewed answer is wrong, you can edit the reply into the ideal answer in Chat logs or the Inbox to train the agent. For missing information, add a Q&A entry, a text snippet, a file or a website in Data sources, or connect a documentation tool under Channels. Use Playground to test the corrected behaviour before you rely on it, and Playground A/B model compare if you want to see whether a different model answers a problem topic better (advanced models need Hobby or higher).

Identify repeat contacts. The Collect leads action and custom forms create contacts, which you can view and export as CSV. Having a name or email attached to a conversation makes repeat-contact checks possible.

You can also ask Synthia, the built-in fleet manager, to guide you to these screens. See /features for the full list and /pricing for plan details.

Conclusion

A trustworthy resolution rate is built, not read off a dashboard. Separate containment from resolution, exclude noise consistently, sample randomly, and correct the headline number with what reviewers actually find. Then use the unresolved chats as a to-do list for your knowledge base.

Frequently asked questions

Is containment rate the same as resolution rate?

No. Containment counts chats where no human was needed, while resolution asks whether the customer's problem was actually solved. Containment is an upper bound on resolution, and the gap between the two is what a sampled review reveals.

How many chats should I review each month?

As many as you can review carefully and randomly. A small sample gives a rough indication, a larger one gives a steadier number. Whatever you choose, report the sample size alongside the rate so readers can judge its reliability.

Should I count customer thumbs-up ratings as resolved?

Treat them as supporting evidence, not as the resolution rate. Only some customers rate, and the ones who do are not a random group. Use ratings to find chats worth reviewing and to watch trends.

Does AI SNS CHAT calculate the resolution rate automatically?

It provides chat logs, ratings, filters, export, escalation handling and analytics such as Chats, Topics and Sentiment. It does not output a single verified resolution figure, so you combine those inputs with a sampled review as described above.

#KPIs#resolution rate#AI support#chat logs#analytics

Put this into practice with AI SNS CHAT

Build an AI agent trained on your own content and connect it to your website, LINE, WhatsApp and more. Free to start, no credit card required.