How to A/B test chat flows on website chat, WhatsApp and Messenger: pick one metric, size the test, avoid bias, and read results honestly.
Most teams change their chat flow on a hunch: a friendlier greeting, a shorter lead form, a faster handoff. Then they look at the numbers a month later and cannot tell whether the change helped or the month was simply different. This guide shows how to run a fair A/B test on a chat flow, whether it lives on your website, WhatsApp or Messenger, and how to read the result without fooling yourself.
What counts as a "chat flow" worth testing
A chat flow is everything a customer experiences between the first message and the end of the conversation. In practice, the parts you can change fall into a few groups:
- The opening: the welcome message, suggested questions, or the first question the agent asks.
- The persona and tone: formal or casual, long answers or short ones, how often the agent asks a clarifying question.
- The knowledge behind it: which sources the agent answers from, and how well they are written.
- Data capture: when and how the agent asks for an email address, phone number or order number.
- The handoff: when the agent offers a human, and what it tells the customer will happen next.
- The model: which AI model generates the replies.
Each of these can be tested. The trick is to test one at a time.
Choose one primary metric before you change anything
The most common mistake is running a test, then picking the metric that looks best afterwards. Decide the metric first, write it down, and treat everything else as a secondary check.
| If your flow aims to... | Primary metric | Watch out for |
|---|---|---|
| Answer questions without a human | Share of chats resolved without handoff | Customers giving up rather than being helped |
| Collect leads | Share of chats that leave contact details | Lower-quality leads from a shorter form |
| Improve satisfaction | Thumbs-up share on rated replies | Few customers rate at all, so the sample is small |
| Hand off cleanly | Share of handoffs where the customer got a human reply | Longer chats before handoff |
A good primary metric is a simple ratio you can compute from your chat logs: chats where X happened, divided by all chats in that group.
Change one thing at a time
If variant B has a new greeting, a shorter persona prompt and a different model, and it performs better, you have learned almost nothing. You do not know which change mattered.
Write the hypothesis in one sentence before you start:
- Change: "The welcome message offers three suggested questions instead of an open prompt."
- Expected effect: "More visitors send a first message."
- Metric: "Share of widget openers who send at least one message."
- Guardrail: "Thumbs-down share does not rise."
If you cannot fit the test into that format, it is too big. Split it.
Split your audience fairly
A fair test means both variants see similar customers at the same time. How you split depends on the channel.
Website chat
The cleanest split is by page or by visitor group. For example, variant A runs on one set of product pages and variant B on a comparable set. Avoid comparing a pricing page with a help page, because intent is completely different.
WhatsApp and Messenger
Chat apps usually attach one agent to one number or page, so you cannot easily show two versions to the same inbound audience. Two practical options:
- Alternate by time block: run A for a week, then B for a week. Choose weeks that are alike, and avoid sales events or holidays.
- Split by entry point: if you have two ads, QR codes or links that open the same channel, point each to a different variant. This works only if both entry points attract similar people.
What to avoid
- Letting staff choose which customers get which variant.
- Comparing this week with last week without checking for seasonal effects.
- Changing anything else (pricing, campaigns, opening hours) during the test.
- 01Write hypothesis and metric
- 02Build variant B from a copy of A
- 03Run both for a fixed period
- 04Compare, then keep or roll back
How long to run it, and how many chats you need
Small differences need many conversations to show up. If the numbers are too small, the result is noise.
Here is a hypothetical example. Suppose a shop receives 300 chats a month, and you split them evenly, so each variant gets 150. If variant A resolves 90 chats without handoff and variant B resolves 96, the gap is six chats. A different week with a few unusual customers could produce the same gap in the other direction. You would not act on that.
Some practical rules:
- Fix the end date in advance. Stopping the moment one variant looks ahead is a reliable way to find false winners.
- Run at least one full weekly cycle. Customers behave differently on Monday morning and Saturday night.
- Prefer big changes on low-volume channels. If you only get a few dozen chats a week, test bold changes (a completely different opening) rather than small wording tweaks.
- Extend rather than conclude. If the result is close, run longer or accept that the change makes no meaningful difference. "No difference" is a useful answer, because you can then pick the cheaper or simpler option.
In this made-up example, a six-chat gap on 150 chats each is within normal week-to-week variation. The honest conclusion is "not proven", not "B wins".
Read the results without fooling yourself
Before declaring a winner, check these:
- Same period, same mix. Did both variants see similar topics? Compare the topic breakdown, not just the totals.
- Read real conversations. Open 15 to 20 chats from each variant. Numbers tell you what changed; transcripts tell you why.
- Check the guardrail. Did handoffs, thumbs-down ratings or negative sentiment move the wrong way?
- Look at the segments. A variant might help first-time visitors and hurt returning customers. If you have enough chats, split by topic or language.
- Count cost. Longer, more thorough answers may use the same number of replies, but more clarifying questions mean more replies per conversation.
Common pitfalls
- Testing during a campaign. A promotion changes who shows up. Wait until traffic is normal.
- Novelty effects. A new greeting may get attention simply because it is new. Run long enough for that to fade.
- Stale knowledge. If you update the knowledge base in the middle of the test, the variants are no longer comparable.
- Ignoring language. A flow tested in English may behave differently in Japanese or Spanish. Test per language when volume allows.
- Over-testing. Not every sentence needs a test. Reserve A/B tests for changes with a real chance of moving a business metric.
Doing this in AI SNS CHAT
AI SNS CHAT does not currently document an automatic traffic splitter that serves two flows to random visitors. You run the split yourself, using tools that are there for exactly this kind of controlled change. Here is a workable routine.
1. Build variant B as a copy. Use Duplicate on your agent. It copies the persona, model and training data (but not conversations), so B starts identical to A. Change only the one thing you are testing: the persona text, the model under Agent → Settings → AI, or a data source. Advanced models need Hobby or higher, and each plan limits how many agents you can have, so check your plan on /pricing.
2. Compare models before you go live. If your test is about the model, use the Playground and its A/B model compare page to look at two models side by side on the same questions. This is a cheap way to rule out weak candidates before real customers see them.
3. Split the audience. Deploy A and B on different pages, links or time blocks through Sidebar → Channels (chat widget, WhatsApp, Messenger and the others on /connections). A shareable link for each variant is useful for entry-point splits.
4. Measure. Conversations from non-email channels are in Activity → Chat logs, where you can filter, flag, export and review thumbs up/down ratings. Analytics shows Chats, Topics, Question history and Sentiment; advanced analytics need Pro. If your flow collects leads, Activity → Leads shows what came in, and each lead also creates a contact. The Usage page shows credits per agent, which helps you compare cost, since 1 reply equals 1 credit.
5. Keep a record. Workspace settings → Activity logs every create and edit with name, email, feature and time, so you can show exactly when each variant changed. Before editing, you can rely on restore points (up to 40 per agent) to roll back if B underperforms.
6. Learn from the transcripts. In chat logs you can edit a reply into the ideal answer to train the agent, which turns what you find in a test into a lasting improvement.
For a tour of where these screens live, ask Synthia, the in-dashboard assistant, and she will navigate to the screen and highlight the control. See /features for the full list.
Frequently asked questions
Can AI SNS CHAT automatically split visitors between two flows?
The product facts do not describe an automatic random splitter. You create variant B with Duplicate and split by page, link, channel entry point or time block, then compare results in chat logs and analytics.
How many chats do I need for a reliable result?
It depends on how big the effect is. Large changes show up in a few hundred chats per variant, while small tweaks may need far more. If the gap is only a handful of chats, treat the result as inconclusive and extend the test or move on.
Which metric should I use for a WhatsApp or Messenger flow?
Pick the one that matches the goal of the flow, such as the share of chats resolved without handoff, or the share that leave contact details. Add a guardrail such as thumbs-down share, and fix both before the test starts.
Is it safe to roll back if variant B performs worse?
Yes. Restore points let you return an agent to an earlier state, and you can keep variant A as its own agent throughout the test, so there is always an unchanged version to fall back to.


