How to review AI support conversations: a weekly routine that catches silent failures

Riellvriany Indriawan
Written by

Riellvriany Indriawan

Katelin Teen
Reviewed by

Katelin Teen

Last edited November 16, 2026

Expert Verified
Hand-drawn support lead reading a stack of AI chat transcripts with a magnifying glass and a red pen, sticky labels on a few pages and one flagged for a fix
TL;DR

To review AI support conversations, read every escalation, low-CSAT, reopened and refund or policy conversation, plus a random 20 to 30 of the ones marked resolved, because "resolved" often just means the customer went quiet. Check each for a correct, sourced answer, policy, actions, handoff timing and tone. Then label the root cause (missing knowledge, instruction gap, wrong action, late handoff), fix it at that source and replay the same past tickets before the change goes live. Review daily in week one, then weekly. eesel shows the sources and reasoning behind every reply and replays resolved tickets with nothing sent.

Why doesn't "resolved" mean the AI got it right?

I work eesel's support queue, and the thing I've learned to distrust most is a green "resolved" badge. An AI agent can close a conversation with a fluent, polite, wrong answer, and nothing in the dashboard flinches. The customer either gives up or comes back angry two days later on a different channel.

The vendors say this themselves. HubSpot sets resolution status on a customer agent conversation 72 hours after the last visitor response, and later negative feedback "will not change the resolution status." The same page is blunt: "Deflections do not always indicate resolution." Zendesk's new quality evaluation dashboard, released on November 10, 2026, scores comprehension and empathy, then warns that "Comprehension does not mean the issue was resolved." The deflection vs resolution gap has its own write-up.

Here's what that looks like from the operator side, from a team running a self-hosted support bot:

Reddit

"just ast week a customer sent a screenshot where the bot told him accidental damage was covered under warranty and we'd repair it for free. It isn't and we don't. The doc it pulled was about the extended plan, which he doesn't have, and the bot blended the two we had to honor it ... haunting part is nothing in our logs looked obviously wrong. The answer was fluent and on topic, it was just built on the wrong chunk"

eesel has seen the same pattern on its own rollouts. On one trial, a B2B technical support team watched the agent tell customers a product was supported when it wasn't, because a help article said, word for word, "we support all models." The agent wasn't making things up. It read a sentence that was too broad and repeated it with confidence. That kind of failure only shows up when a person reads the conversation next to the source.

A 2x2 grid: answer right or wrong across the bottom, marked resolved or escalated up the side, with the "silent failure" square highlighted and a note that only a random sample finds these
A 2x2 grid: answer right or wrong across the bottom, marked resolved or escalated up the side, with the "silent failure" square highlighted and a note that only a random sample finds these

Escalations land in front of your team anyway. The silent failures in the top-right square don't, which is why the routine below spends part of its time on conversations that look fine. It is a different job from support QA for human agents: you're not coaching a person, you're finding which input to the agent was wrong.

If your team uses coding agents, the eesel CLI lets one take part in this loop. After you fix a rule, the agent can send the correction with eesel chat, read the saved rule back with eesel instructions, then ask the questions from the failed tickets again and check the answers before anything changes for customers. Connecting the helpdesk in the first place still needs someone with admin access to approve it.

Which AI support conversations should you pull?

You can't read everything, and you shouldn't try. A CX lead at a supplements brand doing around 7,000 tickets a month put it plainly on a call with eesel's sales team:

"The AI will never be able to answer 100% of the questions, but if it tries and just answers 'sorry I don't know this,' I cannot go and check all my 7,000 tickets to see if the AI actually made a good answer"

He's right about the math. The answer is a sampling plan with two parts: read every conversation in the risky buckets, and take a small random scoop from the big "resolved" bucket.

Five bins feed one reviewer tray: all escalations, low CSAT or thumbs-down, reopened tickets, and refund, billing and policy topics are each read in full, while only a random 20 to 30 is taken from the larger marked-resolved bin
Five bins feed one reviewer tray: all escalations, low CSAT or thumbs-down, reopened tickets, and refund, billing and policy topics are each read in full, while only a random 20 to 30 is taken from the larger marked-resolved bin
BucketHow many to readWhy it's in the pile
All escalations and handoffsEvery oneShows what the agent couldn't do, and whether it gave up too early or too late
Low CSAT or thumbs-downEvery oneCustomers telling you something went wrong, even when the ticket closed
Reopened ticketsEvery oneThe answer didn't hold. Often a wrong or partial answer marked resolved
Refund, billing and policy topicsEvery one at firstA wrong answer here costs money or a promise you have to honor
Random slice of "resolved"20 to 30 a weekThe only way to find silent failures

A note on CSAT: an AI-generated satisfaction score is not the same as asking the customer. eesel's own docs say "AI CSAT is not customer feedback" in its Reports page, and treat it as an internal signal only. Use real CSAT for AI replies where you have them.

Where you pull these from depends on your helpdesk, and every major one now has a place for it:

  • Zendesk. Conversation logs sit under Optimize > Monitoring > Conversation logs and show the past 7 days by default. The filters map well onto the buckets above: messages not understood, drop-offs, escalation failed, BSAT, and whether a knowledge answer was used. The Zendesk conversation logs guide walks through them.
  • Freshdesk. In AI Agent Studio, Analyze > Ticket logs filters by status, including agent hand-off, and separates real conversations from preview tests. Freshdesk's own advice is that "reading two or three actual conversations is the fastest way" to confirm a topic is failing (Freshdesk docs). More on its analytics here.
  • Gorgias. The AI Agent ticket view has a To review section that collects every ticket where AI Agent sent a response, and an ai_failed_action tag for failed actions (Gorgias docs). A ticket tagged both ai_answered and ai_handover is worth reading first: it replied, then escalated anyway.
  • HubSpot. The customer agent groups failures under Analyze > Coaching opportunities, with the exact message it couldn't answer highlighted in red.

One practical detail: the last 72 hours are still moving. Gorgias only bills an interaction as automated once 72 hours have passed without a human handover, and HubSpot waits the same 72 hours. Pull last week's conversations, not yesterday's, for anything you plan to count.

What should you check in each conversation?

Six checks per conversation. Most take seconds once you're used to them. The slow one is grounding, and it's the one that catches the warranty-style failures.

CheckWhat to look atRed flag
Correct answerWould your best agent have said the same thing?Right topic, wrong detail (plan, region, date, amount)
Grounded in a sourceWhich article, macro or past ticket did it use?No source, the wrong source, or two sources blended
Policy followedRefund windows, eligibility, what needs approvalPromised something your policy doesn't allow
Actions taken correctlyRefunds, cancellations, tags, order lookupsAn action fired on the wrong order, or failed silently
Handoff timingWhen it escalated, and what the handoff note saidLooped after the customer asked for a person, or the note doesn't match the transcript
ToneDoes it sound like your team?Over-apologizing, robotic, or chirpy with an upset customer

Grounding needs the agent's working, not just its reply. Zendesk's per-message View details shows the agent's Plan, the response before customization and the Active instructions it followed, which is how you tell a knowledge problem from an instruction problem.

Zendesk AI Agent Message Overview panel showing the use case, the agent's Plan, the response before customization, the Active Instructions it followed and its personalization settings, as taken from Zendesk's help center
Zendesk AI Agent Message Overview panel showing the use case, the agent's Plan, the response before customization, the Active Instructions it followed and its personalization settings, as taken from Zendesk's help center

For the policy check, pay special attention to money. If your agent can issue refunds, read every refund conversation for the first month, whether or not it was flagged. The post on refund guardrails covers which limits belong in settings rather than in the prompt.

Treat the handoff note as its own check. It's easy to read the AI's reply, see it escalated, and move on. But the note your human agent receives can be wrong too:

Reddit

"My place added an AI bot that handles the chat first and gives the summary of what the client said to the agent, but the thing is, its summaries are often just lies that it hallucinated lol. It just makes up stuff that the customer never said, and you still have to check the whole history."

Compare the summary with the transcript. If they disagree, that's a failure even if the escalation itself was right. The handoff best practices post covers what a good note contains.

Tone is the last check and the easiest to over-weight. A slightly stiff reply that's correct beats a warm one that's wrong. Fix tone in bulk once you have several examples, using a brand voice rule rather than one-off edits.

How do you label what went wrong?

Once a conversation fails a check, give it one root-cause label. The label decides where the fix goes, and the usual mistake is fixing everything in the prompt. I use four labels:

  1. Missing or outdated knowledge. The fact wasn't in any source, or the source was wrong.
  2. Instruction gap. The facts were right, but the agent did the wrong thing with them.
  3. Wrong action or permission. It took, skipped or botched an action, or was allowed to do something it shouldn't.
  4. Should have handed off. It kept going when a person should have taken over, or handed off something it could have answered.

Your helpdesk may already have a version of this. HubSpot's coaching opportunities sort failures by Reason: Knowledge, Handoff, Experience, Action and Other (HubSpot docs). Zendesk lets you add labels to conversations and warns against a unique label per conversation, since labels are "meant to group conversations for analysis." Keep your labels to a short, fixed list so the counts mean something.

HubSpot help desk insight on a customer agent reply, showing the knowledge source it used, a Knowledge gap flag, and the Rate reply and Coach agent controls, as taken from HubSpot's knowledge base
HubSpot help desk insight on a customer agent reply, showing the knowledge source it used, a Knowledge gap flag, and the Rate reply and Coach agent controls, as taken from HubSpot's knowledge base

One trap to know about: in some tools, rating a conversation doesn't change the agent at all. Gorgias says its Bad, Okay and Good ratings "don't immediately change the way AI Agent behaves in conversations with your customers"; to change behavior you rate or add sources instead (Gorgias docs). Zendesk's built-in feedback options for generative replies (Incomplete answer, Made-up facts, Missing article and so on) go to Zendesk's AI researchers. Both are useful, but neither is a fix on your side.

Pick a failure type below to see what it looks like in a transcript, where the fix goes in each helpdesk, and how to retest it.

Failure type to fix map

Label the failure, then fix it where it lives. A fix in the wrong place hides the problem for one ticket and leaves it for the next.

In the transcript

  • It says it can't find the answer, or escalates with no source
  • It cites an old version of a policy
  • It blends two articles into one answer

Where the fix goes

  • Write or update the help article or source doc, not the prompt
  • Zendesk: Missing article or Incorrect article used
  • Freshdesk: Improve tab, New content or Edit content
  • Gorgias: Was something missing? > Add source
  • HubSpot: Knowledge gap or Knowledge conflict
  • eesel: the knowledge gap flag on the run

How to retest

Replay the tickets that failed and check that the new or edited source is the one it cites.

Rule of thumb: if the right fact isn't written down anywhere, no instruction will fix it.

In the transcript

  • The facts were right, the behavior wasn't
  • It offered a discount or credit it shouldn't
  • It skipped a step, like asking for the order number
  • Wrong tone for an upset customer

Where the fix goes

  • One rule in the instructions; edit the existing rule instead of adding a new one
  • Zendesk: the Active instructions on the procedure
  • Gorgias: Guidance
  • HubSpot: Coach agent > Experience tab
  • eesel: correct it in the chat beside the run

How to retest

Replay the failed tickets plus a few that passed before, so the new rule doesn't break answers that were fine.

Rule of thumb: instructions shape behavior, not facts.

In the transcript

  • An action ran on the wrong order or customer
  • An action failed and the reply pretended it worked
  • It promised a refund or cancellation the system never made

Where the fix goes

  • Action settings and permissions, not the prompt
  • Gorgias: Actions > View Events, plus the ai_failed_action tag
  • Zendesk: the API error on the integration step
  • HubSpot: Coach agent > Actions tab
  • eesel: set the action to Ask first or Off

How to retest

Test against a fake customer or with actions held for approval. Live actions in a test can change real order data.

Rule of thumb: anything that costs money stays behind an approval until the drafts are right every time.

In the transcript

  • The customer asked for a person and it kept answering
  • It looped on the same answer
  • It escalated an easy question it could have answered
  • The handoff note doesn't match what the customer said

Where the fix goes

  • Handoff rules and topics
  • Gorgias: Settings > Handover and exclusion > Handover topics
  • HubSpot: Train > Human handoff guidelines
  • Zendesk: escalation rules on the AI agent
  • eesel: escalation rules in the instructions

How to retest

Replay the failed ticket, plus a rephrased version where the customer is angrier or asks twice.

Rule of thumb: a late handoff costs more than an early one.

Pick a failure type to see where its fix goes in Zendesk, Freshdesk, Gorgias, HubSpot and eesel.

The best proof of this approach I know is Qontro, which makes ERP software and runs eesel on Freshdesk as a co-pilot. The team compares each eesel draft with the reply a person actually sent. Where the draft was wrong, that usually points to a missing or incomplete help article, or a rule that needs tightening, and they fix that rather than the single ticket. About one in ten drafts now goes out with little or no editing, and the share is rising slowly. That's the four labels working in practice, on about 190 Freshdesk tickets drafted a month.

When one label keeps coming up, step back from single tickets. Lots of "missing or outdated knowledge" means it's time for a knowledge base audit, and AI can help with detecting outdated content before the agent repeats it.

How do you fix it and retest before it goes live?

Fix it at the source the label points to, then replay the same past tickets that failed. Not new questions you invent, the actual tickets. A fix that passes your own test question can still fail on the customer's wording, and the failed tickets are the wording you already know breaks things.

One commenter described the habit that makes this compound:

Reddit

"Keep the transcript reads too, but let them compound instead of evaporate: every escalation like this becomes a case in a small regression set, re-run on every prompt, chunking, or model change, so the failure classes you've already paid for stay fixed."

Each helpdesk retests a little differently, and the difference matters:

ToolWhere you retestWhat the test questions are
ZendeskTest widget and the conversation simulatorScenarios from a library or ones you write
FreshdeskTest tab, up to 100 queries per runPasted queries, or 50 generated from your knowledge
GorgiasPlayground, including draft skills and articlesMessages you type, as a new or existing customer
HubSpotBatch testing, up to 8 test setsGenerated from live conversations (5+ similar per theme) or written by you
eeselHelpdesk simulationYour resolved past tickets, scored against what your team sent

Gorgias adds a warning worth copying into your own runbook: if you turn actions on in a test, they "can affect real customer and order data" (Gorgias docs). Use a fake customer profile.

The reason eesel works from past tickets is that it's the closest thing to a dry run on your real queue. Its simulation replays resolved tickets, drafts a reply to each, puts it next to what your team sent, and groups the results by theme. "It sends nothing to customers," per eesel's docs. The example run in the helpdesk docs sampled 20 resolved tickets, matched 17 of them, and returned five ranked fixes. That 3-in-20 miss list is exactly the review pile you want before customers see anything.

A horizontal loop of five steps: pull the sample, check each one, label the root cause, fix at the source, replay the same tickets; passing goes to go live, still failing loops back, with week 1 daily and weekly after that noted above
A horizontal loop of five steps: pull the sample, check each one, label the root cause, fix at the source, replay the same tickets; passing goes to go live, still failing loops back, with week 1 daily and weekly after that noted above

Only go live when the replay passes. If it still fails, the label was probably wrong, which happens. A "missing knowledge" failure that persists after you add the article can be an instruction that tells the agent to prefer a different source. For more on test design, see the guide to AI support agent testing.

How often should you review AI support conversations?

Daily in the first week, then weekly. That's the cadence I'd start with for any new agent, or any agent you've just given a new channel or action.

Week one, daily. Read every flagged conversation and about 10 random resolved ones each day. Failures cluster early because the first real customers find the gaps your test questions missed.

After that, weekly. Every flagged conversation plus 20 to 30 random resolved ones, and a look at the label counts. If one label's count drops after a fix, it worked. If the random slice stops turning up failures for three or four weeks, shrink it, never to zero.

Go back to daily after any big change. A new channel, a new action, a model change, or a large knowledge import resets the clock.

How much you review also depends on how much the agent does unsupervised. In eesel, every action is set to On, Ask first or Off, and the actions docs suggest starting supervised and moving routine actions to On "once the drafts are consistently right." One exception: live chat replies are always On, because nobody can pause a live chat for approval. That makes chat the channel where the random sample matters most.

The eesel Reports page has an Approvals area that shows the same thing from the other side. Its docs say an action "approved every time without edits is ready to run on its own," while one that "keeps getting rejected needs an instructions fix."

eesel Reports page showing the total number of tasks over the last 30 days, trigger events by type, and the start of the approval and rejection usage per tool chart
eesel Reports page showing the total number of tasks over the last 30 days, trigger events by type, and the start of the approval and rejection usage per tool chart

To keep the weekly pass from slipping, put it on a schedule. eesel has a Weekly digest skill that summarizes conversations handled by topic and channel, approvals waiting, and what it skipped and why, and the card sets it up to send every Monday (eesel docs). Whatever tool you use, the digest should end with a short list of conversations for a person to read, not just numbers.

What mistakes make a review routine fall apart?

The common ones:

  • Only reading escalations. You'll learn what the agent can't do, and miss what it does wrong with confidence. Keep the random resolved slice.
  • Fixing everything in the prompt. A missing fact written into the instructions works for one ticket and goes stale the day the policy changes. Facts go in your knowledge base or macros.
  • Trusting the resolution rate as a score. Use it to decide where to look, not as proof the answers were right.
  • Retesting with new questions only. Replay the tickets that failed. Otherwise you never prove the fix worked on the wording that broke it.
  • No owner. Reviews done "when someone has time" quietly stop. Name one person per week, and use QA calibration if several people share it so the labels stay consistent.
  • Using a human QA scorecard as is. Scorecards built for people, like the ones in Zendesk QA, score empathy and process. Add the grounding and actions checks, or you'll score a polite wrong answer as a pass.

The cost of skipping the labels shows up fast. When a team can't tell which AI decisions are wrong, it ends up checking all of them, as this Hacker News comment describes:

Hacker News

"The AI would confidently misroute 15-20% of tickets, requiring human review of ALL AI decisions"

Review your AI support agent's work with eesel

eesel is an AI helpdesk teammate that joins your existing queue in Zendesk, Freshdesk, Gorgias or HubSpot. It was built to be reviewed. Every run lands in Activity with a link to the ticket, what triggered it, which sources it searched, every action it took, its reasoning step by step, and whether it hit a knowledge gap. If an answer is wrong, you open the run and tell it what to change in the chat beside it, and the fix goes into its instructions or a memory.

eesel Activity list showing each conversation the agent handled, with filters for 24 hours, 7 days, 30 days and all time, and status chips for approved, rejected and pending
eesel Activity list showing each conversation the agent handled, with filters for 24 hours, 7 days, 30 days and all time, and status chips for approved, rejected and pending

You can also choose how much it does alone while you build trust. STIL Classics runs eesel on Gorgias as drafts: in one month it left about 430 internal notes, most of them draft replies for the team to check and send, and when a fact is missing the draft leaves a gap in brackets rather than guessing. That makes review fast, because the gaps point straight at the missing knowledge.

SamCart runs separate agents on Freshdesk tickets and its website chat, and when a seller asks for a person or the question goes beyond the help center, the ticket is tagged for handoff and the agent stops replying. That tag gives the weekly review a clean bucket to read.

Before anything goes live, the simulation replays your resolved tickets and shows where the agent and your team disagree, with nothing sent. Plans start at $299 a month for 500 credits after a 7-day free trial, per eesel pricing.

Try eesel on your own past tickets and see what your review pile looks like before your customers do.

Frequently Asked Questions

How do you review AI support conversations without reading all of them?
Read every escalation, low-CSAT, reopened and refund or policy conversation, then add a random 20 to 30 from the ones marked resolved. That covers the risky slice and the silent failures a dashboard can't see. A tool like eesel logs the sources and reasoning for each run, which makes each read faster.
How many AI support conversations should I review each week?
In week one, review daily: every flagged conversation plus 10 or so random resolved ones. After that, a weekly pass of every flagged conversation and 20 to 30 random resolved ones is enough for most teams. Grow the random slice if it keeps turning up problems, the same way you would with AI support QA.
What should I check when reviewing an AI agent conversation?
Six things: was the answer correct, was it grounded in a real source, did it follow policy, were actions taken correctly, did it hand off at the right moment, and was the tone right. The grounding check catches the confident answers that read well but cite the wrong article.
Is reviewing AI support conversations different from support QA for human agents?
Yes. Human QA scores a person against a rubric and ends in coaching. Reviewing an AI agent ends in a change to its knowledge, instructions, actions or handoff rules, then a retest. The scoring habits from QA calibration still help when several reviewers share the work.
How do I know if an AI support answer failed because of knowledge or instructions?
Look at what the agent searched and cited. If the source was missing, outdated or contradicted the policy, it is a knowledge problem, so fix the article. If the source was right and the agent still did the wrong thing, it is an instruction gap. A knowledge base audit helps when many failures point at the same articles.
How do I retest an AI support agent after fixing a problem?
Replay the same past tickets that failed, plus a few that passed, and compare the new answers. eesel's simulation replays resolved tickets and scores them against what your team sent, with nothing going to customers. HubSpot and Freshdesk offer batch tests from written or generated questions.
Can I trust my helpdesk's AI resolution rate instead of reviewing conversations?
Not on its own. HubSpot sets resolution status 72 hours after the last visitor reply and says deflections do not always indicate resolution. Use the rate to decide where to look, then read conversations to see why. The guide to resolution rate covers the numbers.
Which tools help review AI support agent conversations?
Most helpdesks now ship one: Zendesk conversation logs, Freshdesk ticket logs, Gorgias's To review view and HubSpot's coaching opportunities. eesel adds an Activity log with sources and reasoning for every run, plus plans that include reports and simulation. For QA scorecards, see the best AI for support QA.

Share this article