Five reviewed tickets per agent per month gives a quality score that can be off by about 31 points, so the fix is not a better spreadsheet, it is scoring every ticket with AI and treating the result as a signal a person checks. Zendesk, Gorgias, Front, HubSpot and Freshdesk each ship some version of this, and they cover very different slices of a ticket (Gorgias scores 3 criteria with AI, HubSpot scores 3, Zendesk lets you write your own).
The part most guides miss: once an AI agent takes the repetitive tickets, the tickets left for your humans are harder, so handle time rises, volume per agent falls, and a scorecard built before AI will mark your best people down. At Question Beauty, eesel closed about 280 of 990 Gorgias tickets on its own in a month and assigned about 250 to a person with a note, so the human queue is a different job from the one the old targets were set for. I walk through the sample-size math, the re-weighted scorecard I would start from, and how to keep AI scores fair.
How many reviews does one agent score actually need?
Far more than five. A score from five reviewed tickets is mostly luck, and the math is the standard one for any pass rate.
Take an agent who truly passes a criterion 85% of the time. With 5 reviews a month, the 95% margin of error is about 31 points: the dashboard could say 55% or 100% for the same person in the same month. At 20 reviews it is about 16 points, at 100 about 7, and at 500 (every ticket) about 3. That is my own arithmetic using the normal approximation, with a finite-population correction in the widget below.
It gets worse when you compare people. To tell an 85% agent from a 75% agent with confidence, you need about 120 scored tickets each. A team lead reviewing 5-10 tickets a week out of one agent's roughly 500 monthly tickets never gets there. Leo Gasperin described exactly that on his old team, and a commenter gave it a name:
"sampling creates confidence, but not clarity."
Try your own numbers. The widget below uses your ticket volume and your review count.
Type your own ticket volume and review count to see how wide the range is. The last card is the review count needed to separate two agents 10 points apart.
I covered the broader case for dropping sampling in call center quality assurance, and the hands-on version in support QA with AI. The short version here: AI scoring is worth buying for the denominator alone.
What should an agent performance scorecard measure?
Four things, none of which is speed on its own: quality, outcome, customer view, and efficiency per ticket type. The usual advice is a weighted score (say quality 40, productivity 25, customer experience 25, reliability 10), and that shape is fine. The weights are where teams go wrong.
| Bucket | Example measure | What it tells you | The trap |
|---|---|---|---|
| Quality | AI-scored empathy, accuracy, resolution | Did the reply do the job? | A criterion the AI cannot judge (Gorgias leaves accuracy and brand voice to humans) |
| Outcome | Reopen rate, first contact resolution | Did it stay fixed? | Agents who close fast and reopen later |
| Customer view | CSAT | Did the customer feel helped? | Few survey responses per agent, so the same noise problem |
| Efficiency | Average handle time per ticket type | Is the work well organised? | One blended number across easy and hard tickets |
For the full metric list, my posts on customer service KPIs and call center agent productivity go deeper. If you are writing review criteria from scratch, QA feedback examples gives you wording that coaches instead of grades.
What changes once an AI agent takes the easy tickets?
Your human queue gets harder, and every per-agent number you set before AI is now measuring a different job. AI answers the order-status and how-do-I questions, so what reaches a person is the refund decision, the angry customer, and the thing the bot could not solve.
A call center agent on Reddit described it after an AI voice agent went in front of the phones:
"At this point, it's handling a lot of "routine" calls. Which means the calls that come through to us humans are more complicated, or people who are upset about the AI messing something up."
The numbers back the feeling. At Question Beauty, eesel handled about 990 Gorgias tickets in a month (8 August to 8 September 2026), sent about 960 replies, closed about 280 tickets itself and assigned about 250 to a person with an internal note. Refunds, complaints and wholesale requests always go to a human. That team's agents are not slower than before; they simply never see "where is my order" any more. Handle time per ticket goes up and tickets per agent go down, and neither is a performance problem.
This is the re-weighting I would start from. They are my starting weights, not a benchmark, so move them after your first calibration round. What changes:
- Drop volume targets for individuals. Tickets closed per agent falls by design once AI deflects the easy ones. Track total agent hours on the queue instead, as I explain in AI for reducing handle time.
- Measure speed per ticket type. A refund ticket and a password reset should never share a benchmark. Gorgias already calculates ticket handle time only on tickets a human worked, and leaves out tickets resolved by its AI Agent. Copy that rule in any helpdesk.
- Add a line for working with the AI. Agents now approve, edit or reject drafts and receive handoffs. At smava, eesel prepared about 15,300 draft replies in a month on a shared Zendesk, and an agent approves a draft or writes @eesel in a note to ask for a different macro. Reviewing that well is a skill: a score for "caught a wrong draft" and "left a useful handoff note" belongs on the card.
- Review the handoff, not just the reply. If the AI hands over with the order already checked, that is handoff quality, and it affects the agent's numbers more than their typing speed does.
A Reddit commenter with close to 30 years in support put the cost of ignoring this plainly:
"Except now imagine an eight-hour shift where practically EVERY conversation you receive is an escalation, an angry customer, an unusual problem or something the AI already couldn't solve."
That is also a coaching and rota question, and I cover the staffing side in AI for support staffing and the shift side in AI for shift handover.
Which helpdesk tools score agent quality with AI?
Five helpdesks ship something native, and they differ more than the marketing suggests. I read each vendor's own docs and pricing page, so the table below uses their wording and prices only.
| Helpdesk | What it scores | How much of the ticket the AI judges | Price as listed |
|---|---|---|---|
| Zendesk QA | Every interaction, human and AI agent; built-in categories such as Empathy or Solution, plus your own prompt-based categories | Whatever you write into the scorecard | QA add-on on every Suite plan, price not listed; Workforce Engagement Bundle is $50 per agent per month billed annually |
| Gorgias Auto QA | Closed tickets, scored 12 hours after close | 3 criteria (resolution completeness, communication, language); 4 more (accuracy, efficiency, internal compliance, brand voice) are manual only | On by default with an AI Agent subscription, which is $1.50 per automated interaction past the plan limit |
| Front Smart QA | Every ticket against customizable scorecards, plus Smart CSAT | Criteria you define | Listed as included on Enterprise, $105 per seat per month billed annually |
| HubSpot QA score | Closed help desk tickets, 1 to 5 | 3 criteria (tone, clarity, empathy); beta | Service Hub Enterprise, $150 per seat per month |
| Freshdesk Quality Coach | Resolved conversations, shown to agents in their inbox, plus an admin Quality Score report | 6 parameters (opening and closing, timeliness, quality of response, sentiment, grammar, filler words) | Enabled under the Freddy Copilot settings; Freddy AI Copilot is $29 per agent per month on Pro and Enterprise |
A few things I noticed reading them side by side:
- Zendesk is the most flexible and the least transparent on price. You write the criteria, and Spotlight flags churn risks and stuck loops. Its AI Agent QA also compares scores for bots and humans side by side. The cost is a sales conversation. I go through setup in Zendesk QA scorecard criteria and Zendesk QA for AI agents.
- Gorgias is the most honest about its limits. Its docs say plainly that accuracy and brand voice can only be scored manually, that custom criteria are not available at the moment, and that tickets under 250 characters, social tickets and phone calls are skipped. That is useful: you know which half of your scorecard still needs a human. See Gorgias AI Agent pricing for the cost side.
- Front bundles QA with CSAT inference, so you get a quality view and an AI-inferred satisfaction view in one place. Front pricing has the plan detail.
- HubSpot's is the most cautious. It is beta and Enterprise-only; whether that is worth the seat price is a question I take up in HubSpot Service Hub AI.
- Freshdesk's is the oldest, and the help article is from 2024, so check it against your plan before building a process on it. Plans are in Freshdesk pricing.
If you are on none of these or want a vendor-neutral view, customer support quality assurance tools compares the standalone QA platforms.
How do you keep AI scores fair to agents?
Treat every score as a signal that picks tickets for a human to read, never as a verdict. The vendors say so themselves: HubSpot's help article warns that its QA scores may be inaccurate and should not be the sole basis for coaching, performance, compensation, discipline or other decisions about a person.
The loop I would run:
- Score every ticket with AI. This is the denominator fix from the first section.
- Read the flagged ones yourself. Twenty flagged tickets a week per team beats five random ones per agent, because you are reading where the signal is.
- Calibrate. Have two humans score the same 20 tickets, then compare against the AI. Where they disagree, fix the criterion wording, not the people. A G2 thread on this asks exactly how teams keep human calibration alongside AI scoring, and the usual answer is that AI is the first pass and human sessions stay.
- Let agents dispute. Gorgias lets a reviewer edit any part of a score and uses those edits to fine-tune the model. Even where your tool does not, an agent who can say "this score is wrong, here is why" makes the system more accurate and more trusted.
- Never discipline off sentiment alone. Sentiment and tone scores punish people who get the angriest customers, which, after AI deflection, is everyone.
My post on AI sentiment analysis for support explains why sentiment is a poor judge of an individual agent. Fairness is also why I would test a scorer before trusting it, the same way you would simulate an AI agent.
How do you turn scores into coaching?
Pick one behaviour per agent per month, show them two tickets, and agree one change. Scores find the tickets; the conversation does the work.
A 30-day plan, in the order I would do it:
- Week 1, set the denominator. Turn on your helpdesk's AI QA and let it score every closed ticket. Do not look at per-agent rankings yet.
- Week 2, calibrate. Run the 20-ticket check above and rewrite any criterion the humans and the AI read differently.
- Week 3, re-weight. Move to the post-AI scorecard: no individual volume target, speed per ticket type, a line for draft review and handoffs.
- Week 4, start one-on-ones. One strength and one habit per agent, with the two tickets as evidence. coaching agents with AI and customer support coaching software cover the tooling, and AI agent coaching for support covers coaching the AI itself.
For new hires the same loop works faster, since AI drafts act as worked examples: see customer support agent onboarding.
How does eesel fit into agent performance?
eesel does not score your human agents. It changes what they work on. eesel is an AI helpdesk teammate that sits in your existing helpdesk, answers or drafts the repetitive tickets, and hands the rest to a person with a note, so the score you build for your team reflects the work that is left. eesel has run AI on live support queues for years, and every rollout is simulated against historical tickets first, which gives you the before-and-after view of the human queue that a fair scorecard needs.
Three published customer stories show the pattern on different helpdesks:
- Gorgias: Question Beauty had eesel send about 960 replies and close about 280 of 990 tickets in a month; about 250 went to a person with the order lookup already attached.
- Zendesk: smava handles about 27,000 tickets a month on one Zendesk across two brands. eesel categorised about 14,600 of them and prepared about 15,300 drafts, and people approve or redirect each draft.
- Freshdesk: SamCart has eesel reply on tickets only while no agent is assigned, and a teammate taking the ticket ends the agent's part, which is a clean line for deciding whose tickets belong on whose scorecard.
To judge the AI itself, use the AI support pilot scorecard, the AI resolution rate guide, and containment and escalation quality. For the cost side, AI versus human agent cost has the comparison.
Try eesel for the tickets your team should not be scoring on
If your agents are being scored on password resets and order-status questions, hand those to eesel first and re-weight the scorecard around what is left. It connects to Zendesk, Freshdesk, Gorgias, HubSpot and Front, and it learns from your help center and past tickets.
eesel pricing is fixed credit plans: a ticket or a chat is 1 credit however long it runs, the free plan has 100 credits to try every feature, and paid plans start at $299 a month for 500 credits. Try eesel with your own tickets and see how much of your queue is the easy kind.