Help Scout AI quality assurance: what it measures, and how to score the rest

Rama Adi
Written by

Rama Adi

Katelin Teen
Reviewed by

Katelin Teen

Last edited November 21, 2026

Expert Verified
Hand-drawn illustration of a smiling support lead holding a clipboard with three blue check marks, next to two chat bubbles, a laptop and the Help Scout logo on a blue background
TL;DR

Help Scout has no AI QA scorecard: nothing in it scores your team's conversations for accuracy, tone or process; the quality signals it does have are customer ratings and a report on whether its AI agent resolved each chat. A QA lead gets the customer's own rating on each reply (Great, Okay or Not Good, rolled up into a Happiness Score), a Scout report that marks each AI conversation Resolved, Unresolved or escalated, and read-only access for Claude or ChatGPT through Help Scout's MCP server. So Help Scout tells you how the customers who click a rating felt, and leaves "was the answer right for our policy?" to you. You can cover that with a weekly hand-scored sample, an AI assistant reading conversations over MCP, or an AI teammate like eesel that scores every closed conversation against a rubric you write and leaves its reasons as an internal note. A coding agent such as Claude Code can write that rubric into eesel and set up the schedule through the eesel CLI, though a Help Scout admin still approves the connection.

Does Help Scout have AI quality assurance?

No, not as a scorecard. I build eesel's integrations, so I read helpdesk docs for what a feature actually does under the marketing copy. Help Scout's AI features page lists Scout (its AI agent), AI Drafts, AI Summarize, AI Assist, Topics and an MCP server. None of them scores a conversation your team handled against a set of quality assurance criteria.

That fits how Help Scout describes its AI: built around how it "can be used to improve support", not around deflecting tickets. It also means a support lead who wants to know whether replies were accurate has to put the pieces together. Here is what each piece measures:

Help Scout featurePlanWhat it tells a QA leadWhat it doesn't
Happiness ratingsStandard keeps 2 years of reports; Plus and Pro, the account's lifetimeHow customers who clicked a rating felt about a replyAnything about replies nobody rated
Scout reportScout is on Standard, Plus and Pro, billed per resolutionWhich AI conversations were resolved, unresolved or handed to a personWhether a resolved answer was correct
AI AssistStandard, Plus and ProFixes spelling and grammar, and adjusts tone, before a reply is sentScores nothing after the fact
TopicsPlus and ProWhat customers write in about, by themeHow well the team answered
MCP serverStandard, Plus and Pro (report tools need Plus or Pro)Lets your own AI assistant read conversations and reportsIt can't write notes or tags back
Hand-drawn chart in two columns: what Help Scout gives you (Happiness ratings, the Scout report, views that surface low ratings, AI Assist fixing spelling before you send, and MCP read-only access) and what it doesn't have (a QA scorecard, reviewer scores on a conversation, criteria you write), with a note that ratings come from the customers who click
Plenty of signals, and no place to score a conversation yourself.

What do Help Scout's Happiness ratings tell you?

They tell you how the customer felt about a reply, in their own words. Turn on satisfaction ratings per inbox under Settings, Inboxes, then Satisfaction Ratings, and every email reply carries rating links. The customer picks Great, Okay or Not Good and can add a comment.

The Help Scout rating page asking How would you rate the support you received, with Great selected over Okay and Not Good, a comment box and a Send button
One click per reply, and an optional comment, as shown in Help Scout's ratings docs.

The rules matter if you plan to read it as a quality signal:

  • Every reply can be rated. If an agent sends five replies, the customer can rate all five, one rating each.
  • The window is 10 days. Customers can rate, or change a rating, for 10 days after the reply. Ratings can't be deleted.
  • The score is a net number. The Happiness report shows the share of Great, Okay and Not Good, and a Happiness Score that is the share of Great minus the share of Not Good, calculated much like NPS.
  • Ratings count on the reply's date, not the date the customer clicked, so last week's number can still move. My notes on Happiness report dates cover what that does to a weekly review.
  • Each agent has their own tab. Open the Company report, then a User report, for one person's ratings.
The Help Scout Happiness report showing 67 percent Great, 33 percent Okay and 0 percent Not Good, with a Happiness Score of 67 based on 3 ratings from 23 percent of customers
Three ratings from 23% of customers, as shown in Help Scout's Happiness report docs.

Look at the small print in that screenshot: three ratings from 23% of customers. That is the limit of ratings as QA. Like any CSAT survey, they hear from the customers who bother to click, and a customer who got a confidently wrong refund answer may rate it Great until the refund doesn't arrive. A rating says how the reply felt. It doesn't say whether the reply followed your policy, which is the gap AI CSAT tools try to close.

Ratings are still a good place to start a review. Since May 2026, Help Scout's views can filter on what happens inside a conversation, so you can "spot low-rated interactions that need follow-up", and a workflow can email a lead when a rating comes in. If you want to know more about how customers feel, my post on Help Scout sentiment analysis goes further.

How do you check Scout's AI answers in Help Scout?

Use the Scout report, under Reports, then Scout. Scout is Help Scout's AI agent (the product you pay for is called Envoy, at $0.75 per resolution), and the report sorts every conversation it took part in into three groups:

  • Resolved: Scout gave an answer and the customer didn't say they needed more help.
  • Unresolved: the customer said they needed more help, or Scout escalated because of an exclusion, but the customer didn't reach out.
  • Human Escalation: the conversation went to your team.

Each Beacon (Help Scout's chat widget) has its own report, with a sessions table showing the customer's first question, the resolution, how the session ended, the rating and the Topic. Help Scout's Scout best practices say to sort by resolution, read the conversations where customers weren't helped, and add an Improvement where Scout should have answered better. Account Owners and Administrators can also see the sources Scout checked when it failed.

That is a sensible review loop for the AI's misses. Notice what "Resolved" means, though: the customer didn't ask for more help after the last answer. That is also when Help Scout bills a resolution. A wrong answer the customer believed counts as resolved, so it's worth reading a sample of resolved sessions too, not only the failures. Measuring AI containment and escalation covers that trap in more depth.

How much does QA in Help Scout cost?

Nothing on top of your plan: ratings, the Happiness report and the Scout report don't cost extra. What changes by plan is how far back reports go and which AI tools you get. From Help Scout's pricing page, checked 21 November 2026:

PlanPer user, billed annuallyMonthly billingWhat matters for QA
Free$0$0Up to 5 users, "No AI tools included"
Standard$25$30Happiness reports for the last 2 years, AI Assist, MCP access without report tools
Plus$45$54Lifetime reports, Topics, AI Drafts, MCP report tools
Pro$75$90Everything in Plus, minimum 10 users
Envoy (add-on)$0.75 per resolutionBilled monthly in arrearsScout and its report

For a team of eight on Plus, that is $360 a month billed annually, with the Scout report included if you also pay for Envoy resolutions. My Help Scout pricing notes cover the rest of each plan.

What does a QA scorecard need that Help Scout leaves to you?

Most QA scorecards (here are some scorecard criteria for reference) check three things Help Scout never scores: whether the answer was right for your policy, whether the agent followed your process (tags, saved replies, escalation), and whether it sounded like you. Those need someone, or something, that has read your rules.

Pick what you want to check and see what covers it:

What do you want to check on a Help Scout conversation?
Help Scout covers it: Happiness ratings
The customer rates each email reply, and the Happiness report turns that into a score per team and per person.
Who scores itThe customer, within 10 days
ScaleGreat, Okay or Not Good
Blind spotReplies nobody rated
Build a view of low-rated conversations and read those first.
Partly: the Scout report
Each AI conversation is marked Resolved, Unresolved or Human Escalation, with the customer's rating.
WhereReports, then Scout
Fix a missAdd an Improvement
Blind spotA wrong answer the customer believed counts as Resolved
Read a sample of resolved sessions too, not only the unresolved ones.
Before sending, not after: AI Assist
AI Assist fixes spelling and grammar, and makes a draft friendlier or more professional, when the agent asks it to.
PlanStandard, Plus and Pro
Who triggers itThe agent, per reply
Blind spotNo score on what was sent
To score tone after the fact, you need a reviewer reading sent replies.
Not in Help Scout: a person, or a reviewer given your policies
Nothing in Help Scout checks a reply against your refund rules, shipping times or plan limits.
By handA lead reads a weekly sample
With AIA reviewer that has your policy docs
Write firstThe rule each answer must follow
This is where wrong refunds and invented promises hide, so it's the check worth automating first.
Not in Help Scout: a reviewer that reads tags and notes
Tags, saved replies and escalation steps sit on the conversation, but nothing scores whether they were right.
By handA lead checks tags while reading
With AIA reviewer that reads the tags with the thread
Write firstWhich tag each topic gets
Topics can show you the themes, so you know which rules to write down.

Pick a check to see what covers it in Help Scout, and what covers it if nothing does.

There are three ways to cover the gap.

Score a sample by hand

The cheapest route needs nothing new, and it's the one most guides to support QA with AI start from. Each week, a lead opens a view of the last week's closed conversations (low-rated ones first), reads ten or twenty, and scores each against a short rubric in a spreadsheet. Agree first on what each score means, so two reviewers give the same conversation the same number: support QA calibration helps with that, and so do these QA feedback examples.

The limit is the obvious one: a person reads a sample, not every conversation, and the scores live outside Help Scout.

Ask an AI assistant over MCP

Help Scout's MCP server lets Claude, ChatGPT and Cursor list, search and summarize conversations, including their threads, tags and custom fields, and pull Happiness and other reports on Plus and Pro. Each person connects with their own Help Scout login, so the assistant sees only what that person can.

The Authorize Help Scout MCP screen, where Claude asks for Inbox access marked read-only, which reads conversations, customers, tags and users and can't create, edit or delete anything, plus optional Knowledge Base access with a Docs API key field
The connector asks for read-only inbox access, as shown in Help Scout's MCP docs.

That makes a decent weekly QA session: paste your rubric, ask the assistant to read last week's closed conversations in the Support inbox, and have it score each one with a reason. Two limits to know. New connections are read-only, so the assistant can't add notes or change tags on what it scored, and the results stay in your chat. And it runs when someone remembers to ask, not every morning. On Claude Team and Enterprise plans, only a Claude Owner can add the connector. My notes on Claude for Help Scout cover the setup.

Have an AI teammate score against your rubric

The third route is an AI agent that runs the review on its own, with your rules, and writes the result back into Help Scout. This is what I'd do with eesel. eesel's Help Scout integration can list conversations by status, tag or the day they were last updated, read each one with every reply, note and tag, leave an internal note and add tags.

Put that on a schedule and every morning it:

  1. Lists the conversations closed since yesterday.
  2. Reads each one against the criteria you wrote in plain words ("a refund offer must match the 30-day policy in the returns doc", "shipping questions get the shipping tag", "we sign off with the agent's first name").
  3. Leaves the score and the reasons as an internal note, and tags the misses qa-review so a lead can build a Help Scout view of them.
  4. Sends the lead a summary of the day, in Slack or by email if you connect them.

You write the rubric, so policy accuracy and voice are just more lines in it, and you change them whenever your policies change. It can grade Scout's conversations with the same rubric as your team's, and the same eesel agent can also draft replies or tag and route new conversations if you want it to. Two things to know before you start: eesel's notes show the name of the Help Scout user it connects as, so give it its own user, and it can't set custom fields, so scores live in notes, tags and the reports eesel writes, not in Help Scout's Happiness report.

Scheduled checks like this already run for eesel's customers. Cactus Club Cafe runs a second eesel agent on its Zendesk helpdesk that audits the daily finance extract every morning: it checks each store and either solves the ticket as balanced or leaves it open with the imbalance named. Between 8 August and 8 September 2026 it ran about 60 times. A QA review is the same shape: a job every morning, against written rules, with the result left on the ticket.

Is Help Scout's approach to quality enough?

Credit where it's due: Help Scout doesn't pretend. (Gorgias, by comparison, ships Auto QA, which scores three criteria by AI and leaves the rest to a person.) Ratings are honest customer feedback, the Happiness Score is easy to read, the Scout report tells you plainly where the AI fell short, and AI Assist catches typos before they reach anyone. For a small team, ratings plus a weekly read of the low-rated conversations is a fair QA program.

My take: treat Happiness ratings and the Scout report as your triage list, not your QA score. They show you where to look, but they only hear from customers who click, and "resolved" only means the customer stopped asking. Once you need to know whether answers followed your policies on every conversation, you need a reviewer that has read those policies: a person's time, an assistant over MCP once a week, or an AI teammate that does it every morning and writes back to Help Scout.

Try eesel next to Help Scout

Keep your Happiness ratings and the Scout report for what they do well. What eesel may do depends on who is asking: you choose which of its actions run on their own, which ask first and which stay off. Let it read your closed Help Scout conversations against the rest of your rubric: your refund rules, your tags and your voice, written in plain words and changed whenever your policies do. It leaves its reasons in an internal note on each conversation, tags the ones a lead should read, and answers to you in plain language when you want to change what it checks.

eesel's New Scheduled Job form, with a title field, a prompt box asking what the agent should do each time the schedule runs, a Daily schedule set to 9:00 AM and a UTC timezone
A morning QA pass is one schedule and one set of instructions.

Try eesel on your Help Scout inbox, starting with yesterday's closed conversations.

Frequently Asked Questions

Does Help Scout have AI quality assurance?
Not as a QA scorecard. Help Scout's AI features (Scout, AI Drafts, AI Summarize, AI Assist and Topics) don't score your team's conversations. What it gives a QA lead is customer Happiness ratings, the Scout report on AI conversations and read-only MCP access for your own AI tools. AI support quality assurance covers how AI QA works in general.
How does the Help Scout Happiness Score work?
Customers rate each email reply Great, Okay or Not Good, within 10 days of the reply. The Happiness Score is the share of Great ratings minus the share of Not Good ratings, much like NPS, and ratings count on the date of the reply, not the date of the rating. Help Scout Happiness report dates explains the dating rule.
How do I review Scout's AI answers in Help Scout?
Go to Reports, then Scout. The report splits Scout's conversations into Resolved, Unresolved and Human Escalation, with the customer's rating. Open a Beacon's report and sort by resolution to find the sessions where customers weren't helped, then add an Improvement where Scout got it wrong. Help Scout AI agent covers what Scout handles.
Can Claude or ChatGPT score Help Scout conversations?
They can read them. Help Scout's MCP server lets Claude, ChatGPT and Cursor list, search and summarize conversations on Standard, Plus and Pro plans, so you can ask an assistant to grade a batch against your rubric. Access is read-only, so it can't leave a note or tag on the conversations it scored. Claude for Help Scout goes deeper.
Is there a QA tool in the Help Scout app directory?
Not when I checked Help Scout's integrations page on 21 November 2026: none of the listed apps scores conversations against a QA scorecard. Support QA tools lists the main ones.
How much does it cost to run QA in Help Scout?
Help Scout charges nothing extra for ratings or the Scout report. Standard costs $25 per user a month billed annually and keeps two years of Happiness data; Plus at $45 keeps it for the life of the account and adds Topics and MCP report tools. Help Scout pricing has the full breakdown.
Can an AI agent score Help Scout conversations and leave notes?
Yes. eesel connects to Help Scout, can run on a schedule, lists the conversations closed since yesterday, reads each one with its tags and notes, scores it on criteria you write in plain words, and leaves the reasons as an internal note, with a tag on the ones a lead should read. eesel for Help Scout has the details.

Share this article