
There's no single accuracy number for an AI chatbot in financial services; one figure averages very different questions. Explaining what APR means is easy. Answers that depend on today's fee schedule, a live rate, the customer's account tier or an exception are where bots go confidently wrong, usually from a stale or wrong source, not a weak model. So accuracy is decided by routing: approved docs for concepts, live systems for account facts, a person for waivers and disputes, and a replay of your past tickets before launch. eesel's helpdesk teammate works this way, tested on your own tickets first.
How accurate are AI chatbots in financial services?
The honest answer is "it depends on the question," and that's not a dodge. I build the AI agents at eesel, and the most useful thing I've learned from watching them answer real tickets is that accuracy isn't one number, it's four, one for each kind of question a customer asks.
Banks have been running chatbots for years. The CFPB's chatbots in consumer finance report found that all of the top 10 US commercial banks use them, and that over 98 million people, about 37% of the US population, used a bank's chatbot in 2022. Its headline finding is the one that matters here: chatbots "may be useful for resolving basic inquiries, but their effectiveness wanes as problems become more complex."
That matches what I see. Here's how the questions split:

| Question type | Example | Risk of a wrong answer | Right source | What the bot should do |
|---|---|---|---|---|
| Explain a concept | "What is APR?" | Low | Approved help articles | Answer and link the article |
| Look up a fixed fact | "What's the international wire fee?" | Medium: stale or duplicate docs | The current fee schedule, one owner, dated | Answer only from the current version |
| Depends on the account | "Why was I charged $35 yesterday?" | High: the fee page can't know | Live transaction or account lookup | Look it up, quote the record, or hand off |
| Judgement call | "Can you waive it?" | Highest: it's a decision, not a fact | A person with authority | Collect details and hand off |
A bot can score near-perfect on the first row and still be dangerous on the third. If a vendor gives you one blended percentage, ask what share of the test set was "what is APR" questions.
eesel has spent years putting AI agents on live support queues, and the pattern holds outside finance too. On one real-traffic trial on an e-commerce inbox, eesel's team went through every draft the support agents rewrote. Most rewrites were about length and tone. About 20% needed data the bot couldn't see, like order and logistics records, and only about 5% were cases where the bot was factually wrong on its own. In other words, most "accuracy" problems were missing-data problems. In finance, the missing data is the fee schedule, the account and the transaction history.
If your team uses coding agents, the eesel CLI lets one do the setup work: upload your approved fee schedules and policy documents, then ask the agent test questions about each fee and check the answers before anything goes live. Connecting your helpdesk still needs someone with admin access to approve it.
Why do finance chatbots get fees and rates wrong?
Because a language model answers whatever it's given, fluently, whether or not the source was right. NIST's generative AI profile calls this confabulation: "confidently stated but erroneous or false content," better known as hallucinations. In a support bot, I see it come from five places.
- A stale document wins the search. In a typical retrieval pipeline, last year's fee schedule is still in the help center, and it matches the customer's words better than the new one. The bot quotes it with full confidence.
- Two tiers get blended. The premium account waives ATM fees, the basic one doesn't. Both pages mention ATM fees, and the bot merges them into one answer that is true for nobody.
- The bot does the math. Payoff amounts, prorated interest and currency conversions are arithmetic, and language models are not calculators.
- The question needs account data the bot doesn't have. "Why was I charged" can't be answered from a fee page. A bot without a transaction lookup will guess the most likely fee instead of the actual one.
- Retrieval comes back empty and the bot fills the gap. eesel has seen this on real rollouts: when the knowledge base had nothing relevant, an early agent answered from general knowledge instead of saying so. That's why a hard "I don't have that information" fallback is now part of how we set agents up.
The math point gets underrated. Here's a Hacker News user's attempt to get a payment figure out of ChatGPT:
"I asked it to calculate my total mortgage payments over 10 years for $100,000 at 2% interest. The result ($102,945.60) looked a bit suspicious to me. But it gets really interesting with 4.5% interest."
At 4.5%, the model went on to produce a negative monthly payment and then explained why that made sense. Nobody should be quoting a payoff figure to a customer from a language model's own arithmetic. The figure has to come from the loan system, and the bot's job is to repeat it.
The bigger risk is that customers believe these answers, whether they come from a bank's bot or from ChatGPT in finance. Another commenter on a personal finance forum put it this way:
"Our robot overlords are terrible at anything financial. I'm on a financial forum where, lately, people are constantly posting stuff that they got from The Oracle and asking what they're doing wrong because they don't understand the result, and the answer is inevitably that ChatGPT or whatever fed them plausible-looking rubbish"
"Plausible-looking" is the problem in one word. A wrong fee in a confident, well-formatted reply reads exactly like a right one. The hallucination prevention guide and the post on keeping an agent grounded go deeper on the general mechanics. In finance, all five causes above trace back to one question: where did this answer come from?
Who is liable when a chatbot gives a wrong answer?
The company that deployed it. The case everyone cites isn't a bank, but it's the clearest ruling there is.
In Moffatt v. Air Canada (2024 BCCRT 149, decided February 14, 2024), British Columbia's Civil Resolution Tribunal looked at a chatbot that told a grieving customer they could apply for a bereavement fare retroactively, within 90 days of buying the ticket. The airline's own bereavement page, which the chatbot linked to, said the opposite. Air Canada argued it couldn't be held liable for what its chatbot said. The tribunal called that "a remarkable submission," found the airline hadn't taken reasonable care to make sure the chatbot was accurate, and ordered it to pay $812.02, including $650.88 in damages. The line finance teams should remember: "It makes no difference whether the information comes from a static page or a chatbot."
Notice the mechanism. The bot didn't lack the right policy. It had a correct page right there and still produced a contradicting answer. That's failure number two from the list above, two sources disagreeing, and it's the most common way a fee answer goes wrong.
Finance regulators say the same thing in their own words:
- CFPB (US). Chatbots "must comply with all applicable federal consumer financial laws, and entities may be liable for violating those laws when they fail to do so." The report also flags bots that fail to recognize disputes, trap people in "doom loops" and give inaccurate information, and notes that automated systems "may be less likely to waive fees."
- FCA (UK). Its AI good and poor practice page, updated October 20, 2026, lists as good practice that firms "limited chatbots and generative AI assistants to approved source material, with checks and escalation routes." As poor practice, it lists firms that "overstated the accuracy, reliability or role of AI systems."
- EU. Under the AI Act's transparency rules, which took effect in August 2026, people using a chatbot should be made aware they're talking to a machine.
None of this bans chatbots. It means you own every answer, so you need to know which answers your bot is allowed to give. The EU AI Act guide, the post on regulated support and the SOC 2 and GDPR checklist cover the compliance side in more depth.
Where should each answer come from?
This is the part I'd spend the most time on, because it's where accuracy gets decided. Every question your bot might answer should map to exactly one source of truth before it goes live.

Approved docs carry concepts, how-tos and published policy. Each document needs an owner, a version and an effective date, and the old version has to leave the bot's knowledge the day the new one starts. When two versions of a fee page are both searchable, the bot has no way to know which one is current. A regular knowledge base audit and tooling for detecting outdated content do more for accuracy than any model upgrade. The guide to an AI knowledge base for fintech covers how to structure it.
Live systems carry anything that is specific to one customer or changes daily: balances, the fee actually charged, the current rate, a payoff quote, a payment's status. The bot reaches these through actions, meaning API lookups it is allowed to make. eesel's Stripe integration, for example, lets the agent read invoices, payments and subscriptions to answer a billing question from the record itself. The post on AI for Stripe billing support shows what that looks like. If the lookup fails, the bot says so and hands off. It never falls back to the general fee page.
A person carries decisions: waivers, refunds outside policy, hardship arrangements, disputes and complaints. More on those in the next section.
This is how smava, a Berlin loan comparison platform, runs eesel on Zendesk. Around 27,000 tickets a month come in across smava and Finanzcheck, almost all in German. smava's own framing is blunt: this is regulated finance, so wording matters. eesel drafts from about 820 macros and help center articles, kept apart by brand, and the drafting agent uses the matching macro body word for word. Customer-ready replies only go to tickets where no credit advisor is assigned, about 1,100 of the 27,000 in a month. The agents' instructions forbid promising that the team will check, forward or arrange anything, and the drafting agent may not claim to have searched unless it did. The bot's freedom is narrow on purpose, and that narrowness is the accuracy strategy.
What should a finance chatbot never answer on its own?
Anything that is a decision rather than a fact, and anything where the bot can't point to a single current source. My default list for a finance queue:
- Fee waivers and refunds outside written policy. The bot can explain the policy. It shouldn't decide the exception.
- Disputes and unrecognized transactions. The CFPB specifically flags bots that fail to recognize a dispute or just repeat back the information the customer is disputing. Recognizing a dispute and opening the case properly is the bot's job. Resolving it isn't.
- Hardship, collections and vulnerability. The FCA's guidance warns against assuming an LLM can reliably spot vulnerability without testing.
- Eligibility and credit decisions. "Will I be approved?" isn't something a support bot should answer.
- Any answer where two sources disagree. If the bot finds two fee figures, it should hand off, not pick one.
- Calculations. Payoffs, interest and conversions come from the system of record or not at all.
Pick a question below to see where it sits and what the bot should do with it.
Can my bot answer this on its own?
Source: an approved help article.
Bot does: answers directly and links the article. This is the easy row, and where most of a bot's headline accuracy comes from.
Source: the current fee schedule, with an owner and an effective date.
Bot does: answers from the current version only. If it finds two different figures, it hands off instead of picking one. Stale fee pages cause most errors here.
Source: a live transaction lookup.
Bot does: looks up the actual charge and quotes the record. It never guesses from the fee page. If the lookup fails, it says so and hands off.
Source: a person with authority to decide.
Bot does: explains the written policy, collects the account and reason, and hands off. It never promises a waiver.
Source: a payoff quote from the loan system.
Bot does: requests the quote and repeats the number. It never does the interest arithmetic itself.
Source: your dispute process and a person.
Bot does: recognizes this as a dispute, opens the case or routes it with the details, and tells the customer what happens next. Opening a case should be an approval-required action at first.
Writing these limits down is the easy part. The bot has to follow them, which means two settings: what it may say, and what it may do.
What it may say lives in its instructions. In eesel, those are plain-language rules, and the most important one for finance is the refusal rule: answer only from retrieved documentation, and if nothing relevant comes back, say you don't have that information.

What it may do lives in action permissions. In eesel, every action is set to On, Ask first or Off, per agent, per the actions and approvals docs. For a finance queue I'd start with lookups on, anything that moves money or opens a case on ask first, and refunds off until you've seen enough runs. The refund guardrails post explains why limits like these belong in settings rather than the prompt.
The handoff itself has to work, too. Enterprise support buyers, including one in medical financing, have told eesel's sales team the same thing: they won't accept an AI that replies to everything, even with "I don't know," and want it to auto-reply only when it can handle the ticket and route the rest. The customer side of a broken handoff looks like this:
"I told it to go get me a human and it said it would find a human... then put me through a questionnaire before telling me to call the main support line. sigh"
That's the CFPB's doom loop at small scale. A handoff should pass the conversation and the collected details to a person in the same channel. The guides to chatbot escalation, human handoff and confidence thresholds cover the mechanics.
For dispute-heavy queues, see AI for billing disputes and chargeback inquiries. Payment disputes follow the same pattern: the bot gathers the facts, a person decides.
How do you test accuracy before launch?
Replay your own past tickets through the bot and compare its answers with what your team actually sent. A generic benchmark tells you how a model does on someone else's questions. Your resolved tickets tell you how this bot does on your fees, your account tiers and your customers' wording.

Here's the routine I'd run:
- Pull a few hundred resolved tickets, weighted toward the risky rows: fee questions, charges, rate questions, disputes. Don't let password resets dominate the sample.
- Replay them with nothing sent. eesel's simulation runs the full agent on real or generated tickets in a sandbox and scores each answer against what your team sent.
- Score by topic, not as one average. A
90%overall score can hide a60%score on fee questions. The topic split is the number that matters. - Label each failure by cause. Stale document, blended sources, missing lookup, missing refusal rule. Each cause has a different fix, and none of them is "try a different model."
- Fix and replay the same tickets. Retest on the tickets that failed, so you know the fix worked on the exact wording that broke it.
- Go live one topic at a time, starting with the explain-a-concept row, and keep the riskier rows as drafts for a person to approve.
The FCA describes the same idea from the regulator's side. Its good practice list includes testing consumer outcomes across different customer groups and scenarios, keeping records of prompts, outputs and escalation decisions, and comparing a model's performance against the human-led process. Smaller firms, it notes, can use sample outputs and carefully filtered test data rather than testing at full scale.
Testing doesn't stop at launch either. Qontro, a cloud ERP company in New Zealand, runs eesel as a co-pilot on Freshdesk: every ticket gets a draft, billing questions go straight to a person, and when a ticket doesn't have enough in it to answer confidently, eesel says so instead of guessing. The team compares each draft with the reply a person actually sent, and where a draft was wrong, it usually points to a missing help article or a rule to tighten. About one in ten drafts now goes out with little or no editing. That's a slow, honest number, and the gap tells the team exactly where to work.
For the ongoing part, the guides to reviewing AI conversations, AI support QA and adversarial testing cover what to read each week.
Also watch your resolution rate with some suspicion: the post on deflection vs resolution explains why a closed conversation isn't proof of a correct answer.
Try eesel for finance support
If you're putting AI in front of customers who ask about fees, rates and charges, eesel is an AI helpdesk teammate that joins your existing queue in Zendesk, Freshdesk and other helpdesks. It learns from your help center, macros and past tickets, answers only from what it retrieved, looks up live data through integrations, and hands waivers and disputes to your team with the details collected.
The difference for a finance team is the testing. Before a single customer sees it, eesel replays your resolved tickets and shows, topic by topic, where it would have quoted a wrong fee or missed a handoff. smava runs it across 27,000 loan tickets a month with every rule written in the agents' instructions, so changing what eesel may say is a text edit rather than a project. Plans start at $299 a month for 500 credits after a 7-day free trial, per eesel pricing.
For more on the finance use case, see the guides to AI for fintech support, banking chatbots and conversational AI for banking.
Try eesel on your own past tickets and see which fee questions it gets right before your customers find out.