AI chatbot accuracy in financial services: why fees and rates trip bots up

Kira
Written by

Kira

Katelin Teen
Reviewed by

Katelin Teen

Last edited November 17, 2026

Expert Verified
Hand-drawn friendly robot explaining a fee card with a percent sign and a warning symbol to two thoughtful support team members, with a calculator on the desk
TL;DR

There's no single accuracy number for an AI chatbot in financial services; one figure averages very different questions. Explaining what APR means is easy. Answers that depend on today's fee schedule, a live rate, the customer's account tier or an exception are where bots go confidently wrong, usually from a stale or wrong source, not a weak model. So accuracy is decided by routing: approved docs for concepts, live systems for account facts, a person for waivers and disputes, and a replay of your past tickets before launch. eesel's helpdesk teammate works this way, tested on your own tickets first.

How accurate are AI chatbots in financial services?

The honest answer is "it depends on the question," and that's not a dodge. I build the AI agents at eesel, and the most useful thing I've learned from watching them answer real tickets is that accuracy isn't one number, it's four, one for each kind of question a customer asks.

Banks have been running chatbots for years. The CFPB's chatbots in consumer finance report found that all of the top 10 US commercial banks use them, and that over 98 million people, about 37% of the US population, used a bank's chatbot in 2022. Its headline finding is the one that matters here: chatbots "may be useful for resolving basic inquiries, but their effectiveness wanes as problems become more complex."

That matches what I see. Here's how the questions split:

A four-step staircase of finance questions, from explaining a concept to looking up a fixed fact, an account-dependent question and a judgement call, with the risk of a confident wrong answer rising at each step and a label for who should answer each
A four-step staircase of finance questions, from explaining a concept to looking up a fixed fact, an account-dependent question and a judgement call, with the risk of a confident wrong answer rising at each step and a label for who should answer each
Question typeExampleRisk of a wrong answerRight sourceWhat the bot should do
Explain a concept"What is APR?"LowApproved help articlesAnswer and link the article
Look up a fixed fact"What's the international wire fee?"Medium: stale or duplicate docsThe current fee schedule, one owner, datedAnswer only from the current version
Depends on the account"Why was I charged $35 yesterday?"High: the fee page can't knowLive transaction or account lookupLook it up, quote the record, or hand off
Judgement call"Can you waive it?"Highest: it's a decision, not a factA person with authorityCollect details and hand off

A bot can score near-perfect on the first row and still be dangerous on the third. If a vendor gives you one blended percentage, ask what share of the test set was "what is APR" questions.

eesel has spent years putting AI agents on live support queues, and the pattern holds outside finance too. On one real-traffic trial on an e-commerce inbox, eesel's team went through every draft the support agents rewrote. Most rewrites were about length and tone. About 20% needed data the bot couldn't see, like order and logistics records, and only about 5% were cases where the bot was factually wrong on its own. In other words, most "accuracy" problems were missing-data problems. In finance, the missing data is the fee schedule, the account and the transaction history.

If your team uses coding agents, the eesel CLI lets one do the setup work: upload your approved fee schedules and policy documents, then ask the agent test questions about each fee and check the answers before anything goes live. Connecting your helpdesk still needs someone with admin access to approve it.

Why do finance chatbots get fees and rates wrong?

Because a language model answers whatever it's given, fluently, whether or not the source was right. NIST's generative AI profile calls this confabulation: "confidently stated but erroneous or false content," better known as hallucinations. In a support bot, I see it come from five places.

  1. A stale document wins the search. In a typical retrieval pipeline, last year's fee schedule is still in the help center, and it matches the customer's words better than the new one. The bot quotes it with full confidence.
  2. Two tiers get blended. The premium account waives ATM fees, the basic one doesn't. Both pages mention ATM fees, and the bot merges them into one answer that is true for nobody.
  3. The bot does the math. Payoff amounts, prorated interest and currency conversions are arithmetic, and language models are not calculators.
  4. The question needs account data the bot doesn't have. "Why was I charged" can't be answered from a fee page. A bot without a transaction lookup will guess the most likely fee instead of the actual one.
  5. Retrieval comes back empty and the bot fills the gap. eesel has seen this on real rollouts: when the knowledge base had nothing relevant, an early agent answered from general knowledge instead of saying so. That's why a hard "I don't have that information" fallback is now part of how we set agents up.

The math point gets underrated. Here's a Hacker News user's attempt to get a payment figure out of ChatGPT:

Hacker News

"I asked it to calculate my total mortgage payments over 10 years for $100,000 at 2% interest. The result ($102,945.60) looked a bit suspicious to me. But it gets really interesting with 4.5% interest."

At 4.5%, the model went on to produce a negative monthly payment and then explained why that made sense. Nobody should be quoting a payoff figure to a customer from a language model's own arithmetic. The figure has to come from the loan system, and the bot's job is to repeat it.

The bigger risk is that customers believe these answers, whether they come from a bank's bot or from ChatGPT in finance. Another commenter on a personal finance forum put it this way:

Hacker News

"Our robot overlords are terrible at anything financial. I'm on a financial forum where, lately, people are constantly posting stuff that they got from The Oracle and asking what they're doing wrong because they don't understand the result, and the answer is inevitably that ChatGPT or whatever fed them plausible-looking rubbish"

"Plausible-looking" is the problem in one word. A wrong fee in a confident, well-formatted reply reads exactly like a right one. The hallucination prevention guide and the post on keeping an agent grounded go deeper on the general mechanics. In finance, all five causes above trace back to one question: where did this answer come from?

Who is liable when a chatbot gives a wrong answer?

The company that deployed it. The case everyone cites isn't a bank, but it's the clearest ruling there is.

In Moffatt v. Air Canada (2024 BCCRT 149, decided February 14, 2024), British Columbia's Civil Resolution Tribunal looked at a chatbot that told a grieving customer they could apply for a bereavement fare retroactively, within 90 days of buying the ticket. The airline's own bereavement page, which the chatbot linked to, said the opposite. Air Canada argued it couldn't be held liable for what its chatbot said. The tribunal called that "a remarkable submission," found the airline hadn't taken reasonable care to make sure the chatbot was accurate, and ordered it to pay $812.02, including $650.88 in damages. The line finance teams should remember: "It makes no difference whether the information comes from a static page or a chatbot."

Notice the mechanism. The bot didn't lack the right policy. It had a correct page right there and still produced a contradicting answer. That's failure number two from the list above, two sources disagreeing, and it's the most common way a fee answer goes wrong.

Finance regulators say the same thing in their own words:

  • CFPB (US). Chatbots "must comply with all applicable federal consumer financial laws, and entities may be liable for violating those laws when they fail to do so." The report also flags bots that fail to recognize disputes, trap people in "doom loops" and give inaccurate information, and notes that automated systems "may be less likely to waive fees."
  • FCA (UK). Its AI good and poor practice page, updated October 20, 2026, lists as good practice that firms "limited chatbots and generative AI assistants to approved source material, with checks and escalation routes." As poor practice, it lists firms that "overstated the accuracy, reliability or role of AI systems."
  • EU. Under the AI Act's transparency rules, which took effect in August 2026, people using a chatbot should be made aware they're talking to a machine.

None of this bans chatbots. It means you own every answer, so you need to know which answers your bot is allowed to give. The EU AI Act guide, the post on regulated support and the SOC 2 and GDPR checklist cover the compliance side in more depth.

Where should each answer come from?

This is the part I'd spend the most time on, because it's where accuracy gets decided. Every question your bot might answer should map to exactly one source of truth before it goes live.

A customer question splits three ways: approved docs for definitions, how-tos and policies; live systems for balances, rates and fees charged; and a person for waivers, disputes and hardship, with a dashed path from live systems to a person when a lookup fails
A customer question splits three ways: approved docs for definitions, how-tos and policies; live systems for balances, rates and fees charged; and a person for waivers, disputes and hardship, with a dashed path from live systems to a person when a lookup fails

Approved docs carry concepts, how-tos and published policy. Each document needs an owner, a version and an effective date, and the old version has to leave the bot's knowledge the day the new one starts. When two versions of a fee page are both searchable, the bot has no way to know which one is current. A regular knowledge base audit and tooling for detecting outdated content do more for accuracy than any model upgrade. The guide to an AI knowledge base for fintech covers how to structure it.

Live systems carry anything that is specific to one customer or changes daily: balances, the fee actually charged, the current rate, a payoff quote, a payment's status. The bot reaches these through actions, meaning API lookups it is allowed to make. eesel's Stripe integration, for example, lets the agent read invoices, payments and subscriptions to answer a billing question from the record itself. The post on AI for Stripe billing support shows what that looks like. If the lookup fails, the bot says so and hands off. It never falls back to the general fee page.

A person carries decisions: waivers, refunds outside policy, hardship arrangements, disputes and complaints. More on those in the next section.

This is how smava, a Berlin loan comparison platform, runs eesel on Zendesk. Around 27,000 tickets a month come in across smava and Finanzcheck, almost all in German. smava's own framing is blunt: this is regulated finance, so wording matters. eesel drafts from about 820 macros and help center articles, kept apart by brand, and the drafting agent uses the matching macro body word for word. Customer-ready replies only go to tickets where no credit advisor is assigned, about 1,100 of the 27,000 in a month. The agents' instructions forbid promising that the team will check, forward or arrange anything, and the drafting agent may not claim to have searched unless it did. The bot's freedom is narrow on purpose, and that narrowness is the accuracy strategy.

What should a finance chatbot never answer on its own?

Anything that is a decision rather than a fact, and anything where the bot can't point to a single current source. My default list for a finance queue:

  • Fee waivers and refunds outside written policy. The bot can explain the policy. It shouldn't decide the exception.
  • Disputes and unrecognized transactions. The CFPB specifically flags bots that fail to recognize a dispute or just repeat back the information the customer is disputing. Recognizing a dispute and opening the case properly is the bot's job. Resolving it isn't.
  • Hardship, collections and vulnerability. The FCA's guidance warns against assuming an LLM can reliably spot vulnerability without testing.
  • Eligibility and credit decisions. "Will I be approved?" isn't something a support bot should answer.
  • Any answer where two sources disagree. If the bot finds two fee figures, it should hand off, not pick one.
  • Calculations. Payoffs, interest and conversions come from the system of record or not at all.

Pick a question below to see where it sits and what the bot should do with it.

Can my bot answer this on its own?

Explain a concept

Source: an approved help article.

Bot does: answers directly and links the article. This is the easy row, and where most of a bot's headline accuracy comes from.

Look up a fixed fact

Source: the current fee schedule, with an owner and an effective date.

Bot does: answers from the current version only. If it finds two different figures, it hands off instead of picking one. Stale fee pages cause most errors here.

Depends on the account

Source: a live transaction lookup.

Bot does: looks up the actual charge and quotes the record. It never guesses from the fee page. If the lookup fails, it says so and hands off.

Judgement call

Source: a person with authority to decide.

Bot does: explains the written policy, collects the account and reason, and hands off. It never promises a waiver.

Account plus calculation

Source: a payoff quote from the loan system.

Bot does: requests the quote and repeats the number. It never does the interest arithmetic itself.

Dispute

Source: your dispute process and a person.

Bot does: recognizes this as a dispute, opens the case or routes it with the details, and tells the customer what happens next. Opening a case should be an approval-required action at first.

Writing these limits down is the easy part. The bot has to follow them, which means two settings: what it may say, and what it may do.

What it may say lives in its instructions. In eesel, those are plain-language rules, and the most important one for finance is the refusal rule: answer only from retrieved documentation, and if nothing relevant comes back, say you don't have that information.

eesel's Instructions editor showing a workflow rule that the agent must only answer from retrieved documentation and should tell the user it doesn't have information when no relevant document is found, with the chat panel used to update the rule
eesel's Instructions editor showing a workflow rule that the agent must only answer from retrieved documentation and should tell the user it doesn't have information when no relevant document is found, with the chat panel used to update the rule

What it may do lives in action permissions. In eesel, every action is set to On, Ask first or Off, per agent, per the actions and approvals docs. For a finance queue I'd start with lookups on, anything that moves money or opens a case on ask first, and refunds off until you've seen enough runs. The refund guardrails post explains why limits like these belong in settings rather than the prompt.

The handoff itself has to work, too. Enterprise support buyers, including one in medical financing, have told eesel's sales team the same thing: they won't accept an AI that replies to everything, even with "I don't know," and want it to auto-reply only when it can handle the ticket and route the rest. The customer side of a broken handoff looks like this:

Hacker News

"I told it to go get me a human and it said it would find a human... then put me through a questionnaire before telling me to call the main support line. sigh"

That's the CFPB's doom loop at small scale. A handoff should pass the conversation and the collected details to a person in the same channel. The guides to chatbot escalation, human handoff and confidence thresholds cover the mechanics.

For dispute-heavy queues, see AI for billing disputes and chargeback inquiries. Payment disputes follow the same pattern: the bot gathers the facts, a person decides.

How do you test accuracy before launch?

Replay your own past tickets through the bot and compare its answers with what your team actually sent. A generic benchmark tells you how a model does on someone else's questions. Your resolved tickets tell you how this bot does on your fees, your account tiers and your customers' wording.

A loop of four steps: past tickets are replayed through the bot, compared with what your team sent, and the source or rule is fixed before repeating, with an exit arrow to going live one topic at a time
A loop of four steps: past tickets are replayed through the bot, compared with what your team sent, and the source or rule is fixed before repeating, with an exit arrow to going live one topic at a time

Here's the routine I'd run:

  1. Pull a few hundred resolved tickets, weighted toward the risky rows: fee questions, charges, rate questions, disputes. Don't let password resets dominate the sample.
  2. Replay them with nothing sent. eesel's simulation runs the full agent on real or generated tickets in a sandbox and scores each answer against what your team sent.
  3. Score by topic, not as one average. A 90% overall score can hide a 60% score on fee questions. The topic split is the number that matters.
  4. Label each failure by cause. Stale document, blended sources, missing lookup, missing refusal rule. Each cause has a different fix, and none of them is "try a different model."
  5. Fix and replay the same tickets. Retest on the tickets that failed, so you know the fix worked on the exact wording that broke it.
  6. Go live one topic at a time, starting with the explain-a-concept row, and keep the riskier rows as drafts for a person to approve.

The FCA describes the same idea from the regulator's side. Its good practice list includes testing consumer outcomes across different customer groups and scenarios, keeping records of prompts, outputs and escalation decisions, and comparing a model's performance against the human-led process. Smaller firms, it notes, can use sample outputs and carefully filtered test data rather than testing at full scale.

Testing doesn't stop at launch either. Qontro, a cloud ERP company in New Zealand, runs eesel as a co-pilot on Freshdesk: every ticket gets a draft, billing questions go straight to a person, and when a ticket doesn't have enough in it to answer confidently, eesel says so instead of guessing. The team compares each draft with the reply a person actually sent, and where a draft was wrong, it usually points to a missing help article or a rule to tighten. About one in ten drafts now goes out with little or no editing. That's a slow, honest number, and the gap tells the team exactly where to work.

For the ongoing part, the guides to reviewing AI conversations, AI support QA and adversarial testing cover what to read each week.

Also watch your resolution rate with some suspicion: the post on deflection vs resolution explains why a closed conversation isn't proof of a correct answer.

Try eesel for finance support

If you're putting AI in front of customers who ask about fees, rates and charges, eesel is an AI helpdesk teammate that joins your existing queue in Zendesk, Freshdesk and other helpdesks. It learns from your help center, macros and past tickets, answers only from what it retrieved, looks up live data through integrations, and hands waivers and disputes to your team with the details collected.

eesel working inside Zendesk, drafting a reply on a support ticket from the team's own knowledge

The difference for a finance team is the testing. Before a single customer sees it, eesel replays your resolved tickets and shows, topic by topic, where it would have quoted a wrong fee or missed a handoff. smava runs it across 27,000 loan tickets a month with every rule written in the agents' instructions, so changing what eesel may say is a text edit rather than a project. Plans start at $299 a month for 500 credits after a 7-day free trial, per eesel pricing.

For more on the finance use case, see the guides to AI for fintech support, banking chatbots and conversational AI for banking.

Try eesel on your own past tickets and see which fee questions it gets right before your customers find out.

Frequently Asked Questions

How accurate are AI chatbots in financial services?
There is no single accuracy figure, because it depends on the question. Bots answer concept questions like "what is APR" well, and get fees, rates and account-specific questions wrong when they read a stale document or guess instead of looking up live data. Measure accuracy by question type, as in this guide to AI support agent testing.
Why do banking chatbots give wrong answers about fees?
Usually the bot retrieved an outdated fee schedule, blended two account tiers, or did the arithmetic itself. The fix is to keep one current fee source with an effective date and send account-specific questions to a live lookup. The post on hallucination prevention covers the gates in more detail.
Is a company liable for its chatbot's wrong answers?
Tribunals have said yes. In Moffatt v. Air Canada (2024), the tribunal held the airline responsible for its chatbot's wrong refund advice and ordered it to pay about $812. The CFPB says chatbots must follow federal consumer financial law. See AI for regulated support for the wider rules.
What questions should a finance chatbot hand to a human?
Fee waivers, disputes and unrecognized transactions, hardship, complaints, eligibility decisions and anything where two sources disagree. The bot can collect the details first so the person starts with context. Good handoff practices make that transfer clean.
How do you test AI chatbot accuracy before launch?
Replay a few hundred of your resolved tickets through the bot, compare each answer with what your team sent, and score the results by topic, not as one average. eesel's simulation does this on your own tickets with nothing sent to customers.
Can AI chatbots calculate interest or payoff amounts accurately?
Do not let the language model do the math. Have the bot call the loan or billing system for the payoff quote and repeat that number. Language models can produce plausible but wrong arithmetic, which is why billing questions need a system of record behind them.
Do AI chatbots in fintech need to say they are AI?
In the EU, yes: the AI Act's transparency rules took effect in August 2026 and require that people know they are talking to a machine. The UK FCA also lists explaining what an AI tool can and cannot do as good practice. The post on the EU AI Act has the details.
What is the best AI chatbot for fintech customer support?
Pick one that answers only from approved sources, looks up account data through real integrations, lets you set which actions need approval, and can be tested on your past tickets before launch. The roundup on AI chatbots for fintech compares the options, and eesel covers all four.

Share this article