Every AI helpdesk vendor claims 90%+ accuracy. Almost none of them show you the math. Before you hand any ticket queue to an AI classifier, you need to know exactly what accurate means — and how to verify it yourself.
This is not a buyer's guide. It's a methodology piece. Whether you're evaluating Clawbak, a PSA-bundled AI, or building something in-house, the measurement framework below applies. Get comfortable with these numbers before any vendor demo, and you'll ask questions most MSPs never think to ask.
What "Accuracy" Actually Means in a Ticket Classifier
When a vendor says their classifier is "92% accurate," they're usually reporting a single top-line number on a curated test set. That number collapses at least three distinct things into one, and each of them can fail independently.
Here are the three dimensions you need to decompose before trusting any accuracy claim:
The Three Numbers That Actually Matter
- Classification confidence. When the model assigns a ticket to a category — say, "password reset" or "printer offline" — how confident is it? A well-calibrated classifier reports a probability score, not just a label. A score of 0.94 on a password-reset ticket means the model has seen enough examples that it's highly certain. A score of 0.61 means it's guessing. You want to see confidence distributions across your real ticket volume, not averages across a vendor's benchmark set. Ask specifically: what percentage of tickets in my queue score above your confidence floor, and what does that floor actually gate?
- Grounding rate. Confidence and grounding are not the same thing, and conflating them is the most dangerous mistake in MSP AI evaluation. A classifier can be highly confident it knows what a ticket is asking about — and still produce a reply that isn't grounded in any verified procedure in your knowledge base. Grounding rate measures how often a proposed reply can be traced back to a KB article, a documented runbook, or an approved response template. High confidence with low grounding is a red flag: it means the model is extrapolating, not retrieving. That extrapolation might look correct 80% of the time and silently wrong 20% of the time.
- False-positive rate on auto-resolved tickets. The accuracy metric that costs you the most is not the one where the classifier gets the label wrong — it's the one where it gets the label right, generates a confident reply, and sends something to an end-user that's incorrect or inappropriate for that specific environment. False positives on auto-resolved tickets are the ones that generate escalations, damage trust, and get your team blamed for the bot. You need to track this separately from overall classification accuracy.
"Confidence without grounding is autocomplete. It sounds right until it isn't — and by then your tech is already on the phone explaining why the AI told a client to restart a server that's in production lock."
How to Run Your Own Accuracy Audit in a Week
You don't need a data science background to do this. You need a sample of historical tickets, a spreadsheet, and about two hours. Here's the process I'd run before evaluating any AI helpdesk system.
Step 1 — Pull a representative sample. Export a representative batch of tickets from your PSA covering the last 90 days. Aim for at least 200 tickets if you can. Don't cherry-pick resolved or easy tickets — include your messiest subjects, your longest threads, and anything that required escalation. The classifier will perform well on clean tickets; what matters is how it handles the edge cases that make up a real queue.
Step 2 — Label a ground-truth set. Have one or two techs manually assign each ticket a category using whatever taxonomy you'd want the AI to use. Don't use more than 15–20 categories at this stage; too many labels and your sample size per category gets too small to be statistically meaningful. This is your ground truth.
Step 3 — Run the classifier and capture confidence scores. Feed those same tickets into the AI classifier and record: (a) the assigned label, (b) the confidence score, and (c) whether a KB article was cited or used to generate the draft reply. Do not just look at whether the label matches — you want the full output.
Step 4 — Build a confusion matrix, then segment by confidence band. A confusion matrix shows you where misclassifications cluster. Most vendors won't show you this unprompted — ask for it. Then segment the results into confidence bands: above 0.90, 0.70–0.90, and below 0.70. Calculate accuracy separately in each band. A well-calibrated classifier should have much higher accuracy in the top band. If accuracy in the 0.90+ band is only marginally better than the 0.70–0.90 band, the confidence scores aren't calibrated — they're noisy.
Step 5 — Calculate grounding rate independently. For every ticket where the classifier generated a draft reply, count how many were anchored to a real KB article versus generated from model weights alone. If your AI doesn't surface a KB citation with drafts, that's not just a UI omission — it means you have no mechanism to verify grounding at all. You're flying blind on a metric that matters more than confidence.
Step 6 — Track the false-positive rate on gate-pass candidates. Not every ticket will be a candidate for auto-resolution. But for the subset that would qualify — the common, repeatable, low-risk ticket types — calculate what percentage of auto-send-eligible replies your techs would have flagged as wrong or inappropriate. This is your false-positive rate on the population that actually matters for automation risk. Keep it as close to zero as you can before you consider live auto-send on any ticket type.
Run this audit before you sign anything, and repeat it quarterly after deployment. Classifier performance drifts as your client environment changes, as ticket language evolves, and as your KB gets stale. A one-time benchmark means nothing six months later.
What Good Numbers Look Like — and Where the Industry Falls Short
Let's be concrete about benchmarks, because vendors rarely are.
A classification confidence score in isolation tells you very little. What matters is the calibration: when the model says 0.90, it should be correct about 90% of the time on your ticket population — not on a benchmark dataset from a different industry or a different MSP's KB. Ask every vendor: "Is your confidence score calibrated to my data, or is it a raw logit from a model trained on generic IT support data?" The answer will tell you a lot.
For grounding rate, the honest floor for production use is high. If your auto-send-eligible replies aren't almost entirely grounded in your KB, you should not be running live auto-send on that ticket type. Grounding is what gives a tech (or an audit trail) a path back to "why did the bot say that." Without it, you can't improve, you can't explain, and you can't defend.
A real-world example: DTC Networks, an MSP using Clawbak, went from 82% classifier confidence to 97% in 24 hours — not by retraining the model, but by filling the KB gaps that were causing low-confidence classifications. That number matters because it shows what grounding work actually does to confidence scores. The two metrics are linked: when the classifier has a well-structured KB article to retrieve from, its confidence on that ticket type goes up. When it's pattern-matching on thin air, confidence is volatile and grounding is absent.
The industry's disclosure problem is structural. Most PSA-bundled AI tools — whether that's Atera Robin, NinjaOne Ninja AI, SuperOps MonicaAI, or ConnectWise Sidekick — report accuracy in marketing materials without publishing methodology, confidence calibration data, or grounding rate definitions. That's not necessarily because the numbers are bad; it's because "accuracy" is a more defensible claim than "grounding rate," and most MSP buyers don't ask the harder question yet.
Vendor-agnostic tools have the same obligation to be transparent. Claiming vendor-agnosticism doesn't automatically make your accuracy claims more credible — the methodology still has to be published and reproducible.
The framework here isn't Clawbak-specific. It applies to any classifier you're evaluating. Run the audit. Ask for the confusion matrix. Demand grounding rate as a separate metric from confidence. If a vendor can't show you those numbers on your own ticket sample, that's the answer.
Getting Started: Build Your Measurement Baseline First
The biggest mistake MSPs make when evaluating AI helpdesk tools is starting with a vendor demo on curated tickets rather than running their own historical data first. Establish your baseline before any vendor touches your queue.
Pull that 200-ticket sample. Label it. Calculate your current manual classification consistency across your team — you'll probably find it's lower than you expect, which is itself useful data. Then when you run a classifier against it, you have a real comparison point instead of a vendor-supplied benchmark.
If you want to see how Clawbak's confidence scoring and KB-grounding checks behave against your actual ticket history — not a demo dataset — we'll run that audit with you. No slide deck, no curated examples. Your tickets, your categories, your KB.