Bookkeeping is the first real step in tax preparation, and it turns on one question asked thousands of times a day: which account does this transaction belong to? This is how we went from answering three quarters of them automatically to nearly all of them - what each attempt measured, where it broke, and what the last stretch still needs.
97.4% of bookkeeping now runs without anyone categorizing it, up from 75.3% when rules were all we had.
Rules got us three quarters of the way, and then stopped
Most bank transactions do not need judgement. A recurring vendor, a bank fee with a fixed shape, a transfer between two accounts we already know about: these are settled by deterministic rules and by merchant knowledge shared across the companies we keep books for.
Across 3.1 million categorized bank transactions in our ledger, 75.3% are settled deterministically, without an AI of any kind touching them. That is where the volume lives, and it is the cheapest, fastest and most auditable part of the pipeline.
The remaining quarter is the hard quarter. Every transaction there needs a decision that rules could not express, and there are enough of them that they set the whole product's quality. For those, we did what most teams did in 2025: we sent them to a language model.
The LLMs were worse than a lookup table
Every transaction in that residual went to both OpenAI and Claude. What went with it was deliberately thin: the transaction descriptor, the amount and the company's chart of account names - the list of buckets its books are filed into. Not the client's identity, not their balances, not their bank details. We compared the two answers and showed the result to a bookkeeper.
Measured against the account the bookkeeper actually kept, that pipeline was correct 57.8% of the time.
We assumed this was a tuning problem. It was not. We tried different models, rewrote the system prompts, fed in more context about the company and its chart of accounts. The numbers barely moved.
| Approach | Correct |
|---|---|
| Per-merchant lookup table (most frequent past account) | 63.6% |
| OpenAI + Claude, cross-checked | 57.8% |
| Tie-break where the two models disagreed | 34.1% |
The most damning benchmark was the simplest one. File each merchant to whatever account this company used for it most often in the past, and you beat two frontier models by six points. On this task, history beats reasoning.
The reason is structural rather than a matter of model capability. The right answer here is a convention, not a fact. When two of our client companies see the same merchant, they file it under different accounts about 55% of the time, and both are correct for their own books. That information is not in any model's weights and does not fit in a prompt. It lives in thousands of rows of one company's posting history.
And there was no learning loop: a correction made in the morning changed nothing by the afternoon.
Our own model scored well and stayed silent
So we trained a ranker on our own ledger. For each transaction it scores every eligible account in that company's chart of accounts and picks the best. Every feature is a count, ratio or share - never a raw identifier - so one model serves every company, including one that onboarded yesterday.
Offline it looked excellent. In production it was quiet. The model resolved only 20% of what reached it, and the rest kept falling through to the LLM path. Overall quality improved, but not nearly by what the offline numbers promised.
A model that is right and unsure is indistinguishable, in production, from a model that is wrong.
Which left the question the rest of this post answers: on the transactions it had every reason to be sure about, why was the model still declining to answer four in ten?
It was never the model. It was the merchant names.
Every feature that carries the model is a count of what this company did with this merchant. How often has this company posted this merchant to this account? What share of them landed where? Which account does it use for money leaving this particular bank account?
The company half is the entire reason a trained model beats a general-purpose one. It is the convention no amount of world knowledge supplies, the thing that makes the same merchant an operating expense at one client and cost of revenue at another. But the merchant is the key those counts are filed under. Get the merchant wrong and the company's own history cannot be found at all: every count reads as zero, and the model is left guessing about a business it actually knows well.
So we read the raw bank data, field by field, for 77 transactions from one client's Chase account. Seventeen told us who the money went to. For the other sixty, our own code had written down a merchant name anyway.
ONLINE INTERNATIONAL WIRE TRANSFER A/C: MERIDIAN COMMERCE BANK LTD SINGAPORE 048624 SG REF: INVOICE PAYMENT TRN: 4820175596ES 09/01
The whole string was stored as the merchant. It contains a trace number, so it is unique to a single transaction and can never match anything again.
ORIG CO NAME:Harborline Pay ORIG ID: 418602957 DESC DATE:260901 CO ENTRY DESCR:PROPRTYPAY SEC:PPD TRACE#:071450238816402 EED:260902
And on ACH rows the real counterparty was sitting in a labelled field we were not reading.
These two specimens are reconstructions. The field structure, the labels and the shape of every value are exactly as the banks send them; the names and reference numbers have been replaced, because the originals are client transactions.
Different banks structure descriptions completely differently. Our parser had been written years earlier against a feed that delimits its descriptions into parts, where it works correctly. Applied to a bank that writes one long run of prose, the same logic finds a single "part" and returns the entire string. It never errors. It returns something plausible, and everything downstream believes it.
Across those sixty transactions our cleanup produced sixty distinct merchants, every one seen exactly once. Each was then written into our merchant dictionary as a new business, and some had web enrichment run against them. We were paying to research trace numbers.
What the descriptors actually contained
| What the transaction carried | Count |
|---|---|
| A usable merchant name | 17 |
| No merchant name, but a named counterparty we discarded | 9 |
| An ACH descriptor with the originating company inside it | 13 |
| An internal transfer, all twenty between the same two accounts | 20 |
| Nothing identifying at all, wires and bank fees | 18 |
Fixing the parser to read each of those shapes, and to record no merchant rather than invent one when nothing identifies the payee, collapsed those 77 transactions from 61 distinct merchant keys to 6. Merchant names seen exactly once went from 60 to zero. Run against a Mercury account, where the original logic was already correct, all 100 transactions came out identical.
With honest merchant keys, the picture inside the model resolved. Sorted by what evidence each transaction actually had, the accuracy gap is not subtle.
| Evidence | Share | Correct |
|---|---|---|
| Merchant with posting history | 69.2% | 98.3% |
| No merchant name at all | 17.7% | 91.8% |
| Merchant never seen before | 13.1% | 80.2% |
The middle row is the finding. Those transactions were answered correctly nine times in ten, yet with the merchant key broken they were the ones the model most often declined to answer. Contractor income, transfers in and bank fees inside that group were answered correctly every single time. An unfamiliar merchant, meanwhile, is harder than no merchant at all.
Where a hundred transactions end up
For every transaction the model returns an answer and a score for it. We choose where to draw the line on that score. Today it sits at 99.92%. Above the line the model's answer stands on its own. Below it, the same answer goes to a bookkeeper as a one-click suggestion to accept or correct, so a weak answer there costs a click rather than a correction.
That line is policy, not a property of the model. We can move it, tighten it for a book that needs tightening, or set it per client. The trade between automation and review is a dial, and we hold it.
| Where it is settled | Of all | Of the residual | A person decides |
|---|---|---|---|
| Deterministic rules and shared merchant knowledge | 75.3% | - | No |
| Model answers, above the 99.92% line | 22.1% | 89.5% | No |
| Model defers, suggestion for a bookkeeper | 2.6% | 10.5% | Yes |
97.4% of transactions have their account chosen without a bookkeeper involved, and choosing the account is the whole decision here, since the amount, the date and the cash side all come straight from the bank. The other 2.6% reaches a bookkeeper with an answer already attached rather than as blank work. Sampling and close-period checks still run across the 97.4%; what they do not do is put a person on every transaction.
What happens above the line
Accounting does not grade on a curve. The bar is where a transaction stops needing a human to categorize it, not where it stops being checked.
What can go wrong above the line is a classification error, not a cash error. The books reconcile either way. Corrected means a bookkeeper later changed the account the model chose, sometimes because it was wrong, sometimes because that company files that merchant differently. Before a period closes we re-score the answers that new evidence has touched, and push every bookkeeper correction across the merchants it implies. They get caught while the period is still open, not afterwards.
There is a second reason to want coverage high. The model runs on our own infrastructure, so every transaction it settles is one that never leaves our systems at all. Pushing coverage up is a privacy result as much as a cost one.
Benchmarks: the ranker against the LLMs
Everything above is one pipeline's history. This section is the head-to-head, because the two approaches were run against the same transactions and graded against the same answer: the account the bookkeeper kept.
One note on what is being measured. All of these numbers are on the residual, the quarter of transactions that deterministic rules cannot settle. Include the other three quarters and every approach here scores in the nineties, which is the mistake we made for months. Deterministic wins flatter any model.
Take the hardest slice: the transactions where our model declines to answer and hands it to a bookkeeper as a suggestion. On that slice the LLM path is correct 33.6% of the time against the model's 93.7%. Even where we defer, deferring to the model's answer beats deferring to a prompt.
Beyond accuracy, one difference matters more than the rest. A constrained answer space removes a class of error entirely. More than half of the LLMs' measured mistakes were not judgement calls but structurally impossible choices, including suggesting a bank account as the category, which is never valid. A model that ranks a generated list of valid candidates cannot make that mistake, because the invalid answer is never on the list.
Where the LLMs still earn their place is the genuinely novel transaction, the one with no precedent anywhere in the book. That is what they are good at, and it is what they still do here. What changed is that nothing from the LLM path posts without review anymore. International wires stay undecidable by either approach: no merchant, no counterparty, and a category that says only "this is a wire". The model gets 65% of those right, which is not good enough to suggest, and we would rather it said so.
What we would tell someone starting this
Evaluate on the slice that needs judgement. For months our dashboard said the LLM pipeline was 93% accurate. The number was real and meaningless: it was dominated by transaction shapes whose answer is determined by structure. Deterministic wins flatter any model.
A normalizer is not neutral cleanup. It is a set of assumptions about input format, and when it meets a format it was not written for it does not fail loudly. Ours manufactured plausible-looking merchants for two thirds of one bank's feed, and no accuracy work would have surfaced it, because a fabricated merchant does not throw an error.
Read your raw inputs, not your parsed ones. We had been looking at the cleaned merchant field for months. One afternoon reading what the bank actually sent answered a question that model tuning could not.
The company that onboarded yesterday
Everything above rests on history, which raises the obvious objection: a business that joined this morning has none. Its strongest features are all zero.
What carries it is what every other company already taught us. Roughly 75% of a new company's transactions are merchants some other company in the book has already categorized, the software, the payment processors, the airlines, the cloud bills. A new client is mostly not a new problem.
Two things make that usable rather than wishful. Every feature is a count, a ratio or a share, never a raw identifier, so one model serves a company with four years of history and one with four days. And the cross-company signal is computed leaving each company out of its own prior. When we first measured cold-start accuracy without that, a company was reading its own past back through the "global" number and the result was flattered by seven points.
We also train for it deliberately. Transactions with no local precedent are weighted three times in the loss, because they are 22% of the corpus and the model would otherwise spend its capacity getting better at what a lookup table already knew.
| Training | Answered | Precision |
|---|---|---|
| Without weighting the no-history case | 70.3% | 99.1% |
| Weighting it three times | 72.6% | 99.0% |
A company with no history at all gets roughly 72% of its transactions answered at 99% precision, on day one, from other companies' knowledge alone. The same weighting is worth far more at the strict end: coverage above the 99.92% line went from 39.5% to 57.4%.
Two kinds of transaction, two kinds of knowledge
There is an apparent contradiction here worth resolving, because it is the whole argument. Most of this post says a company's own history is what matters and that no amount of general knowledge substitutes for it. Then cold start answers 72% of transactions with no company history at all.
Both are true, because transactions come in two kinds. For some merchants the treatment is effectively universal: cloud hosting is hosting, payroll is payroll, and no company files them differently. Cross-company knowledge settles those completely, and they are most of the volume. For the rest there is a real judgement call, software that could be cost of revenue or an operating expense, a consultant who could be professional fees or a contractor, and that is where two companies disagree 55% of the time. General knowledge cannot settle those, by definition.
So the numbers say exactly what they should. 72% cold, 90% once a company has its own history. That eighteen-point gap is not a shortcoming of the cold-start case. It is the value of the company's own book, measured.
Which is also where the durability sits, in two layers. Cross-company knowledge means a client who signs up tomorrow starts at 72% rather than at zero, and that layer gets denser every week regardless of who joins. Their own history then earns the remaining eighteen points, and that part does not transfer anywhere.
This works as well as it does because the book is concentrated. We keep books for startups and technology companies, so a new client inherits knowledge from businesses that spend on the same things it does: the same cloud providers, the same payroll and billing tools, the same contractor and advertising patterns. The prior is industry knowledge in practice, because the industry is what our clients have in common.
Worth being precise about the mechanism, though: the model does not segment by sector. It draws on every company in the book, and today that is the same thing. If we ever keep books for restaurants alongside SaaS companies, segmenting the prior becomes a lever rather than a no-op. What the model does know the industry of is the merchant. Enrichment tells us a vendor is a telecom rather than a software company, which is why an unfamiliar phone bill lands in the right place even on day one.
Where this goes
The target is bookkeeping that runs without us. Three or four errors in a thousand is the tolerance, and it is a defensible one, because a good share of what gets counted as an error is not wrong, it is a convention. When two of our clients see the same merchant they file it under different accounts 55% of the time, and both are correct for their own books. Our error counts are measured against what one bookkeeper chose, so they are an upper bound on real mistakes rather than the number itself.
At that tolerance, today's model already automates into the low nineties. Closing the rest is not a modelling problem, which is the encouraging part: what remains is data, and data compounds in a way model quality does not.
We know where the remaining gap lives: the payment-processor volume the model is currently blind to, and the conventions it now learns after one or two examples instead of never. Neither needs a better ranker. Both are work we have already started.
Conventions stick faster now. Every correction a bookkeeper makes teaches one company's preference, and with honest merchant keys it holds after one or two examples instead of never. That is the fix in this post, and its value arrives gradually rather than on the day it shipped.
What one company teaches, the next one inherits. Cross-company priors are the only thing a business that onboarded yesterday has to go on, and they get denser every week without anyone doing anything.
The populations the model cannot see are mapping work. Payment processors, where no bookkeeper ever made the decision, do not need a smarter ranker. They need their integration settings read properly.
Retraining is cheap and takes an afternoon. The ledger is the asset, and it gets larger every day this runs. That is the part of the curve that improves whether or not anyone is being clever.
Common questions
How much bookkeeping does Vecty automate?
97.4% of transactions get their account without a bookkeeper: 75.3% settled by deterministic rules and shared merchant knowledge, 22.1% by a model trained on our own ledger. The remaining 2.6% reaches a bookkeeper as a one-click suggestion to accept or correct, not as blank work.
Why did LLMs underperform on transaction categorization?
Because the right answer is a convention, not a fact. Two companies file the same merchant differently about 55% of the time, and that lives in posting history, not in a model's weights. Two frontier LLMs, cross-checked, were right 57.8% of the time; a per-merchant lookup table scored 63.6% on the same transactions.
How does the model handle a company with no history?
From what every other company already taught it. Roughly 75% of a new company's merchants have been categorised elsewhere in the book, so a company with no history gets about 72% of its transactions answered at 99% precision on day one. Its own posting history then earns the rest.