Case study · Proprietary AI model
In short: the product had to make millions of small decisions a month that people were making by hand. A general-purpose AI could not do it: the decisions are specific to the client's business and have to be right. We selected a model, trained it on the client's own history, and made it small and fast enough to run inside the product. It now handles 87% of the work on its own at over 99% accuracy, asks a person about the rest with its best suggestions ready, and answers in milliseconds.
Every item the product receives has to be sorted into one of about thirty categories before the client's customers see correct figures. An earlier model existed, but on the hardest categories it was right about half the time, and nobody knew how much of the work it could safely be trusted with. The product needed AI it could rely on: accurate, fast enough to feel instant, and cheap enough to run on every item.
Chose the right model. We tested ten candidates on the same data and the same terms, from simple statistical methods to six modern language models, so the choice was made on evidence rather than fashion. The winner beat the best simple method by six points of accuracy, and by twelve points on the rare cases that matter most.
Taught it the client's business. The model was fine-tuned on four and a half years of the client's own history, tested on months it had never seen, so the results are honest about how it will perform on next month's data. We also measured where the client's own history contradicts itself, which tells the business what no model can fix.
Taught it when to ask. A model is only useful in a product if you know when to trust it. We calibrated the model's confidence against its real accuracy, so the client chooses the balance: handle 87% automatically at 99.45% accuracy, or 74% at 99.79%. Everything below the line goes to a person with the model's top three suggestions.
Made it fast and cheap to run. The winning model was compressed to a tenth of its size with no loss of accuracy, so it runs on ordinary servers in a few milliseconds per item instead of needing specialised hardware. Memory per server fell by almost half and no longer grows with load.
| Measure | Before | After |
|---|---|---|
| Overall accuracy | 95.7% | 97.6% |
| Two hardest categories | 54% and 65% | 97% and 99% |
| Work handled automatically, at 99.45% accuracy | not known | 87% |
| Time per decision | 38 ms | 3.4 ms (11× faster) |
| Model size | 438 MB | 45 MB |
| Memory per server | 760 MB, growing with load | 424 MB, flat |
Note that the accuracy figures come from 2.5 million held-out items the model had never seen. A few categories moved the other way by a small margin; the client sees the full per-category picture and sets the automation threshold with it in mind. Staff time saved is the client's to measure and is not claimed here.
The broader lesson is that AI embedded in a product is a different job from AI as a chat window. The model has to be the client's own, right almost all the time, honest about the rest, and fast enough that nobody waits for it.
Services: Software development · AI-native engineering
Case studies: A cloud-native platform built for cost and scale · The knowledge base that found the defects · All case studies
Book a free 30-minute call. We will look at the decisions your product makes and tell you honestly whether a model of your own is worth building.
