Case study · Performance & running cost
In short: a platform that had grown over years was slow where it mattered most and larger than anyone could explain. We did not rewrite it. We measured where the time and the space actually went, fixed the structural causes, and put each result under a test so it cannot quietly come back. The slowest screen went from 85 database round trips to 9 and from half a second to two milliseconds per lookup; the largest tables shrank by half; and storage is now declared and reviewed like code.
The platform serves a large organisation and its external partners. It consists of a couple of dozen services, each with its own database, built over several years by changing teams. The data grows with every transaction and is kept for audit, so the databases are among the largest the organisation runs, and the screens that read them are opened far more often than they are changed.
Two symptoms had been reported separately. The most-used screen took five to twelve seconds to open, and the ticket blamed one slow query. And nobody could say how large the databases should be, because storage settings had been applied by hand over the years and were recorded nowhere: on one server, part of a table was compressed and the rest was not, and one table was declared two conflicting ways, so its state depended on which script had run last.
Both symptoms had the same root: the system had never been measured. Without measurement, every fix is a guess, and a guess that happens to work is undone by the next deployment.
Nothing visible could change: the same screens, the same interfaces to partners, the same data. Years of history had to remain provably untouched. Diagnostic instrumentation had to be switched off again once the evidence was captured, and every setting had to live in source rather than in a one-off script, so that the next deployment would keep it.
We started from evidence rather than from the ticket. Three nights of query history showed that the slow screen was not one slow query but a structural pattern: the service loaded a tree of data one branch at a time and issued two further queries for every branch, so the largest and most important records had the slowest experience. We rewrote that read path to load the tree level by level in batched queries, resolved lookups through an existing index instead of a scan, and fixed a projection that had been pulling full rows into memory to filter them there.
For storage we first built the ability to measure. A load-test harness runs representative reads and writes against a database, touches only rows it inserted itself, and proves with checksums that the real data was not modified. A readiness gate blocks any measurement until every change has fully applied, which caught two half-applied changes that would otherwise have been reported as results. With measurement in place we estimated the saving for every table, adopted one simple compression rule, removed indexes that nothing used, and moved every setting into the schema projects, recorded in a decision record and a guide other teams can reuse.
As a general rule, a performance fix should be protected by a test, because a query count is easy to lose in a later refactoring and nobody notices until production slows down again. So each improvement is guarded: a test that fails if the number of round trips grows, and a re-verification script that confirms the storage state on every rollout.
| Measure | Before | After |
|---|---|---|
| Database round trips to open the most-used screen | 85 | 9 |
| One lookup on that screen | 40,704 reads, 506 ms | 10 reads, 2 ms |
| Largest tables, development server | 516 MB | 254 MB (−50.7%) |
| All compressible tables, estimated | 11.1 GB | 2.6 GB (−76.6%) |
| Reads on scan-heavy queries (lab) | baseline | −84% |
| Unused indexes removed | — | about 1 GB |
| Improvements protected by a test | none | all |
Note that these are measurements from the development environment, the test suite and a scratch database built from the real schema; production is re-verified with the same scripts as each rollout proceeds, and the end-to-end latency is confirmed on the client's nightly run.
The broader lesson is that a ticket describes a symptom, and the cause usually sits one level higher. The durable result is not the saving itself but that speed and storage are now measured, declared and reviewed like any other part of the code, so that they cannot drift back to an unknown state.
Services: Architecture assessments · Cloud infrastructure
Case studies: A cloud-native platform built for cost and scale · Standards enforced by the build · All case studies
Book a free 30-minute call. We will tell you what we would measure first, and what the measurement usually shows.
