In short: the product had to process its customers' financial data close to real time, grow with the customer base, and stay cheap to run, and the client would not trade any one of those for the others. We designed a cloud-native platform where cost is part of the architecture rather than something tuned afterwards: processing that scales with the work and idles when there is none, managed services where they are cheaper than running your own, and an operation that grows without growing the team.
Context
Self-managed retirement funds generate a steady flow of transactions, valuations and reports, and the people who run them expect to see current figures rather than last night's. The client was building a product to serve many such funds from one platform. Each new customer adds data and processing but very little revenue per fund, so the economics only work if the cost of serving one more customer is close to zero.
Problem
Scalable and cost-effective pull in opposite directions in most designs. A platform sized for peak load pays for idle capacity most of the day; a platform sized for average load falls behind when the market opens. And near-real-time processing is usually the expensive option, because it means keeping systems warm and ready. The client needed all three at once, and needed the architecture to prove it before the build started, not after the first cloud bill.
Constraints
Financial data has to be correct, auditable and kept for years, so nothing in the design could lose or reorder a transaction to save money. The platform had to be operable by a small team. And the cost model had to be explicit: every component justified by what it cost per customer, not by what it was capable of.
What we did
We designed the platform around events rather than schedules. Incoming data triggers processing as it arrives, so figures are current within minutes, and the processing capacity scales up with the flow and down to nothing when it stops. Storage and compute are separated, so data grows without growing the cost of processing it, and long-term history is kept in the cheapest tier that still meets the audit requirement.
As a general rule, a managed service is cheaper than the engineer who would otherwise run it, so the design uses managed cloud services for everything that is not the product's own logic: queues, storage, scheduling, monitoring. The whole platform is declared as code, so a new environment is a deployment rather than a project, and the cost of every component is visible in the same place as its configuration.
Before the build, we modelled the running cost per fund at several sizes of customer base and checked the design against it, so the client knew what the platform would cost at ten times the customers before committing to it.
Outcome
The client deployed a scalable platform with near-real-time data processing whose running cost grows with usage rather than with capacity, operated by a small team. The design decisions, the cost model and the operational runbooks were handed over with the platform, so the client's own team carries it forward.
In general, the cheapest cloud platform is the one whose cost was designed in: once a system is built to be always on, no amount of tuning afterwards makes it cheap.
Related
Services: Cloud infrastructure · Architecture assessments · Software development
Case studies: Faster screens and half the storage · Near real-time AI in a financial product · All case studies