Case study · Engineering standards · Code review
In short: the organisation had good written standards that nobody enforced. Twenty services each handled failures their own way, and an AI review step ran on every change with an empty context, producing generic comments at a considerable cost. We moved the standards into shared code and into the build, so a new service gets the right behaviour by default, and rebuilt the AI review so that every finding cites a written rule and the reviewer tests itself. Consistency went from 13 of 20 services to 20 of 20, the test suite grew by 612 passing tests, and the review's running cost fell from about USD 20 a day to about USD 20 a week.
The organisation runs several products built over years by different people. Its coding guidelines described how services should behave and how changes should be reviewed. Both were documents: nothing in the build checked the first, and the automated review that was meant to check the second had been set up once and never measured.
Seven of the twenty services had no global error handling at all, so an unexpected failure reached callers as an empty error. The rest used two different rule sets that disagreed, so the same failure could arrive at the front end as two different errors, and one of them copied internal details into the response for any caller to read. Around it, hundreds of log statements were written in a way that defeated the organisation's monitoring.
The AI review step had a different failure of the same kind. The rules it was meant to apply were never loaded, so it reviewed every change with no knowledge of the organisation's standards, truncated large changes, failed silently when a comment could not be posted, and cost about USD 20 a day against a budget of two or three dollars. One week consumed 54 million input tokens to produce 324 thousand of output: the model was mostly re-reading the same material.
No behaviour visible to callers or partners could change. Every change had to be piloted in one place and approved before it was copied to the other services, because a design that evolves during review has to be redone everywhere it was already copied. The AI review had to remain advisory, so that a model outage never blocks a release, and honest about its own state: a review that did not run must never look like a review that passed. And the rules had to be enforced by the build rather than by reviewers, so that they do not erode.
One error handler was introduced in the shared library, reproducing every existing status code from a single map and following one rule: only the organisation's own exceptions may show their message to a caller; everything else is logged in full and returned as a generic error. It was piloted in one module, mounted in seven services, then in all twenty. The same sweep fixed 323 log statements and the latent bugs the change exposed. Two rules were moved into the compiler as analyzers, one of which turns a violation into a build error, and the guidelines were rewritten to match what the build now enforces.
The AI review was rebuilt around the same principle. The organisation's architecture, authorisation and security documents were compiled into compact checklists with numbered rules, and every finding must cite a rule, because a finding without a rule cannot be discussed or dismissed consistently. The review runs in two passes, a fast scan on a small model and a logic review on a stronger one, loads its context from files rather than the prompt, and caches what does not change. Each finding is mapped to its line, de-duplicated across runs and resolved automatically when the code is fixed; the pipeline reports one of three honest states rather than a green tick. A fixture with six seeded violations runs through the reviewer on every change, and at least five must be caught. The harness was extracted into shared scripts that are byte-identical across the three repositories.
| Measure | Before | After |
|---|---|---|
| Services with consistent error handling | 13 of 20, two rule sets | 20 of 20, one |
| Internal details returned to callers | yes | no |
| Rules enforced by the build | 0 | 2 analyzers |
| Automated tests passing | 4,657 | 5,269, none failing |
| AI review findings that cite a written rule | none | all |
| AI review running cost | ≈USD 20 a day | ≈USD 20 a week |
| Review pipeline definition | 1,845 lines, one repository | 453 lines, three repositories |
Note that the cost figures are the team's own observations at two points in time rather than a billing report, and the quality of the review is protected by the seeded-violation test rather than claimed.
In general, a standard that lives only in a document is followed by the people who read the document. Moving it into shared code, the compiler and an automated reviewer that cites it means every change, by a person or by an AI, meets the standard before anyone spends time on it. That is the same principle behind our AI-native engineering work.
Services: AI-native engineering · Architecture assessments
Case studies: AI-native engineering for a software organisation · Faster screens and half the storage · All case studies
Book a free 30-minute call. We will show you which rules are worth moving into the build first, and what your AI tools should be checking.
