Models that ship like software.

01 — The problem

The marketplace had an AI search built on hosted LLM calls. It was accurate enough to be interesting and too slow and fragile to trust: 30 to 60 seconds per query, one query in five failing outright, and a per-query bill. The client's own AI team had a competing approach. Nobody could say, with a number, which one was better, because nothing was measured the same way twice.

Replacing it with self-hosted models raised the real question: how do you train, ship and change models in a product that people are using, without the "it seems better" arguments starting again?

02 — Data: a labelling pipeline, not a spreadsheet

• A canonical taxonomy first. We audited every role assigned across roughly 4,400 user records, found the duplicates, seniority variants and verb-noun variants, and collapsed them into one canonical 106-role taxonomy that both the classifier and the search vocabulary use.
• Real labels, in batches. 475 production profiles were labelled against the taxonomy in 15 reviewed batches, producing 1,839 role assignments at an average of 3.9 roles per profile. The labelling protocol is written down so it can be repeated by someone else.
• Synthetic data where the real data was thin. For role families with too few examples, including the seniority ladders that general models confuse most, we generated targeted synthetic profiles per family and kept them in separate, named batches so their effect on the model can be measured and reversed.

03 — Training: versioned, reproducible, compared

• One training script, mixed precision. A fine-tuning pipeline for a RoBERTa multi-label classifier with fp16 automatic mixed precision, per-class positive weighting for the long tail, linear warm-up, and per-class threshold tuning on a held-out set.
• Every run is a version. v1 (5 epochs) under-trained at 0.43 macro F1; v2 (15 epochs) reached 0.52. Each run's config, data batches and metrics are kept together, so the answer to "which model is live and what was it trained on" is a lookup, not an archaeology project.
• Bake-offs against hosted models. The classifier was scored against a hosted LLM baseline on the same validation set. It won on macro F1 and, more importantly for the product, produced zero invented role names, which the LLM could not manage even with a closed-world prompt.
• The reranker was retrained too. When the search engine plateaued below its target, we retrained the cross-encoder reranker on synthetic pairs, re-ran the judge, and moved the hit rate past the 80% goal. Same loop, different model.

04 — Serving: GPU inference that behaves under load

• Containerised models. Embedding model, reranker and classifier are packaged as images in a registry and deployed to GPU inference instances behind an internal load balancer with an autoscaling group.
• Concurrency handled explicitly. The reranker runs in fp16 with a request lock and a result cache, so concurrent queries do not starve each other of GPU memory. The brief tagger parses in about 10 ms and its first-hit latency fell from 30 seconds to 13 seconds once caching was in place.
• Embeddings stay fresh. A change-stream indexer watches the primary database and re-embeds profiles as they change, so the vector index never drifts from what the product shows.
• Staging mirrors production. The full stack, including a local copy of the vector database, can be stood up from scripts to reproduce any issue offline.

05 — Evaluation: the loop that decides what ships

Nothing above matters if you cannot tell whether a change made things better. This is the part we treat as the product.

• LLM-as-judge with a fixed protocol. Candidate lists from the old and new engines are bundled per query and scored blind by a frontier model on a fixed rubric. 170 verdicts settled the head-to-head: 81.8% hit rate for the new engine against 72.4% for the incumbent. The client's internal AI team was redirected on the strength of that number.
• A 328-fixture regression gate. Real briefs, alias pairs, filter edge cases and geography, run on the production box before and after every deploy. Red gate, no release. The gate has been extended release by release; the thirteenth version ran with zero violations on the day the ranking blend shipped.
• Byte-level replay proofs. Every change is replayed against 75 fixed real briefs. 71 of 75 came back byte-identical after the vocabulary layer went live, and the four that moved were the four that were meant to, each with a written reason.
• Versioned vocabulary. The alias and taxonomy layer is versioned like code, and every release is diffed against the previous one so a vocabulary change can never silently reorder results.

06 — The outcome

Median query time fell from about 49 seconds to 2.4 seconds with no failed queries in the benchmark and no per-query LLM spend. Exact-match accuracy on the client's own audit rose from 46.5% to 89% across releases. More important than either number: the client can now retrain a model, change vocabulary or swap a component and know within minutes whether search got better or worse, with a gate that will not let a regression through.

07 — What this means for you

If your team has a model that works in a notebook and a product that needs it, the gap between the two is not modelling, it is operations: data you can regenerate, training you can reproduce, serving that scales, and an evaluation loop that makes the ship decision for you. That is what we mean by MLOps, and it is what an AI Reliability Audit is designed to put in place first.