ML Models · Serving

Part of the ML Models guide. This page covers the warm endpoints that answer in milliseconds instead of starting a container per request, the copies they scale across, and how a candidate version is tried on real traffic before it is promoted.

Warm endpoints#

By default a prediction starts a container, boots Python, imports the ML stack, downloads and digest-checks the artifact, scores, posts the answer back and exits — about twenty seconds before any scoring happens. That is the right shape for a batch job over a million rows and the wrong one for scoring a row behind a web request. A deployment holds the version it serves in memory and answers over HTTP instead, from Automation → Warm endpoint on the model page. While a candidate is being shadowed or run as a canary it holds that one too.

• The scoring is identical. The sandbox loads the same program the batch path runs and calls the same _predict, so the same fitted pipeline scores the same digest-verified artifact. Only the waiting is different.
• It pins a version. Promoting a new one marks the endpoint stale and leaves it serving what it was serving. An endpoint that silently changed its answers is the opposite of what pinning is for.
• A missing endpoint is slower, never wrong. Down, loading, or serving a different version, the prediction falls back to the sandbox. Every reply says which path answered, in served.
• Measured, one row, end to end: ~1.3 s warm against ~27 s cold on a laptop with a remote database. The scoring itself is 45 ms either way — that is the model, and it does not change. What the endpoint removes is the container start; what is left is the platform's own book-keeping, which on that setup is almost entirely round trips to a database in another datacentre.
• Recorded exactly like a cold prediction: the same row, the same drift check, the same ml.predict_query audit event. Faster, not less accountable — and those writes are part of the number above.
• It costs memory while it is up, so it is off by default, an idle one is stopped after its timeout unless Keep warm is on, and the instance caps how many may be open.

More than one copy#

One sandbox is one Python process scoring one request at a time, so the second caller waits for the first — and at that point the twenty seconds a warm endpoint saved are being spent again in the queue, somewhere less visible.

An endpoint can therefore hold several copies of the model, each in its own sandbox. Set the range on the deployment panel: the first number is how many are held even with no traffic, the second the ceiling. Both default to 1, so nothing changes until you raise the second — every copy is a container holding the ML stack and a fitted pipeline resident on your machine, and starting more because a feature shipped would be spending your memory without asking.

Requests go to the copy that has gone longest without one. That is the same rule the scaler uses to choose what to stop, deliberately: two notions of “quietest” would have the two disagreeing about the same endpoint.

When copies are added and removed#

The platform clock measures the endpoint's request rate — the change in its counter between two readings, not a sample — and compares it with ML_SERVE_TARGET_RPM_PER_REPLICA (120), the load one copy is sized for.

Adding is immediate. A queue is the thing a warm endpoint exists to prevent, so there is no cooldown before relieving one. One copy is added per pass however far behind the endpoint is: the next pass is a minute away and will add another if it is still needed, by which time the first has loaded — so the decision is made knowing what it bought. Starting four at once on a burst is how a machine runs out of memory serving a spike that ended before they loaded.

Every decision is recorded on the endpoint in plain words — the panel shows the last one — and every change is audited as ml.scale with the rate that caused it.

What a copy actually is, and how far it gets you#

A copy is a sandbox, started the same way every other sandbox is — so what it lands on depends entirely on the runtime backend. Nothing in the scaling code mentions either one: it asks the orchestrator for a scoring sandbox and gets back an address.

On Kubernetes these are bare Pods the app creates, not a Deployment. There is no ReplicaSet and no Service in front of them: the app holds each Pod's address and picks between them itself. So the HorizontalPodAutoscaler is not involved — the platform clock is the control loop — and copies do genuinely spread across nodes, which means an endpoint can outlive one of them.

Either way the warm-container limits still apply, and they count copies rather than endpoints, because the thing being bounded is resident memory.

Trying a version on real traffic#

A new version is normally adopted by switching to it. Which means the first evidence that it behaves differently from the old one is production behaving differently — noticed, if it is noticed, by whoever the difference landed on.

Shadowing asks the question first. Pick a version on the deployment panel and the endpoint starts a second copy holding it. From then on, every request the endpoint answers is mirrored to that candidate, its answer is compared against the one that was actually served, and then thrown away.

primary

What agree means is not the same question for every model, and pretending it is would make the number meaningless.

Counting exact float matches on a regression would report 0% agreement on two models that are indistinguishable in practice, so the comparison is a tolerance. Clustering, anomaly detection and recommendation are not compared and the mirror does not run for them: their labels are arbitrary between fits, so cluster 3 of one model has nothing to do with cluster 3 of another, and the comparison would report total disagreement between two identical models.

The panel leads with a sentence rather than a figure, and below a hundred compared rows it refuses to give one at all — it says how many more it needs. A percentage on forty rows invites a decision nobody has evidence for, which is the opposite of what shadowing is for.

A candidate is a copy, so it is a container, and it counts against the same warm-container limits as any other. One candidate at a time: choosing another retires the first, and the totals reset, because figures gathered against a different candidate answer a question nobody asked. Stopping it takes the copy down and leaves nothing behind. Adopting it is the ordinary Redeploy to that version — shadowing does not promote anything by itself, and never will. It measures; a person switches. Both starting and stopping are audited, as

ml.shadow.start

ml.shadow.stop

Giving it a share of real traffic#

Shadowing tells you the candidate answers the same way. It cannot tell you the candidate answers at all under real load, on real data, at real concurrency — because nobody was ever waiting for one of its answers.

A canary does. Switch the candidate from Mirror only to Send real traffic and a share of requests are answered by it, for real, and returned to whoever asked. The share is a whole number of per cent and the default is 5. This is the one place in the platform where a version nobody approved answers a real caller, so it is worth being exact about what bounds it.

A canary cannot measure agreement, and no amount of wanting it to will help: each row was answered once, by one version, so there is no second answer to compare it against. That is what shadowing is for, and why the two are separate steps rather than one slider. What a canary measures is failure — on both sides.

Production's figures are there because the question is never “is the candidate failing” but “is it failing worse than the thing it would replace”. Without them, a lakehouse outage reads as a bad model. And because it measures failure rather than agreement, a canary works for every task — including clustering, anomaly detection and recommendation, where shadowing cannot be offered at all.

ml.canary.rollback

Every prediction records the version that actually answered it, not the one the endpoint is nominally serving. Under a canary those differ for some share of rows by design, and recording the endpoint's version would attribute a candidate's prediction to production — wrong on the row somebody reads when they ask why, wrong in the drift figures, and wrong in exactly the cases anybody is looking into.

A candidate already running as a shadow keeps its warm copy when it becomes a canary: same version, same container, and throwing away a loaded model to change one column would cost twenty-five seconds for nothing. The canary figures reset because they describe a run and this is a new one; the shadow totals are left alone because they are still true. Adopting the candidate is the ordinary Redeploy to that version — nothing here promotes anything by itself. The only thing the platform does on its own is take a failing candidate out.

More rows than one container holds#

A dataset bigger than the row ceiling used to be sampled: the trainer took a reservoir sample down to

ml_train_max_rows

Against a single fit on all the rows the same measurement costs a little, and the cost is governed by how many rows each worker still gets rather than by how many workers there are. There is no cliff, so the floor is a line drawn on a curve: below 25,000 rows per worker the platform refuses to split and samples the old way, because a fast answer that is worse than the slow one is not a feature.

ORDER BY

The version records the rows it actually covered and carries a note saying it was split, how many containers over, and that pasting is not the same as one fit over everything. If the workers between them still cannot hold all the rows, the note says how many were left out. The metrics are the search's, measured on the holdout the search kept rather than re-measured on the slices — each worker's own holdout is a piece of its own slice, so a metric averaged over them would be measured on data each fit had seen a neighbour of.

Nothing here can leave you without a model. If no worker will start, if every slice fails, or if no container is free to combine them, the job keeps the model the search already produced and says on the version that this is the sampled fit. A job that splits its rows takes three phases rather than one — search, refit, assemble — and each hands over to the next exactly once, claimed in the database so that of several workers finishing together only one moves the job on.

How many copies you can run#

A deployed model is one scorer replica per copy: a container holding Python, the ML stack and one fitted pipeline, answering requests. Measured on a laptop deployment, a loaded scorer that had just answered a prediction sat at 169 MB resident and 0.02% CPU idle, and served in 0.105 s.

Its memory ceiling is ML_SERVE_MEM_LIMIT_MB (2 GB), and that is a different number from the training budget on purpose: training fits a model on up to two million rows; serving holds one finished model. They used to share ML_TRAIN_MEM_LIMIT_MB — 8 GB — which cost nothing on one host, because a Docker limit reserves nothing.

When the cluster is full, a pod stays Pending — which from outside looks exactly like an image still pulling. Kubernetes writes the difference into the pod's PodScheduled condition, and the platform repeats it: the deployment reports waiting for room in the cluster with the scheduler's own message, instead of ending in "the scorer did not become ready". It stays starting rather than failing, because a pending pod becomes schedulable the moment a node arrives — which is precisely what a cluster autoscaler does when it sees one.