ML Models · Trust

Part of the ML Models guide. This page is the evidence for believing a model today: drift, per-row explanations, accuracy against outcomes that arrived later, calibration, the decision threshold and how groups are treated.

Drift#

Training records the distribution of every feature — decile bins for numbers, the top categories for categoricals. Every batch prediction (and any direct prediction of ten rows or more) bins the new rows the same way and reports a population stability index per feature; the run's Drift badge shows the highest one: below 0.1 stable, 0.1–0.25 moderate, above 0.25 the population has moved. A run above ML_DRIFT_ALERT_PSI (0.25 by default) is audited as ml.drift.alert and notifies the model's owner with the three most drifted features — the cue to retrain, or to schedule retraining. The public API returns the same numbers in /api/ml/predict/status.

Why this row got this answer#

The model page shows what a model relies on overall — permutation importance over the raw input columns, measured once when the version trained. That answers “what does this model key on”. It does not answer “why was this customer declined”, which is the question a person asks when the answer is about them, and in credit, insurance or hiring it is one you may be obliged to answer.

Tick Explain this answer under Try it on the Predictions tab. Each feature comes back with how far the answer moved when its value was replaced with the one a typical training row carried: bars to the right pushed the answer up, bars to the left pushed it down, in probability for a classification and in the target's own units for a regression.

It works on any model, including one registered from a notebook, because it only ever calls predict. The one case it declines is a classifier with no predict_proba: without probabilities the only measurable move is that the label flipped, which is a yes/no rather than a contribution, so it returns nothing rather than dressing a coin flip as a number.

It costs one extra prediction per feature per row, so it is opt-in and bounded — ML_EXPLAIN_MAX_ROWS rows per request, ML_EXPLAIN_TOP_K features back for each — and an explained call takes the sandbox path even when a warm endpoint is up, because the endpoint's serving program would need its own copy of the ablation and a second implementation of “what moved this answer” is a second definition of it. An explanation that fails never costs you the prediction: the answer comes back with a warning attached.

Reason codes on every scored row#

The explanation above answers for one row you are looking at. A batch answers for all of them: tick Write reason codes beside every row on the batch prediction dialog and the scored table gains the drivers as columns.

Flat columns rather than a JSON blob, because the point is that WHERE reason_1 = 'support_tickets' works in plain SQL and a dashboard can group by it. The value that drove the answer is not repeated — it is already in the row, in the column the reason names. A row with fewer features that moved anything than there are slots gets nulls, not blanks.

They cost one extra prediction per feature per row, so a hundred thousand rows with twenty features is two million predictions. The work is chunked so memory stays flat however large the batch is, but the time does not. A batch above ML_EXPLAIN_BATCH_MAX_ROWS (50,000) is therefore refused before the sandbox starts — the row count is already known from the check that enforces the prediction limit, so the answer names the real number and the way out.

Refused rather than truncated, on purpose: a scored table where the first fifty thousand rows carry reasons and the rest are null looks complete and is not, and nothing downstream would know. Narrow the rows with a filter, score without reason codes, or raise the ceiling. ML_EXPLAIN_BATCH_TOP_K (3) sets how many are written, and each one costs two columns.

Was it right?#

Drift and this are different questions, and treating the first as an answer to the second is the most common way a model quietly stops working. Drift says the rows arriving now do not look like the rows the model trained on. Inputs can shift while accuracy holds, and inputs can sit perfectly still while the world changes underneath the label. The only way to know whether a model is still right is to wait for the real answer and compare.

So a model may name an outcome source — the table where the real answers land, and the key that lets a scored row find its own. Set it on the model page under Accuracy: a schema and table, one to eight key columns present in both that table and the scored one, and the column holding what actually happened. Rows where that column is still null are skipped.

An evaluation joins one prediction run's output table to it and recomputes the model's own primary metric — f1_macro for a classification, rmse for a regression or forecast — on the rows that have an answer, then compares it to the same metric on the validation split when that version trained. A run more than ML_DECAY_ALERT_RATIO worse (0.10, ten per cent) is audited as ml.decay.alert and notifies the owner. A ratio rather than a metric value, so it reads the same way for a metric that should rise and one that should fall.

It runs on the platform clock: answers arrive over hours or weeks, so each successful batch run is re-measured once a day while it is less than a month old (ML_EVALUATIONS_PER_SWEEP bounds one pass). Measure now does one immediately.

The metric is recomputed exactly as scikit-learn computes it, because the baseline came out of that same call at training time — two defensible definitions of one metric would fire a decay alert the first time every model was measured, which teaches everyone to ignore decay alerts. Evaluating costs no sandbox: a confusion matrix and five sums are a GROUP BY, so one statement runs through the governed lakehouse chokepoint as the model's owner and the arithmetic happens in the app.

Is 0.8 really 80%?#

Every classification carries a probability, and this interface has always printed it beside the word confidence. For a tree ensemble that number is usually a rank rather than a frequency: a forest that votes 9 trees to 1 reports 0.9 whatever the real rate turns out to be. Good enough for sorting a queue, wrong for a rule that says “auto-approve above 80%”.

So the trainer measures it, and the model page shows the measurement under Accuracy → Confidence and the decision line.

The reliability curve#

Holdout rows are binned by what the model said, and each bin reports what actually happened. A point on the diagonal means the model's 70% really was 70%; above it the model is under-selling itself, below it over-selling. Bins are drawn in proportion to how many rows they hold, because four rows landing far off the line is noise and four hundred is a problem.

The page bands the calibration error rather than leaving a bare decimal: at or under 0.05 it is safe to write a rule against, under 0.15 it is fine for ranking and loose for a rule, and above that the numbers should be read as ranks.

What the trainer does about it#

After the algorithm search picks a winner and before any metric is recorded, classification models get a calibration pass — CalibratedClassifierCV, isotonic regression on 1000 training rows or more and Platt scaling below that, since isotonic needs data to fit its step function and overfits badly without it.

When the pass is discarded the run log says so and the page says left uncalibrated. That is not a failure: a model already well calibrated lands there, and so does one whose holdout was too small to fit a reliable mapping. Because metrics are recorded after this step, every number on the version describes the model that was actually saved. Versions trained before this shipped have no curve and read as not measured, which is the truth — retrain to get one.

Where the line is drawn#

A classifier decides by argmax, which is a threshold of 0.5 that nobody chose. It is the right default and the wrong one for most real decisions: declining a good customer and missing a fraudulent order do not cost the same, and the person who knows the ratio is the operator, not the trainer.

So the trainer measures every operating point and the model page lets you pick one. For a two-class model the holdout is scored at thresholds from 0.05 to 0.95 in steps of 0.05, and every row of the table is a real measurement:

The sweep is always expressed from one side — the second class, named on the page — and that loses nothing: with two classes the probabilities sum to one, so a line at 0.70 on retained is the same rule as a line at 0.30 on churned. Every operating point either class could have is already in the table, read from one end.

The best-F1 row is marked balanced and offered as a starting position, not a recommendation — F1 weights the two mistakes equally, which is the exact assumption this screen exists to let you reject. Choosing a row shows what would change against the line currently in use, and saving it asks first. The picker only offers thresholds the trainer actually measured: interpolating to 0.437 would present a number the platform never checked with the same authority as one it did.

A setting, not a retrain#

The threshold lives on the version, not inside the artifact. Prediction reads it at run time, so moving the line takes effect on the next prediction and the model is untouched. Every change is audited as ml.threshold.set with the value, because “who decided to approve 12% more applications, and when” is a question that gets asked.

• The probability shown is the probability of the answer given. A row declined at 0.45 reports 0.55 against the class it was actually assigned, not 0.55 confidence in a decision nobody made.
• Scored tables record the line that produced them. A batch run with a threshold set writes threshold_applied on every row, so six months later “why was this one declined” is answerable from the row rather than from whatever the setting happens to be by then.

A retrain does not carry the line forward#

Because the threshold lives on the version, a new version arrives without one and decides by argmax again. That is deliberate: a line only means the same thing across two versions whose probabilities mean the same thing, and copying it forward silently would be the platform making a business decision on your behalf.

It is also the sort of change nobody notices until approval volume shifts, so it is not left silent either. When the production version has no line and an earlier version of the same model did, the panel says so — naming the version and the value, with a button to draw it there again. Scheduled retraining with promote when better is exactly the case this is for.

Multiclass models get no threshold and no sweep: there is no single line to draw, so each prediction is simply whichever class scores highest. Regression and forecasting have none either.

How groups are treated#

Two questions, and each hides the other. Selection rate asks how often each group gets the favourable answer — it needs no outcomes at all, so it can be checked the moment a batch runs, and it is the one employment and lending law is written about. Error rates ask whether the model is wrong more often for one group, which needs the real answers and so rides on the same join an evaluation makes. A model can have near-identical selection rates and still be far worse at one group, which is why both are reported.

Set it on the model page under Accuracy → How groups are treated: up to eight columns present in the scored table, and the predicted label that counts as the good outcome. Each column is compared separately and recorded as its own check — two columns are two comparisons, and averaging them would hide the one that matters.

Where the agent layer helps, and where it does not#

This is the first place in the platform where a language model touches a number somebody may have to defend, so the boundary is explicit and tested: the platform measures, the model proposes and narrates, and a number never comes from the language model.

Suggest columns asks the assistant to nominate what to compare by — both directly sensitive attributes and proxies, the columns that are not themselves sensitive but stand in for one: a postcode for ethnicity, a first name for gender, a school for class. Proxies are the valuable half, because they are what careful people miss. Each suggestion comes with a reason you can disagree with, and you tick what applies; nothing is enabled by the suggestion itself, because which attributes are protected is a legal question about your context rather than one this platform can answer.

Only column names, types and cardinalities are sent. Never values — a column of ethnicities is sensitive data, and posting a sample of it to an inference endpoint to ask whether it is sensitive would answer its own question. The suggestion path never queries the lake at all, and a test enforces that. Any column the assistant names that is not in the schema is dropped rather than shown.

Explain this in words passes the already computed figures to the assistant and asks for two or three sentences. The prompt forbids it from computing, estimating, rounding differently or introducing any figure it was not given, and the narration is stored beside the numbers rather than instead of them — so one that drifts is visibly contradicted by the table above it.

Both calls go through the same governed door as every other model call, so IAM model rules, budgets and audit apply; they are recorded as ml.fairness.suggest and ml.fairness.narrate, and the assistant model is ML_ASSIST_MODEL. A check is audited as ml.fairness.check, or ml.fairness.review when the ratio falls below the line, which also notifies the model's owner.