ML Models · Predictions

Part of the ML Models guide. This page is about using a model that exists — on demand, in batch, from an agent or the AI Analyst, over the API and by key; Trust covers whether its numbers can be believed.

Predictions#

Try it#

A form generated from the feature schema, medians and category lists filled in. One row is scored in a sandbox, so allow half a minute; the result shows the predicted class with its confidence and the per-class probabilities, or the predicted number.

Batch prediction#

Pick an input lakehouse table with the same columns, an optional filter, and an output schema you own plus a table name. Every row is written back with prediction, probability, one proba_<class> column per class, _model_version and _predicted_at. The result is an ordinary lakehouse table: agents, the SQL workbench and dashboards query it like any other. The operator's ML_PREDICT_MAX_ROWS is checked before a sandbox starts.

Every prediction run carries its own decision id unless it serves an agent's turn, in which case it adopts that turn's; success is audited as a data read (ml.predict_query) with a digest over the prediction column and the row cap, so a replay can tell “same model, same rows, same answers” from drift.

Agents#

Enable ML Predictions in an agent's tools. The agent gets ml_list_models (name, task, target, feature columns with categories and ranges) and ml_predict (rows in, predictions out). Both are offered only when the caller can use at least one model with a production version; on headless runs grants are re-derived from the run's owner. Forecast models return their projected periods.

Which models the agent may use is a picker under the tool: every model you can use — your own and those shared with you — with its task, and whether it has a production version yet. Leave it untouched and the agent may predict with all of them, which is what every agent did before the picker existed. Select some and the agent is restricted to exactly those: ml_list_models shows nothing else, and ml_predict refuses anything else by name, even a model it was told about in an earlier turn. The same picker sits on an agent node on the swarm canvas, and importing an agent into a node carries its selection across — including an empty one, which means no models and must not turn into all of them on the way.

The model's health travels with it. The platform already notices a model's rows drifting from what it learned on (a PSI on every prediction run, an alert past ML_DRIFT_ALERT_PSI) and its accuracy decaying against outcomes that arrived later (an evaluation with a verdict) — and told the owner. Now every place a model is offered carries the latest reading of each as one sentence: ml_list_models returns a health block per model and, when an alert is open, a note telling the agent to say so beside any prediction it reports; ml_predict appends the alerts to its notes, so a prediction never arrives without them; the AI Analyst's planner sees the health beside each scorable model and a scored step's badge and disclosure carry it; the Playground's model list marks an alert. Below the threshold and stable, it still says what was measured and when — "no alert" and "never measured" are different facts.

Scoring by key. A model bound to a feature view (Automation → Input on the model page) can be scored by naming the row instead of describing it: ml_predict takes keys — [{"order_id": 1000}] — and the platform reads the features from the same table training read, through the online store when the view is served from one. ml_list_models marks such models with their feature_view and key columns, and the tool's own description tells the agent to prefer keys for them: an agent that types twenty feature values from a conversation is the training–serving skew feature views exist to remove. The answer carries the key column(s) on every prediction, keys_not_found for any key that matched nothing (never a row of nulls scored quietly), and features_served_from (online, mixed or lakehouse). Rows and keys are never accepted together; more than fifty keys is refused rather than trimmed, because a key dropped silently is an entity reported as scored.

On the canvas without an LLM. A swarm's tool node offers Score with model (ml_predict): pick the model, give keys — a JSON array such as [{"order_id": {{input}}}], templated from flow state like any tool argument — or rows with the feature values, and the node writes the same JSON the agent tool returns to its output variable. No model decides whether to score, which is the point: when every run should score the same rows the same way, a model choosing to is cost and variance. It runs the same code as the agent tool (one implementation, runMlPredict), honours the node's model list, and in a deployed or scheduled run scores as the swarm's owner. Any error — an unknown model, keys and rows together, a list that is not valid JSON, a key set that matches nothing at all — fails the node rather than flowing downstream as if it were a result. A key that matches nothing among others is not an error: the rows that matched are scored and it comes back named in keys_not_found beside them.

What the Playground shows. The inspector's Tool Calls panel shows a prediction as a table — key columns first, then the prediction and its probabilities — with the model and version above it and, beside it, how many rows were scored, where the features came from and which keys were not found; ml_list_models shows the models with their task, version, headline metric and key columns. The table is built on the server from the tool's full result (the panel's generic preview is a 400-character slice, which for a prediction ends mid-probability), and an error is shown as the error.

In the AI Analyst. A plan may score a step's rows with a model in scope ("score": { "model": "<name>" }): the step's SQL selects the entities, the model adds the prediction columns, and the step carries a scored badge with the model, version, headline metric, rows scored and where the features came from. The write-up reports predictions as the model's estimates, never as observed values. A forecast model is offered as a forecast step ("forecast": { "model": "<name>", "horizon": N }) — no SQL, the model's projected periods with their interval — and a plan may rank the scored rows by a model output ("rank": { "by": "anomaly_score", "desc": true, "limit": 10 }), the only way to ask for the most anomalous or most likely rows, because a model's columns exist in no table. Which models an analyst may use is its own setting, beside its reasoning model and its data, with the same rule as an agent's ML tool — any, exactly these, or none — enforced when a step scores. Details under Business Intelligence → AI Analyst.

Automation#

The model page's Automation tab schedules two kinds of work, each running as you, in the same sweep, under the same cron lease and with the same reaper as ETL pipelines and materialized views.

• Retrain — a new version from the current table every hour, day, week or on a cron expression, with its own budget and tuning mode. With promote when better on, the new version becomes production the moment it is ready if its primary metric beats the incumbent; you are told either way.
• Batch prediction — score a lakehouse table (optionally filtered) into a table you own with the production version, so a scored table stays fresh for dashboards and agents without anyone clicking.

Run now starts a schedule immediately; pause keeps it without running it; resuming schedules from now, never from the missed past. Every start is audited (ml.schedule.run / ml.schedule.failed) and the schedule rows are audited by trigger.

Public API#

A model can be published as an API. Publish as API on the model page mints a key that looks like mlk_…, shown once and stored hashed, scoped to that one model with any of predict (score rows, start batch runs), train (train a version, register an external one) and read (list the model, poll jobs and runs). Every call runs on the same service the app uses — the same limits, the same lakehouse guard, the same audit trail — and is attributed to its key; a denied call (unknown, revoked, expired, wrong scope, rate-limited) is audited as ml.api_key.denied with the caller's address.

curl -X POST https://your-instance/api/ml/predict \
  -H "Authorization: Bearer mlk_…" -H "Content-Type: application/json" \
  -d '{"rows":[{"region":"EMEA","net_usd":480,"payment_rows":1}]}'
# → {"prediction_id":"…","columns":["region","net_usd","payment_rows","prediction","probability",…],"rows":[[…]]}

curl -X POST https://your-instance/api/ml/predict/batch \
  -H "Authorization: Bearer mlk_…" -H "Content-Type: application/json" \
  -d '{"input":{"schema":"analytics","table":"revenue_facts"},"output":{"schema":"analytics","table":"revenue_scored"}}'
# → 202 {"accepted":true,"prediction_id":"…","output":"analytics.revenue_scored"}

Answers use ordinary status codes: 401 for a missing, unknown, revoked or expired key, 403 for a missing scope, 404 for a job or run of another model (never 403, so a key learns nothing about what it cannot see), 409 when the service refuses (no trained version, a limit reached, a schema you do not own) and 429 above the per-key rate limit, ML_API_RATE_LIMIT_PER_MIN calls a minute (sixty by default, edited like every other limit).

Bring your own model#

A model trained elsewhere — a notebook, a laptop, another platform — can serve through the same registry. Write the artifact into the lake bucket as a joblib dictionary with task, pipeline (any object with predict, plus predict_proba for a classifier), features (the input columns, in order) and, for a classifier, classes, then register it with its SHA-256; inference verifies the digest before loading it, hands the pipeline the raw feature columns, and returns the same columns a trained version would. Classification, regression, clustering and anomaly models accept external versions; the first one is promoted when the model has no production version.

What a forecast period is#

A forecast is one value per period: the total (or average) of the target over every hour, day, week, month or quarter, as chosen in the wizard. Automatic infers the period from the gaps between timestamps, which turns a table of dated orders into a daily series — fine for a month of data, surprising when you expected months. Pick the period you will read the answer in. A last period the data only partly covers is left out and the version says so. Five candidates compete on a holdout of the most recent periods — last value, moving average, seasonal naive, Holt-Winters, gradient boosting on lags — and the lowest RMSE serves, so a flat line means the flat baselines beat the rest on your series. The model page, the agent tool and the API all state the period, the aggregation, the last observed period and the method.

Forecasting in BI#

Line charts on a dashboard project ahead with the platform's shared forecaster: seasonal exponential smoothing when the history shows a season that beats a straight line, a linear trend otherwise, with a residual band that widens with distance. The AI Analyst and the alert engine use the same module, so a chart, its write-up and its alert cannot disagree.

A forecast model from the registry can be attached instead: in the widget's time-series options choose it as the Source beside the period count, and the chart draws the model's projected periods. An alert's basis can be the forecast — the aggregate over the next N projected periods — evaluated at each scheduled refresh against the model's current projection, as the dashboard's owner.

Feature views#

A model trained on a table whose columns were built by SQL is normally scored by POSTing those same column names with values the caller computed itself, in its own code, months later. Nothing checks that its arithmetic matches the training set's, so the model receives numbers of the right shape and the wrong meaning and answers confidently. That is training-serving skew, and it is quiet.

A feature view removes the caller's arithmetic. It names a table, the column(s) that identify a row, and which columns are features; serving then takes a key and reads the values from the same table training read. Under ML Models → Feature views, attached to a model under Automation → Input.

curl <origin>/api/ml/predict \
  -H "Authorization: Bearer mlk_…" -H "Content-Type: application/json" \
  -d '{"keys": [{"customer_id": "c-1"}]}'

• It materialises nothing. The table is whatever built it — a SQL model is the natural author, since its schedule keeps the table fresh and its unique test can assert the key.
• It does not guess. Rows come back in the order the keys were asked for, matched by key rather than by result order. A key matching nothing is named in keys_not_found rather than filled with nulls, because a row of nulls scores happily and means nothing.
• It refuses an ambiguous key. A key matching two rows fails unless the view names a column for latest row wins. Picking one of two arbitrarily is how a feature store starts lying.
• Nothing changes without one. Callers keep sending whole rows, and keep owning them.
• It builds point-in-time training sets. See below.

Serving them in milliseconds#

A feature lookup reads the lakehouse, and the lakehouse is an analytics engine. Measured across 300 audited statements on a laptop deployment, none finished in under 127 ms and the median was 336 ms — before reading a single row. The cost is not the scan; it is the access check in front of every statement, which is a round trip to the application database, and it is paid whether the query touches eight hundred rows or eight million. Fair for a dashboard. Poor for a prediction asking for one customer's six numbers, which pays it every time.

The online feature store keeps a view's latest row per key one hop away. Turn it on per view with Serve online, then Refresh to fill it — the refresh reads the table through the same governed path as everything else, so the owner's access, the row-level policies and the audit row all still apply.

End to end, the same Look up against the same key went from 439 ms to 252 ms. The lookup is the part that collapses; the remainder is the request's own work — checking who is calling, loading the view — which this does not touch and which is now the larger half. Quoting the 2 ms as the whole request would be a measurement of the component sold as a measurement of the system.

• It refuses a view whose key is not unique. A view with no timestamp column declares its key unique. If the refresh finds two rows sharing one it stops and names the key, rather than storing whichever it happened to see first.
• It refuses rows from a view that changed. The store records the view's table, keys, features and timestamp column, and compares them on every read. Edit the view and its stored rows are the old shape, so they are ignored until a refresh replaces them.
• It says what it actually holds. A store bounded by FEATURE_STORE_MAX_KEYS or by its own memory holds part of a view, which is correct — the rest is read from the lakehouse — and the panel says how much.
• Staleness is a setting. FEATURE_STORE_STALE_MINUTES (60 by default, overridable per view) is how old the rows may be before lookups stop trusting them. A table rebuilt nightly and one rebuilt every five minutes do not want the same number.

The panel reports where lookups are actually answered from rather than what the switch is set to: a view that is on but stale is being served from the lakehouse, and that is the moment the difference matters. Start one with docker compose up -d and point FEATURE_STORE_URL at it. It is valkey rather than Redis for a licensing reason — Redis moved to RSALv2/SSPL, which this project cannot ship in its own stack — and the protocol is the same, so that variable may name a Redis you already run.

Point-in-time training sets#

Serving asks what an entity's features are now. Training has to ask what they were at the moment the label was true, and the difference is the most expensive mistake in applied ML. Join a February label to the feature table's latest row and the model learns from June's numbers: it scores beautifully in the notebook, because the answer was in the features, then fails in production where June has not happened yet.

Training set on a view builds the honest version. Give it the table of labels, the column saying when each was true, and the label column matching each key column; every row then keeps the features with the greatest feature timestamp at or before its own — an ASOF LEFT JOIN, resolved by the engine rather than by a window function you have to get right. Left, because a key whose features start later is still part of the training set. An optional max feature age refuses to join something stale without dropping the row.

The build reports how many rows would have differed under the join written by hand — the leak, as a number, on your own data. The view's timestamp is never joined in as a feature: a model that trains on the feature clock learns the shape of your ETL schedule. A view with no timestamp column cannot build a training set and says so, rather than joining the latest row and calling the result one.