ML Models · Training

Part of the ML Models guide. This page follows a model from a lakehouse table to a versioned, promoted model; Predictions is what happens once it exists.

Prepare a training set#

The wizard's Prepare the data panel filters rows and fills, scales and encodes columns for one model. For the wrangling that comes before that — joins across tables, dedupe, split and replace, pivots, derived columns, aggregation — use Data preparation with lakehouse tables: link the tables, build the steps with a per-step preview, and Save as → lakehouse table. The result is an ordinary lakehouse table in a schema you own, rebuilt on the flow's schedule, and it appears in the wizard's table picker at once.

Train a model#

1. 1

   Data

   Pick a lakehouse table you own or that was shared with you. The profile shows each column's kind, distinct count, nulls and samples.
2. 2

   Goal

   Predict a column (the task follows from the column; a forecast also takes a time column, the period - hourly to quarterly, or automatic from the dates - the periods ahead and how rows in one period combine), find groups (a fixed number or the best by silhouette), find anomalies (the share you expect, 2% unless told otherwise), or recommend items (a user column, an item column and an optional strength).
3. 3

   Options

   Name, description, features (identifier-like and constant columns are off by default) and Prepare the data: a row filter (SQL WHERE) or a custom SELECT for joins and derived columns — Check runs it through the lakehouse guard as you and reports how many rows match — how missing values are filled, standardisation, one-hot or ordinal encoding, balanced class weights, winsorised targets; hyperparameter tuning (none, quick or thorough, run on the two best candidates while at least 40% of the budget remains and kept only when it beats the untuned model on the holdout); the time budget and the row limit.
4. 4

   Review and Train

   The model page streams the job's logs while it runs and shows metrics the moment it finishes.

A search across several sandboxes#

A training job tries several algorithms and then tunes the best of them, and by default it does all of that inside one container, one candidate after another. Set Search workers under Admin → Developer runtime (or ML_TRAIN_WORKERS) above 1 and the search is dealt out instead: worker w of n takes candidates w, w+n, w+2n… and the job keeps whichever worker's model scored best.

• Only classification and regression have a search to split. Clustering picks its k from the row count and forecasting its methods from the shape of the series, both inside the sandbox, so the server cannot deal out their candidates.
• Never more workers than candidates, or than the runtime allows. Sessions per user (3 by default) is the ceiling that actually bites, and it counts open notebooks too. A job takes fewer workers rather than failing to start the extras.
• A worker that dies does not lose the job. Three of four finishing still produces a model; the leaderboard is merged from everyone who reported, each row keeps the worker that ran it, and the version's warnings say how many did not come back. Only an all-workers-failed search fails.
• Ties break on the lowest worker number, so re-running the same job on the same data picks the same model.
• A split search is not the same search. Tuning runs per worker, on that worker's own best candidates, so several workers tune more models than one container would have and can land on a different winner from identical data. Usually a better search — but the two are not comparable runs, so train with the same worker count each time when exact reproduction matters.

Read the results#

• Metric tiles — the primary metric first, then accuracy, ROC AUC, log loss, MAE, R², MAPE as the task allows.
• What the model relies on — permutation importance on the holdout set: how much the score drops when a column is shuffled. It names the columns a person recognises, not one-hot fragments.
• Groups for clustering: every group's size and share with its typical row — the mean of each number, the most common category.
• Confusion matrix for classification; a forecast chart with history, projection and a residual-based band for forecasts.
• Leaderboard — every candidate tried, scored on the same holdout, with fit time, status and tuning trials.
• Lineage — rows (and whether sampled), snapshot, decision id, artifact digest, warnings.

What the trainer warns about#

A score can be right and still mislead. The trainer checks for the usual ways and writes what it found on the version: the Versions tab counts them as notes on every version and opens them in place, the compare view lists them side by side, and the model card, the agent's prediction tool and the public API's model listing repeat them.

• Possible leakage — a single feature that predicts the target almost perfectly on its own (98% balanced accuracy, or 98% of a numeric target's variation) is usually the target in disguise: a code for it, a column filled in after the fact, a key the model memorises. The warning names the column; if it is derived from the target or unknown at prediction time, leave it out and train again.
• The do-nothing baseline — when nine rows in ten share one class, that share is the accuracy of predicting it every time. The warning says so and points at F1 (macro), the primary metric, and the confusion matrix.
• No signal — a regression whose R² is at or below 0.05 explains about as much as the mean would.
• Columns that decide a distance on their own — clustering and anomaly detection compare rows by distance, so a column with more than 20 categories groups rows by its value rather than describing them, and a time column groups them by when they happened: the "segments" become customers, the "anomalies" the earliest and latest dates. Both are left out when the features were chosen automatically, and kept with a warning when you picked them yourself.
• The anomaly rate is a setting — the detector flags the top 2% (or the contamination you set) on clean data as much as dirty. Read the score, and set the share you expect.
• Strength is not sentiment — a recommendation's strength column adds up, so a 1-star rating still counts as a weak like. If low values mean dislike, filter those rows out first.
• Forecast history — a first or last period the data only partly covers is left out; an empty period of a total counts as 0 (an empty period of an average is interpolated); the holdout is at least three periods once there are twelve; a projection of a series that never goes below zero is floored at zero.
• A random holdout is not a time split — classification and regression hold out rows at random. If your rows are events over time, the score describes rows like the ones you have, not next quarter's; train on a prep flow that stops at a date to see how the model ages.

How the winner is chosen#

Training tries several algorithms and keeps the best. What “best” is measured against is the question this section answers — because for a long time the answer was wrong here, in a way that flattered every model the platform produced.

The mistake, and what it cost#

Every candidate used to be fitted on the training rows and scored on the holdout. The best of those scores picked the winner, the tuner then searched against the same holdout, and that very number was published as the version's metric.

What happens now#

Selection happens inside the training rows, and the holdout is read once, at the end, by code that is only reporting.

Which scheme a version used, and why, is written on the version and shown under the metric tiles — together with the spread between folds, which is what makes the headline number readable. The spread is how much the score moves when the same model meets different rows, and therefore the scale below which a difference between two versions is noise.

Folds cost k fits per candidate, so they are not always worth paying for. What decides is the size of the holdout, not the size of the training set: a few thousand held-out rows already pin the score to well under a point, while a few dozen pin nothing at all. The line sits at ML_CV_MIN_HOLDOUT_ROWS (2000), also editable under Admin → Developer runtime. A class with fewer examples than folds lowers the fold count; a class with a single example turns folds off altogether, because no set of folds can each contain one.

The winner is refitted on every training row before it is saved. The folds existed to measure; the model that ships should have seen all the data selection was entitled to use.

The holdout is read once#

Two numbers therefore appear on a version, and they are not the same number: across the folds, which chose the winner, and on the held-back rows, which nothing was allowed to optimise against. The second is what the version reports and what a decay alert compares production against.

They are shown side by side on purpose. When they disagree the disagreement is information: a winner that looked good on the folds and did not repeat itself on untouched rows is telling you something a single number would have hidden. Calibration is decided the same way — keeping or discarding it is also a choice, so it is made on a slice of the training rows, and only then are the Brier score and calibration error measured again on the holdout for reporting.

Rows that are ordered in time#

A table with a time column must not be split at random. Shuffling rows that have an order puts next month in the training set and last month in the holdout, and the score that comes back is the score for predicting the past from the future — reliably flattering, and reliably wrong the first time the model runs for real.

Name a time column and three things change: rows are sorted by it, the most recent slice is what gets held back, and the folds become TimeSeriesSplit — every fold trains strictly before the rows it scores, on an expanding window of history. Time order wins over every other consideration, including a holdout large enough that a random split would otherwise have been used.

If the column turns out to hold no readable dates, the run falls back to a random split and says so in the run log. Quietly shuffling rows after being told they are ordered is the version of this bug nobody would ever find.

Versions#

Tick two or more trained versions on the Versions tab to compare them side by side: every metric they share, rows, tuning and training time, with the best value in each row marked. The Model card button assembles one Markdown document from the registry rows — intended use, training data and snapshot, preparation, features and dropped columns, metrics and leaderboard, importance, groups, warnings, governance, how to call it — copied or downloaded; nobody types it, so it cannot drift from what shipped. The Versions tab lists every version with its stage, algorithm, primary metric, rows and snapshot. Promote makes a ready version the production one and archives the previous; Archive withdraws a version without deleting its metrics or passport; Restore returns it to the candidates. Train new version re-reads the table as of the current snapshot with the model's saved data preparation, and takes its own budget, row limit and tuning mode.

Experiments#

Versions record what you shipped. Experiments record what you tried: the twenty runs behind the one version worth keeping, which otherwise live in a notebook's output cells until somebody re-runs it. Then "why is this the learning rate" has no answer a month later, and a colleague cannot see that the obvious idea was tried and did not work.

An experiment is a named question; a run is one attempt at it, with the parameters it used and the metrics it got. Anything that can reach the platform can log one — a notebook with its session token, a script with a user token. From a notebook the client is already injected:

import agentswarms

with agentswarms.start_run("churn-v2", params={"lr": 0.01, "depth": 6}) as run:
    for epoch in range(10):
        run.log_metric("loss", loss, step=epoch)
    run.log_metrics({"auc": 0.91, "accuracy": 0.88})
    run.finish(artifact_uri=uri, artifact_sha256=digest)

• It fails in the right direction. start_run raises if it cannot start — a run you believe is recording and is not is worse than one that never began. Every later call warns and continues: losing an epoch's metrics is not worth losing the epoch.
• The curve survives, and so does the score. log_metric(key, value, step=n) keeps the point as key@n and updates the bare key to the latest value. The panel draws those points as a sparkline beside the metric.
• It shows what actually varied. In a list of twenty runs the parameters held constant are noise; the ones that moved are marked, because that is the experiment.
• A finished run is finished. Later writes are refused, so a straggler from a process that outlived its own finish cannot rewrite the record.

Find them under ML Models → Experiments. Nothing is created there: the first start_run() call creates its own experiment.

From a run to a version#

A run that recorded both artifact_uri and artifact_sha256 can be registered from its row as a model version — both, because a version whose artifact nobody can verify is not a version. It goes through the same path a registered external version takes, so the digest is checked before inference loads it and the artifact follows the same contract. It arrives as a candidate: promoting it is a separate, deliberate step on the Versions tab.

Saving the model from the notebook#

Producing that artifact used to be the author's problem: write a joblib file in the registry's contract, get it into the lake bucket without the bucket's credentials (a kernel does not hold them, on purpose), hash it, and only then call finish. So notebook-authored models stayed in notebooks. Two calls close that gap: run.save_model(pipe, features=list(X.columns), task="classification") dumps the pipeline in the external contract, sends the bytes to the platform, and the app writes them beside the artifacts its own trainer produces. The digest recorded is the one the app computes from the bytes that arrived — a digest the caller reported would be a digest nobody verified, and this one is what inference checks before loading. Unlike the logging calls this one raises: a save you believe happened and did not is the same lie as a run that never started.

run.register("churn", task=…, source=…, target_column=…) then turns the run into a version — of a model by id or by name, and a name nothing owns yet creates the model from the lakehouse table the training data came from, checked as you. It arrives as a candidate, except on a model with nothing in production yet — always true of one this call just created — where the registry promotes the first version, as it does everywhere else. Both work from outside the platform with a user token, on /api/ml/experiments/artifact and /api/ml/experiments/register. One upload is bounded by ML_ARTIFACT_MAX_MB (512 MB), editable under Admin → Developer runtime → Machine learning.

Who signs off a promotion#

Promotion is audited but ungated by default: anyone with write access can put a version in front of customers on their own. Where that is not enough — model-risk policy usually asks for a second signature before the change, from somebody who did not make it — name the approvers under Versions → Who signs off a promotion. With approvers named, Promote stops promoting and starts asking: the version keeps serving whatever it serves until one of them agrees, and the request appears under Pending approvals in the header, beside the swarm approvals. The same table, the same inbox — there is no second approvals system.

Only production is gated — moving a version to staging or archiving it changes nothing a customer meets. The button says what it will do: with a gate on, the confirmation asks whether to request the promotion and says the version keeps serving what it serves now, because the dialog is where a gate is first visible and promising an immediate switch there would be a lie told at the moment somebody decides whether to press. The audit names both people — ml.version.promote.requested when it is asked for, and ml.version.promote with approved_by when it happens.