Skip to content

Data & analytics

ML Models · Operations

Sharing and governance, the configurable limits, day-to-day operations, and what to do when something reads wrong.

Part of the ML Models guide. This page is for whoever runs the registry: sharing, the limits and the settings behind them, day-to-day operations and what to do when something reads wrong.

Sharing and governance#

Share a model from Admin → IAM → Access as ML model. A grantee can predict with it — try-it, batch, and through agents — and read its metrics; training, promotion, renaming and deletion stay with the owner. Batch outputs are written to a schema the caller owns, never to a shared or mounted one. Creation, renaming, promotion and deletion are audited by a database trigger (ml_model.create, ml_model.update, ml_model.delete); training and prediction events are audited by the server with the decision id (ml.train.start, ml.train.succeeded, ml.version.promote, ml.predict_query).

Limits#

Every limit resolves settings row → environment variable → default and is edited under Admin → Developer runtime; nothing in the code caps them. A large VM or a Kubernetes node pool is allowed to use itself.

SettingDefaultWhat it bounds
ML_TRAIN_MAX_ROWS2,000,000Rows one training run reads; larger tables are reservoir-sampled
ML_TRAIN_TIME_BUDGET_MINUTES30Default wall-clock budget per run
ML_TRAIN_MEM_LIMIT_MB8192Memory ceiling of a training sandbox
ML_MAX_CONCURRENT_TRAININGS_PER_USER2Training jobs one user may have live at once
ML_PREDICT_MAX_ROWS5,000,000Rows one batch prediction may score
ML_API_RATE_LIMIT_PER_MIN60Calls a minute one ML API key may make
ML_TRAIN_GPUS0GPUs requested per training sandbox
ML_DRIFT_ALERT_PSI0.25PSI above which a prediction run raises a drift alert

Operations#

  • Training and inference need the notebook runtime services: docker compose up -d. The runtime image bakes scikit-learn, LightGBM, statsmodels, DuckDB, pyarrow and s3fs; an older image installs them at job start.
  • The sandbox reads Parquet through the egress proxy; its allow-list is brought up to date with the lake endpoint automatically before a job starts.
  • On Kubernetes training and prediction are batch Jobs in the notebook namespace, bounded by its ResourceQuota and LimitRange and scaled by adding nodes; the egress ConfigMap must admit the object store. The deployment guide's ML platform on Kubernetes section has the three settings that matter.
  • GPUs: ML_TRAIN_GPUS (or the Admin setting) requests that many GPUs for every training sandbox — a Docker device request on a single host, an nvidia.com/gpu limit on Kubernetes. The baked runtime image is CPU-only; point NOTEBOOK_RUNTIME_IMAGE at a CUDA-capable build when you need one.
  • Artifacts live under ml-artifacts/ in the lake bucket. npm run backup mirrors the lake data path; add the artifacts prefix to your object-store backup as well.

Troubleshooting#

SymptomCause and fix
Cannot reach the Docker socket-proxyRead the rest of the message: "start the runtime services" means the proxy is down (docker compose up -d); "did not answer" means the Docker daemon is busy or the proxy wedged — retry, then restart notebook-docker-proxy.
egress proxy refused the lake endpoint (HTTP 403)The allow-list is re-applied at job start; if it persists, save the runtime settings under Admin → Developer runtime and check the egress config mount is writable.
You already have a model called …Names are unique per user; the wizard defaults to <table> · <target>.
A target is greyed outIdentifiers, free text and constant columns cannot be predicted; pick another column or prepare the data.
Every candidate failedOpen the job's logs on the Jobs tab; the first candidate's error is quoted.
Possible leakage: … on its own predicts …A feature is the target in disguise or a key the model memorised. Drop it from the features and train a new version; the score will fall to something real.
Every group is one customer / every anomaly a dateA many-valued category or a time column was selected explicitly and decided the distance. Let the trainer choose the features, or leave that column out.
Projected values below 0 were floored at 0The winning method extrapolated a decline past zero; the floor is the honest answer for a total. A longer history or a coarser period usually steadies the trend.
PreviousTrustNext Workflows