Workflows

Open Data & BI → Workflows. A workflow is a graph: each step is something the platform can already run, and each arrow means after. A step starts when its parents have finished the way its trigger rule requires, steps that nothing orders run at the same time, and a step whose branch was not taken is not run at all.

The fifteen kinds of step#

Nine start work the platform already does. Two reach outside it. Four are control flow, which an orchestrator has to own itself because it is about the graph rather than about any one system.

Work#

ETL pipeline

Runs one of your pipelines, exactly as its own schedule would. Its overlap guard and concurrency cap still apply, and the run's parameters are handed on to it.

SQL models

Builds models. Naming them builds those and their ancestors; leaving it blank builds every active model.

SQL statement

One statement against the lakehouse, under the same rules as the workbench: schema-qualified writes, into schemas you can already write to.

Data prep flow

Refreshes a prep flow's output table.

ML schedule

Runs a retrain or a batch prediction you have already defined, including its promote-if-better rule.

Notebook

Runs a notebook in a batch sandbox, with the run's parameters in its inputs.

Swarm

Runs a swarm's PUBLISHED graph, not the draft on the canvas. A swarm that stops at an approval fails the step — an unattended run has nobody to ask.

Refresh dashboard

Re-queries a dashboard's widget snapshots. A per-widget failure is reported without failing the step; the dashboard did update.

Data monitor

Runs a standing check. An ALERT fails the step on purpose — putting a monitor in a graph is how you stop what comes after it when the data is wrong.

Reaching outside#

HTTP request

One request through the platform's SSRF guard. Headers are edited a row at a time, and a row's value can be one of your secrets picked by name — resolved server-side, never reaching the browser. Any 2xx is success unless you name the statuses that count.

Notify

An in-app notification, mirrored to whatever notification integrations are connected.

Control flow#

Condition

Evaluates one comparison and takes the true or false arrow.

Wait

Pauses for a fixed number of seconds, up to a day.

Approval

Raises a request in the approvals inbox and waits for a person. Give it a timeout if it should not wait forever.

Sub-workflow

Runs another workflow whole, and waits for it.

Building a graph#

1. 1

   New workflow

   Name it for what it produces, not for when it runs.
2. 2

   Pick steps from the palette

   Down the left, grouped by what a family of steps is for. Each kind has its own colour, and the canvas uses the same one.
3. 3

   Join them

   Drag from a step's right edge to another step's left edge. Double-click an arrow to remove it. Two arrows into one step means it waits for both, unless you change its trigger rule.
4. 4

   Configure each step

   The panel on the right has two halves: what the step runs, and how the run treats it — trigger rule, retries, timeout.
5. 5

   Set the schedule and parameters

   Under Settings: manual, hourly, daily, weekly or a cron expression with a timezone.
6. 6

   Save, then Run now

   The run view updates while it works.

A loop is refused the moment you draw the arrow that would create one. Nothing else about the graph blocks an arrow — you can join two steps before either is configured, which is the order people actually build in.

Trigger rules#

Airflow's names, deliberately: somebody who knows one orchestrator should not have to learn a second vocabulary for the same three ideas.

An all_done step is never skipped for an upstream failure — waiting for everything to finish is the whole point of it. A one_success step is skipped only once every parent has finished without one succeeding.

Branching#

A Condition step evaluates one comparison and takes one arrow out of it. The first arrow you draw from it is the true branch and the second the false one; the canvas colours them green and red and labels them. Everything on the branch that was not taken is marked skipped, transitively.

You do not write the comparison. The step editor gives you three controls — this, a test, that — where each side is one of the workflow's declared parameters or a value you type, and the test is said in words: is, is not, is more than, is at least. There is no syntax to get right.

Nothing in the editor asks for a notation#

A step is configured by making choices. Three things used to be small languages, and each is now a control that can only produce something valid: a condition is assembled from pickers, headers are a row each rather than Name: value lines, and the models a build step covers are ticked from the ones that exist rather than typed as a comma-separated list.

The header value is where this matters most. Instead of typing {{secret:NAME}} from memory, the row offers a value I type or any of your secrets by name. Only names reach the browser; the value is resolved on the server at run time.

What stays free text is text that really is text — a SQL statement, a notification, a question for a person. Even there, a Parameter button inserts {{ params.name }} at the cursor so the spelling is never something to remember.

Parameters#

A workflow declares parameters with defaults under Settings; a run may override any of them. The resolved set is pinned onto the run, so what it used stays readable after the defaults change.

{{ params.name }} is filled into a SQL statement, an HTTP URL, header or body, a condition, a notification and a swarm's input, and the whole set is handed to an ETL pipeline step so a pipeline that takes a date window gets the run's window.

Retries, timeouts and failure#

Each step carries retries (attempts after the first, up to 10) and a first wait that doubles each attempt and stops at an hour. A retry reuses the same row, incrementing its attempt count, so a step stays one line in the run view however many times it was tried.

Skipped is deliberately not a failure. A step that never ran tells you nothing about itself, and folding the two together makes a run report say four things broke when one did.

Carry on if this step fails reverses that for one step: its children run anyway, and its failure does not redden the run — the failure was declared acceptable in advance. Use it for the step that refreshes a dashboard, not the one that loads the data.

Runs#

• One at a time. A workflow will not start a second run while one is in flight — two runs of the same graph would start the same pipeline twice and race each other's tables.
• The graph and the parameters are pinned. Editing the workflow never rewrites the history of what ran.
• Re-run repeats a finished run with its own parameters. From failed keeps every step that succeeded and starts at the ones that did not — the point of a re-run is usually the step that broke, not the four-hour load above it.
• Cancel stops anything that has not started. Work already in flight keeps its own life: the pipeline or training job it started is not killed.
• Notifications are per workflow: never, on failure (the default), or every run.

Steps are advanced by the platform scheduler, which sweeps once a minute, and again whenever you open a live run — so a manual run visibly moves while you watch it.

Starting a run from outside#

Under Settings → Start it from outside, mint a bearer token. It is shown once and stored as a hash, so a lost token is rotated rather than recovered.

curl -X POST "$APP_ORIGIN/api/workflows/run" \
  -H "Authorization: Bearer wfk_…" \
  -H "Content-Type: application/json" \
  -d '{"workflow_id":"…","params":{"day":"2026-01-01"}}'

“No such workflow”, “no token minted” and “wrong token” all answer one undifferentiated 404, so a caller holding a valid token for one workflow cannot enumerate the ids of the others. A paused workflow answers 409. Accepted calls are rate limited by WORKFLOW_TRIGGER_PER_MIN (default 6), globally rather than per process so the ceiling holds across replicas.

Knobs#

Who may run it, and what gets recorded#

A workflow belongs to the account that created it. Every server function resolves the caller from their access token and scopes the query by owner, and there is no workflow grant type in IAM — orchestration is not shared, the same way an ETL pipeline is not.

The audit trail#

Two writers, and the split is deliberate. The database records the row changes through a trigger, so creating, deleting or reshaping a workflow is recorded even by a write that never went through the app. The app records what a row change cannot show: that a run started, how it was triggered, and that somebody was refused.

The run event is written in one place inside the runner rather than at each caller, so a run started by the scheduler, by the API or by a parent workflow is recorded on exactly the same terms as one somebody clicked. trigger is the column that tells them apart, and api means a bearer token was accepted.

Getting from a step to its logs#

A step is a remote control, not the machine: when a model build fails, the reason is in the build's own log. Each step in the run view carries a link to where its work keeps its logs, with the first characters of the run id beside it to correlate against. A swarm step links to its own trace; the rest link to the page that owns the run, because only a swarm run has a page of its own today and a link to a 404 is worse than no link. Detached work — a SQL statement, a prep flow, a dashboard refresh, a monitor — never had a run row to point at, so those steps show their output inline instead.

The one-hop chain a pipeline can carry is still there and still works — see ETL Pipelines. Reach for a workflow when the shape is a graph rather than a line.