Data Catalog & SQL

Open Data & BI → Data Catalog.

Which kind of source do you want?#

There are four ways to get data in here and they are not interchangeable. Picking the wrong one is recoverable but tedious — a CSV uploaded monthly by hand is a warehouse connection nobody made, and a warehouse connected for a one-off question is a credential you now have to look after.

Local tables#

1. 1

   Upload

   Drag in a .csv or .xlsx, or paste rows. The first row is treated as headers.
2. 2

   Check the inferred column types

   A column of 2024-03-01 must be date, not text, or date filters and time-series charts will not work on it later. Fix it now — it is far more annoying after dashboards are built.
3. 3

   Name it for the model

   monthly_revenue is chosen correctly by agents far more often than sheet1. Lower case, underscores, no spaces.

External connections — every field#

Configure these under Integrations → Data Sources → Add connection. Every connector has a Name (your label) and a Test connection button; the last test result and its error are kept on the connection so you can see when it started failing. All secrets are encrypted at rest and never returned to the browser.

PostgreSQL#

MySQL / MariaDB#

Microsoft SQL Server / Azure SQL#

Speaks TDS through the tedious driver, so it needs a Node or Docker deployment — there is no REST SQL API to fall back on.

ClickHouse#

Uses the HTTP interface, so it works on any deployment.

Snowflake#

Uses a programmatic access token (PAT), not a password — generate one in Snowflake under your user's settings.

Databricks SQL#

The warehouse_id comes from the SQL warehouse's Connection Details tab.

Google BigQuery#

Amazon Redshift#

Two shapes: Serverless (set workgroup_name) or provisioned (set cluster_identifier and db_user). Fill one pair, not both.

Azure Synapse (dedicated SQL pool)#

Trino / Starburst / Presto#

Amazon Athena#

Oracle Database / Autonomous DB#

Connects over ORDS — plain HTTPS, so no wallet and no Instant Client. Autonomous Database ships ORDS enabled.

Object stores and external table catalogs#

S3, Cloudflare R2, MinIO, DigitalOcean Spaces, Backblaze B2, GCS over its S3 API, any other S3-compatible endpoint, and Azure Blob Storage / ADLS Gen2 (account key or SAS token) are added as catalog sources through Data Catalog → Add source. A crawl lists the bucket, groups a folder of same-format files into one dataset, and records each file’s columns. Iceberg REST and Unity Catalog are connected for metadata only.

Parquet, CSV, JSON, NDJSON and ORC files are queryable. Press Query data on a file in the catalog and it opens in the Workbench with the bucket selected as the engine. The SQL name is the file’s basename without its extension, so data/orders.parquet is orders; a partitioned folder sales/*.parquet is sales. Files in the same bucket can be joined, including across formats — a Parquet fact table against a CSV lookup is an ordinary query.

Direct query vs import#

Sharing a connection with your team#

A connection belongs to whoever created it. Rather than every analyst creating their own — several copies of one credential, each rotated separately, each a place it can leak — a superadmin can share it under Admin → IAM → Access, to a user or a group, as either a Database / warehouse connection or an App source.

A shared connection runs as its owner. The credential is the connection — a grantee has none of their own — so the owner's credential is decrypted server-side and the query runs against the owner's warehouse. A grantee can query it, test it and see its health; they cannot see the credential, edit it or delete it. {{secret:NAME}} references resolve as the owner, never against the grantee's own vault.

A shared app source syncs as its owner, into the owner's datasets — so a teammate who notices stale data can re-run it and refresh the real datasets rather than building a parallel copy under their own account. The audit entry records both who triggered it and whose data moved. Sharing the source lets someone keep it healthy; to let them read the resulting data, share those datasets too.

Apps — SaaS sources#

Databases are queried in place. Apps have no query language, so they are pulled into datasets instead: Integrations → Apps → connect, discover what is in there, choose what to sync. Each stream becomes its own dataset and is then indistinguishable from an uploaded CSV — same type inference, same version history, same use in BI, prep flows and the semantic layer.

Auth is a pasted credential, never OAuth: a redirect flow needs a public callback URL that a self-hosted deployment behind a firewall may not have, so each connector uses the vendor's server-to-server credential instead.

Syncs run on demand or hourly / daily / weekly, and you are notified if one fails or comes back partial.

Following a source instead of re-reading it#

A sync does one of two things, and the Streams button on a connection says which for every stream it syncs.

Full refresh re-reads the source and replaces the dataset — the right semantic where rows are edited and deleted in place with nothing to filter on, because an append would resurrect deleted rows for ever. The previous contents are snapshotted as a restorable version first, so a sync that pulls a truncated source is recoverable.

Incremental asks the API for records changed since the last sync and folds them into the dataset by key. Salesforce, Shopify, HubSpot, Jira, ServiceNow and GitHub follow every stream they offer, as do Linear and Asana; Stripe follows its six immutable object types; Zendesk follows tickets and users; Intercom follows contacts and conversations; Freshdesk follows tickets and contacts; Klaviyo and Notion follow everything they offer, and GA4 follows every report on its date. Everything else is a full refresh — Zendesk has no incremental export for organizations, Intercom none for admins, Freshdesk none for companies or agents, and neither a Google Sheets worksheet nor an Airtable table has a timestamp to follow at all.

The first pass has no high-water mark, so it reads everything and replaces: merging a full read into a stale dataset would leave rows the source has since deleted, for ever. The mark is written only after the rows are committed — advanced first and then lost to a failed sync, the next run would skip that whole window and nothing would say so.

Starting a stream over#

Streams → Start over forgets the high-water mark, so the next sync reads that stream in full. It is the escape hatch for what a cursor cannot see: records the API changed without moving their cursor field, or a backfill predating the connection.

Staying connected#

Three things run underneath every connection without being asked for. All are tunable by whoever runs the instance — see self-hosting.

The catalog#

Query data on any dataset opens it in the workbench with the table loaded.

SQL workbench#

Write SQL against anything connected. Results can be charted, added to a dashboard, or exported to CSV/Excel.

-- Monthly revenue and order count for the last 12 months
SELECT date_trunc('month', o.created_at) AS month,
       COUNT(*)                          AS orders,
       SUM(o.amount)                     AS revenue
FROM   orders o
WHERE  o.status = 'settled'
  AND  o.created_at >= current_date - INTERVAL '12 months'
GROUP  BY 1
ORDER  BY 1;

• Ask in English — describe the question and the assistant writes the SQL. Read it before running: it shows you exactly which join and filter the answer depends on.
• Results are capped for display, with the true match count shown, so a runaway query cannot hang the page.

Which engine runs your SQL#

Queries against a connected warehouse run on that warehouse. Queries against uploaded tables and prepared datasets run on DuckDB — in your browser for the workbench and Ask AI, and on the server for scheduled refreshes, prep flows and agent tools.

The first query in a session is slower#

The engine is WebAssembly — roughly 8 MB, fetched once and then cached by your browser for every later visit. You will see “Starting the SQL engine…” with a progress bar while that happens.

• It starts as soon as you open a data page, not when you press Run, so it usually finishes while you are still choosing a table.
• It downloads once per browser, not per query, per dataset or per tab.
• It is served from your own deployment, never a CDN — so an air-gapped install works, and no third party sees that you loaded it.

Giving an agent access#

1. 1

   Open the agent → Tools

   Enable sql_query. Also enable calculator — see the warning below.
2. 2

   Set Allowed tables

   Only tables in this list are visible to the agent. Leaving it empty means sql_query has nothing to query.
3. 3

   Mention the data in the system prompt

   e.g. "Use sql_query for anything involving counts, totals or dates. Never estimate a number you could compute."
4. 4

   Test with a counting question

   Ask for a total. Check the sources under the answer name the table — if they name a document instead, it answered from prose.

At run time the agent sees each table's name, columns and a small sample of rows, writes a SELECT, and gets real rows back.

Troubleshooting#

Use cases#

The production warehouse, handed to the team#

1. 1

   Integrations → Data Sources → new connection

   Name it, pick the provider, enter a read-only login, test. The credential is encrypted at rest under PROVIDER_CREDS_SECRET and never shown again.
2. 2

   Admin → IAM → Access → share it with a group

   The connection runs as its owner; grantees query without ever holding the credential. Rows returned to an agent are capped, so a runaway SELECT * cannot flood a context window.

Catalog an Azure container#

1. 1

   Data catalog → add a source → Azure Blob Storage / ADLS Gen2

   Container, storage account name, and an account key or SAS token.
2. 2

   Read in place, or mount

   Files are read with DuckDB over az://; mount the container into the lakehouse as a read-only source when agents should query it with SQL.

Ask an agent about Jira, or Zendesk#

1. 1

   Integrations → Apps → Jira

   Site URL, account email, API token, optional project keys. Each project becomes a stream (issues:KEY) that syncs into a local table on a schedule. Zendesk works the same way and exposes tickets, users and organizations.
2. 2

   Give an agent the synced tables as a source

   Which open bugs in PROJ are older than thirty days? The answer comes from your copy of the data, on your schedule, with the same provenance as any other read.