Databricks

Databricks

This connector works with a Databricks workspace on AWS, Azure or Google Cloud through the Databricks REST API with a personal access token: the current user, clusters, SQL warehouses, SQL statements on a warehouse (Statement Execution API), jobs and runs, Unity Catalog catalogs, schemas and tables with columns, workspace notebooks, DBFS and Unity Catalog volumes. It acts with the rights of the token's user.

Setup (about 3 minutes)

1. Open your workspace and copy its host from the address bar into DATABRICKS_HOST, without https://: dbc-a1b2c3d4-e5f6.cloud.databricks.com (AWS), adb-1234567890123456.7.azuredatabricks.net (Azure) or 1234567890123456.7.gcp.databricks.com (Google Cloud).
2. Click your user name (top right) → Settings → Developer → Access tokens → Manage → Generate new token. Give it a comment and a lifetime, copy it (shown once) into DATABRICKS_TOKEN. The workspace admin must allow personal access tokens.
3. For least privilege, create the token for a service principal with only the grants the AI needs (USE CATALOG, USE SCHEMA, SELECT on the data, CAN USE on one SQL warehouse).
4. Install and run databricks_get_current_user.

Costs: Databricks bills compute in DBUs. Running SQL on a stopped warehouse starts it automatically, and a running warehouse or cluster costs money until it stops (warehouses auto-stop after auto_stop_mins, clusters after autotermination_minutes). Before databricks_execute_sql, check the warehouse with databricks_list_warehouses: if its state is not RUNNING, tell the user the query will start it (size cluster_size, serverless or not) and ask first. Prefer a serverless warehouse when one exists: it starts in seconds.

Every write needs the user's confirmation first: databricks_start_cluster, databricks_terminate_cluster, databricks_start_warehouse, databricks_stop_warehouse, databricks_run_job, databricks_cancel_job_run and databricks_cancel_statement. Say what will happen (costs, interrupted users or jobs) and wait for a clear yes. Terminating a cluster or stopping a warehouse ends the queries and notebooks running on it. Nothing is deleted by this connector.

SQL statements

• databricks_execute_sql runs one statement on a SQL warehouse: SELECT reads; INSERT, UPDATE, DELETE, MERGE, CREATE, ALTER, DROP, GRANT change data or permissions. Before any statement that is not a read, show the user the exact statement and wait for an explicit yes. The token's Unity Catalog grants decide what is allowed.
• Name tables fully: catalog.schema.table (or pass catalog and schema). Explore with databricks_list_catalogs → databricks_list_schemas → databricks_list_tables → databricks_get_table (columns and types) before writing SQL.
• Use named parameters for values: WHERE country = :country plus parameters: [{"name": "country", "value": "DE", "type": "STRING"}].
• row_limit defaults to 200 rows (set it explicitly, at most 10,000); aggregate in SQL rather than fetching many rows. rows is a list of arrays in the order of columns; all values are strings or null.
• The call waits up to wait_timeout (default 20s). If status.state is PENDING or RUNNING, call databricks_get_statement with the statement_id every few seconds until SUCCEEDED, FAILED or CANCELED. When next_chunk_index is present, more rows exist: databricks_get_statement_chunk with that index. Fetch the rows soon: Databricks keeps a statement's result only for a limited time.
• databricks_cancel_statement stops a running statement (ask first).

Clusters and warehouses: databricks_list_clusters, databricks_get_cluster, databricks_list_warehouses and databricks_get_warehouse are read-only and free; termination_reason explains why a cluster stopped. Spark configuration, environment variables, init scripts and tags are left out because they can contain secrets.

Jobs: databricks_list_jobs and databricks_get_job show jobs, schedules, parameters and tasks. databricks_run_job triggers a run now (it costs compute; pass job_parameters only with the user's values). databricks_list_job_runs and databricks_get_job_run show runs and per-task state; databricks_get_run_output returns a task run's output (notebook exit value, error and logs): for a multi-task job, use the task's run_id from databricks_get_job_run, not the parent run id.

Files: databricks_list_workspace browses folders, notebooks and files (/Users/<email>, /Workspace/Shared); databricks_export_notebook returns a notebook's source code as text (keep to small notebooks; the answer is capped at about 100 KB). databricks_list_volume_files lists a Unity Catalog volume (/Volumes/<catalog>/<schema>/<volume>/...). databricks_list_dbfs lists legacy DBFS paths; DBFS root and mounts are deprecated and disabled on many newer workspaces, so prefer volumes.

Common errors

• 403 Invalid access token or 401: the token is wrong, expired or revoked, or the host belongs to another workspace.
• 403 PERMISSION_DENIED: the user lacks a grant (USE CATALOG, USE SCHEMA, SELECT, CAN USE on the warehouse, CAN MANAGE RUN on the job).
• 404 / RESOURCE_DOES_NOT_EXIST: wrong id or name, or Unity Catalog is not enabled for the workspace.
• 400 INVALID_PARAMETER_VALUE: e.g. a wait_timeout outside 5s to 50s, or a bad table name.
• 429 REQUEST_LIMIT_EXCEEDED: wait and retry.

Out of scope here: creating, editing or deleting clusters, warehouses, jobs, tables or files; uploading files; Delta Live Tables pipelines, MLflow, Model Serving, Genie and Vector Search (Databricks also offers its own managed MCP servers for Genie, Vector Search and Unity Catalog functions).