Databricks

Connect a Databricks workspace to Agent Context. Supports three execution modes — Spark Connect, SQL Warehouse, and Delta Lake — so you can pick the right balance of flexibility and performance.

Setup

1. Go to Agent Context in the dashboard
2. Click Add Source → Databricks
3. Enter your workspace endpoint and access token
4. Select an execution mode
5. Specify the table to connect
6. Click Test Connection to verify, then Save

Connection Parameters

Parameter	Required	Description
Endpoint	Yes	Databricks workspace hostname (e.g., dbc-xxxxx.cloud.databricks.com on AWS, adb-xxxxx.azuredatabricks.net on Azure)
Access Token	Yes	Personal access token
Mode	Yes	Execution mode: spark_connect, sql_warehouse, or delta_lake
Table	Yes	Fully qualified table name: catalog.schema.table

Execution Modes

Spark Connect

Connects to a running Databricks cluster via Spark Connect protocol.

Parameter	Required	Description
Cluster ID	Recommended	The cluster to connect to (e.g., 0123-456789-abcdefgh)

Best for: interactive exploration, complex transformations, and workloads that benefit from Spark’s distributed engine.

SQL Warehouse

Connects to a Databricks SQL Warehouse for optimized SQL queries.

Parameter	Required	Description
SQL Warehouse ID	Recommended	The warehouse ID (e.g., 2b66f17c065ef1cd)

Best for: SQL-heavy workloads, dashboards, and queries that need fast startup times. SQL Warehouses auto-scale and auto-suspend.

Delta Lake

Reads Delta Lake tables directly from cloud storage, bypassing the Databricks compute layer. Requires cloud storage credentials (AWS, Azure, or GCS) — contact support to configure these.

Best for: bulk reads, cost-sensitive workloads, and scenarios where you want to avoid Databricks compute costs.

Authentication

Generate a personal access token in Databricks: User Settings → Developer → Access tokens.

Service principal authentication (client ID + secret) and Delta Lake storage backend credentials are supported at the backend level but not yet exposed in the dashboard UI. Contact support if you need these options.

Table Reference Format

Tables use Unity Catalog’s three-part name:

catalog.schema.table

Examples:

• main.default.customers
• prod_catalog.analytics.revenue
• hive_metastore.default.events

What Gets Discovered

When you connect a Databricks table, Agent Context discovers:

• The table’s columns, types, and constraints
• Column-level metadata for schema exploration

Each connected table becomes individually queryable by agents via the context-lake catalog and context-lake query commands.

Troubleshooting

Issue	Fix
Endpoint unreachable	Verify the workspace URL. It should be the full hostname without https:// (e.g., dbc-xxxxx.cloud.databricks.com on AWS, adb-xxxxx.azuredatabricks.net on Azure).
Authentication failed	Verify the access token. Tokens expire — regenerate in User Settings → Developer → Access tokens.
Cluster not running	For Spark Connect mode, the cluster must be running. Start it in the Databricks console.
Warehouse not found	Verify the SQL Warehouse ID. Check that the warehouse exists and your user has access.
Table not found	Verify the three-part name (catalog.schema.table). Check that the user has SELECT permission on the table.
Permission denied	Ensure the access token has permission to read the target table, database, and schema.