> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Setup

> Prerequisites, authentication options, and step-by-step instructions for connecting Databricks to Prizm.

<script type="application/ld+json">
  {`{
            "@context": "https://schema.org",
            "@type": "TechArticle",
            "headline": "Databricks Setup",
            "description": "Prerequisites, authentication options, and step-by-step instructions for connecting Databricks to Prizm.",
            "url": "https://docs.dqlabs.ai/sources/databricks/setup",
            "publisher": {
              "@type": "Organization",
              "name": "DQLabs Inc",
              "logo": "https://media.brand.dev/332adc35-5bc4-4d2b-bf78-256aa4a5e414.svg"
            }
            }`}
</script>

## Prerequisites

Before connecting, create a Databricks Service Principal and grant it the minimum required permissions on your Unity Catalog workspace.

### Create a Service Principal (OAuth — Recommended)

1. In the Databricks account console, navigate to **Settings → Identity and Access → Service Principals**.
2. Click **Add service principal** and give it a name (e.g., `prizm-connector`).
3. Open the service principal, go to **Secrets**, and click **Generate Secret**. Secrets are valid for up to 730 days; a maximum of 5 secrets can exist per service principal.
4. Copy the **Client ID** (Application ID) and **Secret** — the secret is shown only once.
5. In your workspace, assign the service principal with **CAN USE** permission on the target SQL Warehouse or cluster.

### Required Permissions

Grant the service principal the following permissions. Run these SQL statements in a Databricks SQL Editor or notebook:

```sql theme={null}
-- Grant catalog and schema access
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_name>`;
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_name>`;

-- Grant read access to information schema (metadata discovery)
GRANT SELECT ON TABLE <catalog_name>.information_schema.tables TO `<service_principal_name>`;
GRANT SELECT ON TABLE <catalog_name>.information_schema.columns TO `<service_principal_name>`;
GRANT SELECT ON TABLE <catalog_name>.information_schema.schemata TO `<service_principal_name>`;

-- Grant access to system tables (lineage, performance, billing)
GRANT SELECT ON TABLE system.access.table_lineage TO `<service_principal_name>`;
GRANT SELECT ON TABLE system.access.column_lineage TO `<service_principal_name>`;
GRANT SELECT ON TABLE system.lakeflow.job_runs TO `<service_principal_name>`;
GRANT SELECT ON TABLE system.query.history TO `<service_principal_name>`;
GRANT SELECT ON TABLE system.billing.usage TO `<service_principal_name>`;
```

<Warning>
  System table access (`system.access.*`, `system.lakeflow.*`, `system.query.history`, `system.billing.usage`) is required for lineage, pipeline observability, query performance, and cost metrics. Without these grants, those features will be unavailable.
</Warning>

### IP Whitelisting

If your organization uses a whitelist to manage Databricks access, Prizm will only access your Databricks workspace through the following IP addresses. Add these to your Databricks network policy before connecting.

| Cloud Platform | Region               | IP Addresses |
| :------------- | :------------------- | :----------- |
| AWS            | US-EAST-1 (Virginia) | `xxxx`       |

### Permission Summary

| **Permission**       | **Scope**                      | **Required For**                 |
| :------------------- | :----------------------------- | :------------------------------- |
| USE CATALOG          | All target catalogs            | Metadata discovery, all jobs     |
| USE SCHEMA           | All target schemas             | Schema, table, column extraction |
| SELECT               | `information_schema.*`         | Metadata queries                 |
| CAN USE              | SQL Warehouse / Cluster        | JDBC/SQL connection              |
| SELECT               | `system.access.table_lineage`  | Table-level lineage              |
| SELECT               | `system.access.column_lineage` | Column-level lineage             |
| SELECT               | `system.lakeflow.job_runs`     | Pipeline/Job observability       |
| SELECT               | `system.query.history`         | Query performance metrics        |
| SELECT               | `system.billing.usage`         | Cost & usage data                |
| ALTER TABLE SET TAGS | Target tables *(optional)*     | Bi-directional tag sync only     |

## Connector Setup

<Steps>
  <Step title="Open the connector wizard">
    Navigate to  **Source** and click **Add**. Select **Databricks** from the source type grid.

    | **Field**       | **Description**                                                                        |
    | --------------- | -------------------------------------------------------------------------------------- |
    | Connection Name | A unique label for this instance (e.g., `databricks-prod`)                             |
    | Description     | Optional free-text description                                                         |
    | Workspace URL   | Your Databricks workspace URL (e.g., `https://dbc-abc12345-d678.cloud.databricks.com`) |
    | HTTP Path       | SQL Warehouse HTTP path: `/sql/1.0/warehouses/<warehouse_id>`                          |
    | Default Catalog | The Unity Catalog name to scope queries                                                |

    <Frame>
      <img src="https://mintcdn.com/dqlabs/u1ZONzcXSEpXKT91/images/databricks_connection-1.png?fit=max&auto=format&n=u1ZONzcXSEpXKT91&q=85&s=bbbdb4d8156ce41bc1931d5f13401df1" alt="Databricks Connection 1" width="2831" height="1870" data-path="images/databricks_connection-1.png" />
    </Frame>
  </Step>

  <Step title="Choose an authentication method">
    Select an authentication type from the dropdown and provide the required credentials.

    <Tabs>
      <Tab title="Token">
        <Warning>
          Token authentication uses a Databricks Personal Access Token (PAT) — a long-lived credential. Use this only when OAuth-based authentication is not available, as long-lived tokens carry a higher exposure risk if compromised.
        </Warning>

        Select **Token** from the Authentication Type dropdown.

        | **Field** | **Description**                                                                                                               |
        | --------- | ----------------------------------------------------------------------------------------------------------------------------- |
        | Token     | A Databricks Personal Access Token — generate from **User Settings → Developer → Access Tokens** in your Databricks workspace |
      </Tab>

      <Tab title="OAuth (M2M)">
        <Tip>
          OAuth (M2M) tokens are short-lived and auto-refreshed by Prizm — no manual re-authentication required. This is the recommended authentication method for production environments.
        </Tip>

        Select **OAuth (M2M)** from the Authentication Type dropdown.

        | **Field**     | **Description**                                                                                                       |
        | ------------- | --------------------------------------------------------------------------------------------------------------------- |
        | Client ID     | Service Principal Application (Client) ID — found in the Service Principal settings in the Databricks account console |
        | Client Secret | OAuth secret generated under the Service Principal → Secrets — stored encrypted in Vault, never shown after save      |

        Prizm automatically refreshes OAuth tokens before expiry. No manual re-authentication is required.
      </Tab>

      <Tab title="OAuth (Microsoft Entra ID)">
        <Tip>
          Use OAuth (Microsoft Entra ID) when your Databricks workspace is backed by Azure Active Directory / Microsoft Entra ID. This is the recommended authentication method for Azure-hosted Databricks workspaces.
        </Tip>

        Select **OAuth (Microsoft Entra ID)** from the Authentication Type dropdown.

        | **Field**     | **Description**                                                                                                  |
        | ------------- | ---------------------------------------------------------------------------------------------------------------- |
        | Tenant ID     | Your Microsoft Entra ID (Azure AD) Tenant ID — found in the Azure portal under **Microsoft Entra ID → Overview** |
        | Client ID     | Application (Client) ID of the registered Azure AD app                                                           |
        | Client Secret | Client secret from the Azure AD app registration — stored encrypted in Vault, never shown after save             |
      </Tab>
    </Tabs>
  </Step>

  <Step title="Configure storage (optional)">
    The **Additional Configuration** section lets you override the organisation-level external storage settings for this specific Databricks source. By default, Prizm uses the storage provider configured at the organisation level (Settings → Configuration → External Storage) to store exception records, pipeline output data, and profiling artefacts.

    Enable **Overwrite storage** if you need this source to write to a different storage location — for example, if your Databricks workspace is on Azure but your org-level storage is on AWS, you can redirect this source's outputs to an Azure-hosted bucket.

    | **Field**         | **Description**                                                                       |
    | ----------------- | ------------------------------------------------------------------------------------- |
    | Overwrite storage | Toggle on to override the org-level storage configuration for this source only        |
    | Storage type      | Cloud provider for the override storage destination (e.g., AWS, Azure)                |
    | Schema            | Target schema or container name within the storage destination                        |
    | Directory         | Target path or directory within the storage destination where outputs will be written |

    <Note>
      Schema and Directory are required when Overwrite storage is enabled. Leave this section disabled to inherit the organisation-level storage configuration.
    </Note>
  </Step>

  <Step title="Configure pipeline observability">
    The **Advanced Configuration** section controls pipeline and job observability for this Databricks source. These settings determine whether Prizm monitors Databricks Jobs, Workflows, and Delta Live Tables — and how far back it looks when building the initial run history.

    | **Field**                 | **Default** | **Description**                                                                                                                                                                                                                                |
    | ------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | Enable pipelines and jobs | Off         | Turns on pipeline observability for this source. When enabled, Prizm discovers all in-scope Jobs, Workflows, and Delta Live Tables and begins monitoring their run history, execution status, freshness, and data quality expectations.        |
    | No. of Runs (Days)        | 7           | Lookback window in days for the initial run history ingestion. Prizm fetches all pipeline runs from this many days in the past to seed the observability baseline. Increase this value if you need deeper historical context on initial setup. |
    | Enable dashboard          | Off         | Enables the Pipeline Analytics dashboard for this source — providing aggregate health, trend analysis, and incident analytics across all monitored pipelines.                                                                                  |

    <Tip>
      Enable **Enable pipelines and jobs** to unlock the [Pipeline metrics](/connectors/databricks/overview#metric-support) (Execution Status, Freshness, Rows Affected, Execution Time, Failure Rate, Dropped Records, Failed Records, Platform Outage) on your Databricks Job and DLT assets.
    </Tip>
  </Step>

  <Step title="Configure asset scope">
    Use Include and Exclude wildcard patterns to control which catalogs, schemas, and object types are onboarded.

    | **Level**   | **Example Patterns**                                                                                   |
    | ----------- | ------------------------------------------------------------------------------------------------------ |
    | Catalog     | Include: `prod_*`, `analytics` — Exclude: `dev_*`, `sandbox_*`                                         |
    | Schema      | Include: `public`, `reporting` — Exclude: `information_schema`, `system`                               |
    | Object Type | Tables (default on), Views (default on), Delta Live Tables (default off), Jobs/Pipelines (default off) |

    <Frame>
      <img src="https://mintcdn.com/dqlabs/u1ZONzcXSEpXKT91/images/databricks_asset_selection.png?fit=max&auto=format&n=u1ZONzcXSEpXKT91&q=85&s=de7d59784ffffb7564307e5df39ba426" alt="Databricks Asset Selection" width="2831" height="2400" data-path="images/databricks_asset_selection.png" />
    </Frame>

    <Note>
      Exclude rules take precedence over include rules when both match the same object. The `information_schema` and `system` schemas are excluded by default and cannot be included.
    </Note>

    **Data Assets** (selected by default): Catalog/Database, Schema, Table (includes External Tables, Iceberg Tables), View, Columns.

    **Pipeline Assets** (off by default): Delta Live Tables, Jobs & Pipelines, Stored Procedures, Functions.
  </Step>

  <Step title="Configure job schedules">
    | **Job Type**  | **Default Schedule**          | **What It Does**                                              |
    | ------------- | ----------------------------- | ------------------------------------------------------------- |
    | Observability | Hourly                        | Volume, freshness, schema change, anomaly detection on tables |
    | Catalog       | Daily                         | Catalogs, schemas, tables, views, columns, tags, procedures   |
    | Performance   | Daily *(disabled by default)* | Query history, warehouse utilization, billing usage           |
    | Pipeline      | Configurable                  | Delta Live Tables, jobs, and pipeline run history             |

    You can override each schedule from the connector detail page after saving. Schedules for core job types are created automatically when the source is configured. Custom metric schedules can be created manually.

    **How schedule overrides work:** Any updated schedule overrides the existing one and applies to subsequent job executions. If a job is triggered manually, it runs immediately without affecting the configured schedule.

    <Note>
      The Performance Job is **disabled by default** and must be explicitly enabled per connector instance. It requires `SELECT` on `system.query.history` and `system.billing.usage`.
    </Note>
  </Step>

  <Step title="Save and verify">
    Click **Save** to save the connection. Prizm runs an initial Catalog scan immediately — assets appear in the catalog within a few minutes. Navigate to **Settings → Connectors → Logs** to monitor scan progress.

    Click **Test Connection** before saving to validate credentials and permissions. A failed test returns specific details about which permissions are missing.
  </Step>
</Steps>

## Concurrency & Performance Impact

Prizm controls workload concurrency at the job orchestration layer. Higher concurrency settings increase the number of parallel SQL queries against the configured SQL Warehouse. Invalid or inactive sources are excluded from scheduling.

| **Setting**                 | **Default**                                    | **Impact**                                               |
| --------------------------- | ---------------------------------------------- | -------------------------------------------------------- |
| Global concurrent jobs      | 20 (overridable via `MAX_JOBS` env)            | Caps how many Prizm jobs run in parallel platform-wide   |
| Catalog metadata extraction | Batch concurrency 8                            | Parallel `information_schema` queries against Databricks |
| Lineage extraction          | On DDL change detected                         | Per-table, triggered only when `LAST_ALTERED` changes    |
| Performance/billing queries | Per-job-type limits from platform `job_config` | Controls parallel system table queries                   |

## Next Steps

<CardGroup cols={2}>
  <Card title="What We Collect" icon="database" href="/sources/databricks/what-we-collect">
    See the full field-level breakdown of every metadata object Prizm extracts.
  </Card>

  <Card title="FAQ" icon="circle-question" href="/sources/databricks/faq">
    Common questions about credentials, scoping, and permissions.
  </Card>
</CardGroup>