> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ADLS Setup

> Prerequisites, authentication, Unity Catalog configuration, and step-by-step instructions for connecting ADLS to Prizm — with or without your own Databricks cluster.

<script type="application/ld+json">
  {`{
            "@context": "https://schema.org",
            "@type": "TechArticle",
            "headline": "ADLS Setup",
            "description": "Prerequisites, authentication, Unity Catalog configuration, and step-by-step instructions for connecting ADLS to Prizm - with or without your own Databricks cluster.",
            "url": "https://docs.dqlabs.ai/sources/adls/setup",
            "publisher": {
              "@type": "Organization",
              "name": "DQLabs Inc",
              "logo": "https://media.brand.dev/332adc35-5bc4-4d2b-bf78-256aa4a5e414.svg"
            }
            }`}
</script>

<Note>
  If you're using **Prizm Spark** (the default) rather than your own Databricks cluster, none of the Azure/Unity Catalog/Databricks setup below is required — skip straight to [Step 1: Configure Your ADLS Source](#step-1-configure-your-adls-source).
</Note>

## Prerequisites Checklist

| #  | Prerequisite                                                                                                                                                        | Owner                     |
| :- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------ |
| 1  | ADLS Gen2 storage account with a read-only source container and a separate read-write container for Unity Catalog managed storage                                   | Azure admin               |
| 2  | Authentication method chosen and provisioned (Managed Identity recommended, or Service Principal), including the Storage Blob Delegator role for credential vending | Azure admin               |
| 3  | Unity Catalog metastore enabled on the Databricks workspace                                                                                                         | Databricks admin          |
| 4  | UC Storage Credential created, referencing the authentication method above                                                                                          | Databricks admin          |
| 5  | UC external locations created for both the read (source) and write (managed storage) containers                                                                     | Databricks admin          |
| 6  | Target UC Catalog and Schema created                                                                                                                                | Databricks admin          |
| 7  | Databricks execution identity for Prizm (OAuth service principal) with required grants                                                                              | Databricks admin          |
| 8  | Cluster designated for Prizm job runs                                                                                                                               | Databricks admin          |
| 9  | Prizm notebooks (Iceberg materializer and exception workflow) deployed to agreed workspace paths                                                                    | Prizm, with client review |
| 10 | Credential values submitted to Prizm via secure channel                                                                                                             | Client                    |

## Authentication Setup (Azure)

Prizm needs **read-only** access to the container holding source files, and **read-write** access to a separate container used as Unity Catalog's managed storage location. Keeping these as two containers, each with its own role assignment, means the source container's identity holds no write or delete permission at all.

<Tabs>
  <Tab title="Managed Identity (Recommended)">
    Uses an **Access Connector for Azure Databricks** — a first-party Azure resource exposing a system- or user-assigned managed identity. One connector serves both containers; no secret is ever generated or stored.

    <Steps>
      <Step title="Create the Access Connector">
        In the Azure Portal, create a resource of type **Access Connector for Azure Databricks** in the same region/subscription as your Databricks workspace. Copy its Resource ID from the Overview page.
      </Step>

      <Step title="Grant the connector access to both containers">
        | Scope                                        | Role                          |
        | :------------------------------------------- | :---------------------------- |
        | Read (source) container                      | Storage Blob Data Reader      |
        | Write (managed storage) container            | Storage Blob Data Contributor |
        | Storage account (root, not either container) | Storage Blob Delegator        |

        <Warning>
          Storage Blob Delegator must be assigned at the **storage-account** scope, not the container scope — Azure evaluates it against the account's blob service endpoint. It only permits signing a delegation token; it grants no data access beyond whatever role (Reader/Contributor) the identity separately holds per container.
        </Warning>
      </Step>

      <Step title="Create the storage credential in Databricks">
        In Catalog Explorer: **+ Add → Create a credential**, Credential Type **Azure Managed Identity**, paste the Access Connector Resource ID. This one credential backs both external locations created next.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Service Principal">
    Use only where a managed identity is not permitted by policy. Same container split applies.

    <Steps>
      <Step title="Register the service principal">
        In Microsoft Entra ID, register an application, generate a Client Secret (recorded immediately — shown only once), and note the Application (client) ID and Directory (tenant) ID.
      </Step>

      <Step title="Grant roles per container">
        Storage Blob Data Reader on the read container, Storage Blob Data Contributor on the write container, and Storage Blob Delegator at the storage-account level — all to this service principal.
      </Step>

      <Step title="Create the storage credential in Databricks">
        Credential Type **Azure Service Principal**; enter the Application (client) ID, Directory (tenant) ID, and Client Secret.
      </Step>
    </Steps>

    <Warning>
      Service principal secrets expire (typically every 6–24 months) and must be rotated before expiry. Store in Azure Key Vault, never in a notebook or config file. Treat any secret ever shared in plaintext as compromised and rotate immediately.
    </Warning>
  </Tab>
</Tabs>

## Unity Catalog Setup

### External locations

Create **two** external locations from the same storage credential — one per container:

```text theme={null}
-- Source (read)
abfss://<read-container>@<storage-account>.dfs.core.windows.net/

-- Managed storage (write)
abfss://<write-container>@<storage-account>.dfs.core.windows.net/
```

Grant the Prizm execution identity access on each:

```sql theme={null}
GRANT READ FILES ON EXTERNAL LOCATION <source_location_name> TO `<service_principal_application_id>`;
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION <write_location_name> TO `<service_principal_application_id>`;
```

<Note>
  For ADLS + Databricks Unity Catalog clients, exception and metadata instruction JSON is staged under the write external location (`abfss://…/staging/{run_id}/…`) rather than uploaded to Prizm SaaS S3 or via a scheduled Azure Blob upload as in earlier releases. The `WRITE FILES` grant above already covers this — no separate grant is required. Open-network clients are unaffected and keep the existing S3 HTTPS path.
</Note>

### Catalog and schema

```sql theme={null}
CREATE CATALOG IF NOT EXISTS <catalog_name>
  MANAGED LOCATION 'abfss://<write-container>@<storage-account>.dfs.core.windows.net/<managed-subpath>';

CREATE SCHEMA IF NOT EXISTS <catalog_name>.<schema_name>;

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_application_id>`;
GRANT USE SCHEMA, CREATE TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_application_id>`;
```

If your metastore already has default managed storage configured at the account level, the `MANAGED LOCATION` clause can be omitted.

### Trino / external engine access

Because tables are written with Delta UniForm (Iceberg compatibility), any Iceberg REST catalog client — Trino, external Spark, Snowflake, DuckDB — can read them once pointed at your metastore's Iceberg REST endpoint (`/api/2.1/unity-catalog/iceberg-rest`). This is governed independently of the grants above and requires two extra steps:

1. **Enable external data access on the metastore** — a one-time, metastore-level setting a metastore admin turns on (Catalog → gear icon → metastore settings), not a SQL statement.
2. **Grant `EXTERNAL USE SCHEMA`** to whichever identity Trino authenticates as:
   ```sql theme={null}
   GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<trino-principal>`;
   ```

<Note>
  This external-access pathway is currently in **Public Preview** on Databricks — confirm with your Databricks account team that it's enabled if you haven't used it before. For Trino catalog authentication, only client ID/secret/tenant ID are provided, not a pre-generated token; Trino requests its own token per call, so there's nothing to rotate by hand.
</Note>

### Consolidated grant script

All grants above, in one pass (substitute your real values for every placeholder):

```sql theme={null}
-- 1. Catalog & schema metadata grants
GRANT USE CATALOG ON CATALOG <CATALOG_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;
GRANT USE SCHEMA ON SCHEMA <CATALOG_NAME>.<SCHEMA_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;
GRANT EXTERNAL USE SCHEMA ON SCHEMA <CATALOG_NAME>.<SCHEMA_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;
GRANT SELECT ON SCHEMA <CATALOG_NAME>.<SCHEMA_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;
GRANT CREATE TABLE ON SCHEMA <CATALOG_NAME>.<SCHEMA_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;

-- 2. Storage credential grant
GRANT READ FILES ON STORAGE CREDENTIAL `<STORAGE_CREDENTIAL_NAME>` TO `<SERVICE_PRINCIPAL_APP_ID>`;

-- 3. Read grant per source external location (repeat for each additional read container)
GRANT READ FILES ON EXTERNAL LOCATION `<READ_EXTERNAL_LOCATION_NAME_1>` TO `<SERVICE_PRINCIPAL_APP_ID>`;

-- 4. Managed storage (write) external location grant — also covers exception-JSON staging
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION `<WRITE_EXTERNAL_LOCATION_NAME>` TO `<SERVICE_PRINCIPAL_APP_ID>`;
```

## Databricks Workspace Access

### Execution identity

Prizm authenticates using an **OAuth service principal** — a Databricks service principal with a client ID and OAuth secret, scoped only to what Prizm needs. Create it under Account Console → User management → Service principals, generate a secret on its Secrets tab, and store both values in a secrets manager.

### Notebook paths

Two notebooks are deployed to workspace paths your team agrees on:

| Notebook                       | Purpose                                                                                                                                                                                               |
| :----------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `adls_uc_iceberg_materializer` | Reads each onboarded source file and creates the corresponding Iceberg-compatible Delta table                                                                                                         |
| `adls_uc_exception_workflow`   | Writes exception records to a dedicated exception table (created once, reused every run). Also runs the `STAGE_EXCEPTION_JSON` job type for exception/metadata JSON staging — not a separate notebook |

The execution identity needs `CAN_RUN` (minimum) or `CAN_MANAGE` permission on both paths.

### Compute

| Requirement        | Minimum                                               | Recommended                             |
| :----------------- | :---------------------------------------------------- | :-------------------------------------- |
| Databricks Runtime | 14.3 LTS (must support Unity Catalog + Delta UniForm) | 16.4 LTS                                |
| Node Type          | Standard\_D4s\_v3 / m4.xlarge                         | Standard\_D4s\_v3 / m4.xlarge or higher |
| Min / Max Workers  | 2 / 16                                                | 2 / 16                                  |
| Memory per Node    | 8 GB                                                  | 16 GB                                   |
| Cores per Node     | 2                                                     | 4                                       |

Provide either an existing all-purpose/job cluster ID, or permission for Prizm to launch a job cluster automatically with an agreed instance type, DBR version, and pool/AZ constraints.

<Note>
  Approximate load handling: a 100 MB / \~1M row file scans in 1–3 minutes; a 10 GB / \~100M row file takes roughly 30–90 minutes, depending on worker count. Actual timings vary with file format, column count, and configured checks.
</Note>

## Submitting Credentials to Prizm

Return credentials through a secure channel only — never by email, chat, or in a shared notebook cell.

**Accepted delivery methods:** Azure Key Vault (preferred — grant Prizm's service principal Get/List access on the specific secrets, share only the vault URI and secret names), Prizm's secure credential intake form, or a password-protected secrets-manager entry with the password relayed out-of-band.

<Warning>
  Treat any credential that has ever been pasted into a notebook cell, chat message, or shared document as compromised — rotate it before use, even if submitting through Key Vault going forward.
</Warning>

## Setting Up the Connection

Steps 1–3 happen in the Prizm portal; Steps 4–5 happen in Databricks.

<Steps>
  <Step title="Configure Your ADLS Source">
    Click **+ Add Source**, select **Azure Data Lake Storage**, and complete:

    | Field               | Required | Notes                                                                                                                                           |
    | :------------------ | :------- | :---------------------------------------------------------------------------------------------------------------------------------------------- |
    | Connection Name     | Required | e.g. `adls-finance-prod`                                                                                                                        |
    | Description         | Optional |                                                                                                                                                 |
    | Tenant ID           | Required | Your Azure AD tenant ID                                                                                                                         |
    | Authentication Type | Required | **Service Principle** (+ Storage Account Name, Client ID, Client Secret) or **Storage Access Key** (+ Storage Account Name, Storage Access Key) |
    | Use Vault           | Optional | Pull credentials from a configured vault instead of entering directly                                                                           |
    | File Type           | Required | csv, parquet, or delta                                                                                                                          |

    Prizm validates credentials on save and uses them both to read files and to list containers in the account. If no containers appear, confirm list access on the storage account.
  </Step>

  <Step title="Choose Your Compute Engine">
    | Option                           | Result                                |
    | :------------------------------- | :------------------------------------ |
    | **Prizm Spark** (default)        | No further setup — skip to Monitoring |
    | **Use Databricks Spark Cluster** | Continue to Step 3                    |

    <Warning>
      Keep your Databricks cluster running during setup — a stopped cluster will not appear in the dropdown.
    </Warning>
  </Step>

  <Step title="Connect Your Databricks Cluster">
    | Field                       | Required | Notes                                   |
    | :-------------------------- | :------- | :-------------------------------------- |
    | Platform Type               | Required | Databricks Spark                        |
    | Databricks Server           | Required | Your workspace URL                      |
    | Catalog Name                | Required | The Unity Catalog catalog created above |
    | Schema Name                 | Required | The Unity Catalog schema created above  |
    | Use Vault                   | Optional | Pull Client ID/Secret from vault        |
    | Client ID and Client Secret | Required | The Databricks execution identity       |
    | Notebook Path               | Required | Path to the generic notebook (Step 4)   |
    | Spark Cluster               | Required | Selected from dropdown                  |

    **Scope the crawl** with Containers to Include, Root Directory Path, and Include/Exclude Folder Patterns (glob-style, e.g. `gold/*`, `*/_staging/*`). Excludes apply after includes are resolved.

    <Note>
      The container dropdown lists every container in the account, not just ones your credentials can read — Prizm can't filter this for a service principal or access token. Selecting an unreadable container shows a toast and auto-removes it from your selection.
    </Note>

    **Crawl limits:** Partition Depth (how many subfolder levels to scan; 0 = root only), Max Files Per Directory, and an Incremental Mode toggle (scan only files changed since the last crawl).

    **Delta Lake tables:** when a folder contains a `_delta_log`, Prizm's technical catalog step identifies it as a Delta table and catalogs the whole table as one asset (named after the table folder) instead of one asset per part file — reading it via Spark's Delta format at the table root rather than as standalone Parquet. Updates trigger incremental rematerialization and metric refresh the same way as other ADLS assets. Plain Parquet folders without a `_delta_log` are unaffected.

    **External storage details (Iceberg tables):** if Iceberg tables in this source live in a *different* storage account than the one configured above, toggle **Use External Storage Details** and, if needed, **Override External Storage Details** to enter separate Storage Provider, Tenant ID, Client ID/Secret (or Storage Account Key/SAS Token), Storage Account Name, Resource Group, Container, and Directory Path. This is separate from the Unity Catalog materialization flow — it governs where Prizm *reads* existing Iceberg tables, not where it *writes* new ones.

    **Advanced options:** define per-asset file-matching rules (Asset Name, File Path, File Pattern) layered on top of Include/Exclude.
  </Step>

  <Step title="Set Up the Generic Notebook">
    A single generic notebook is used for quality-check jobs (separate from the two Unity Catalog notebooks above — all three run independently).

    1. Import the notebook file (provided by your Prizm account team) into your Databricks workspace via Workspace → right-click → Import.
    2. Generate an API token: Prizm platform → your organization → Tokens → Create Token.
    3. Get your MCP host value from your Prizm account team.
    4. Add the token and MCP host to the notebook's designated fields — or, better, store them as Databricks secrets:
       ```bash theme={null}
       databricks secrets create-scope "prizm_notebook_secrets"
       databricks secrets put-secret --json '{"scope": "prizm_notebook_secrets", "key": "access-token", "string_value": "<your-api-token>"}'
       databricks secrets put-secret --json '{"scope": "prizm_notebook_secrets", "key": "mcp-host", "string_value": "<your-mcp-host-value>"}'
       ```
       and reference them instead of pasting values directly: `dbutils.secrets.get(scope="prizm_notebook_secrets", key="access-token")`.
    5. Save the notebook to the path entered in Step 3.

    <Warning>
      Pasting the API token and MCP host directly into the notebook stores them in plain text — treat the notebook as a credential and restrict access accordingly.
    </Warning>
  </Step>

  <Step title="Service Principal & Databricks CLI Configuration">
    ```bash theme={null}
    # Create Databricks secret scope
    databricks secrets create-scope <SECRET_SCOPE_NAME> -p <DATABRICKS_PROFILE>

    # Store Service Principal Client ID and Secret
    databricks secrets put-secret <SECRET_SCOPE_NAME> adls-client-id -p <DATABRICKS_PROFILE>
    databricks secrets put-secret <SECRET_SCOPE_NAME> adls-client-secret -p <DATABRICKS_PROFILE>

    # Set ACL permission for the Service Principal
    databricks secrets put-acl \
      <SECRET_SCOPE_NAME> \
      "<SERVICE_PRINCIPAL_APP_ID>" \
      READ \
      -p <DATABRICKS_PROFILE>
    ```
  </Step>
</Steps>

## Monitoring Your Jobs

Every job Prizm triggers appears in Databricks under **Jobs & Pipelines**, named by run date, a static keyword, and the source file name. Job status and logs behave like any other Databricks job. Once a job finishes, results return to Prizm automatically via webhook — no manual step required.

## Pre-Go-Live Validation Checklist

Before the first production run of Iceberg materialization, confirm from within Databricks:

* `SHOW STORAGE CREDENTIALS;` lists the credential from setup
* `SHOW EXTERNAL LOCATIONS;` lists both locations with the correct URLs
* `DESCRIBE EXTERNAL LOCATION <name>;` plus **Test connection** on each confirms read / read-write access
* `SHOW SCHEMAS IN <catalog_name>;` confirms the target catalog/schema exist
* `SHOW GRANTS ON CATALOG <catalog_name>;`, `SHOW GRANTS ON SCHEMA ...;`, `SHOW GRANTS ON EXTERNAL LOCATION ...;` confirm the execution identity's grants
* The execution identity can list/attach to the designated cluster (or launch a job cluster per the agreed policy)
* A manual test run of a small known file succeeds: the table appears under `<catalog>.<schema>`, `DESCRIBE DETAIL` shows Iceberg-compatible properties, and the webhook payload reaches Prizm with a success status
* External data access is enabled on the metastore, the Trino principal has `EXTERNAL USE SCHEMA`, and Trino can list/query the table via the Iceberg REST endpoint

Once confirmed, notify your Prizm implementation contact to schedule production onboarding.

## Next Steps

<CardGroup cols={2}>
  <Card title="What We Collect" icon="database" href="/sources/adls/what-we-collect">
    Job types, compute metric types, and Delta Lake table handling.
  </Card>

  <Card title="FAQ" icon="circle-question" href="/sources/adls/faq">
    Common questions and troubleshooting.
  </Card>
</CardGroup>