ADLS Setup

If you’re using Prizm Spark (the default) rather than your own Databricks cluster, none of the Azure/Unity Catalog/Databricks setup below is required — skip straight to Step 1: Configure Your ADLS Source.

Prerequisites Checklist

#	Prerequisite	Owner
1	ADLS Gen2 storage account with a read-only source container and a separate read-write container for Unity Catalog managed storage	Azure admin
2	Authentication method chosen and provisioned (Managed Identity recommended, or Service Principal), including the Storage Blob Delegator role for credential vending	Azure admin
3	Unity Catalog metastore enabled on the Databricks workspace	Databricks admin
4	UC Storage Credential created, referencing the authentication method above	Databricks admin
5	UC external locations created for both the read (source) and write (managed storage) containers	Databricks admin
6	Target UC Catalog and Schema created	Databricks admin
7	Databricks execution identity for Prizm (OAuth service principal) with required grants	Databricks admin
8	Cluster designated for Prizm job runs	Databricks admin
9	Prizm notebooks (Iceberg materializer and exception workflow) deployed to agreed workspace paths	Prizm, with client review
10	Credential values submitted to Prizm via secure channel	Client

Authentication Setup (Azure)

Prizm needs read-only access to the container holding source files, and read-write access to a separate container used as Unity Catalog’s managed storage location. Keeping these as two containers, each with its own role assignment, means the source container’s identity holds no write or delete permission at all.

Managed Identity (Recommended)
Service Principal

Uses an Access Connector for Azure Databricks — a first-party Azure resource exposing a system- or user-assigned managed identity. One connector serves both containers; no secret is ever generated or stored.

Create the Access Connector

In the Azure Portal, create a resource of type Access Connector for Azure Databricks in the same region/subscription as your Databricks workspace. Copy its Resource ID from the Overview page.

Grant the connector access to both containers

Scope	Role
Read (source) container	Storage Blob Data Reader
Write (managed storage) container	Storage Blob Data Contributor
Storage account (root, not either container)	Storage Blob Delegator

Storage Blob Delegator must be assigned at the storage-account scope, not the container scope — Azure evaluates it against the account’s blob service endpoint. It only permits signing a delegation token; it grants no data access beyond whatever role (Reader/Contributor) the identity separately holds per container.

Create the storage credential in Databricks

In Catalog Explorer: + Add → Create a credential, Credential Type Azure Managed Identity, paste the Access Connector Resource ID. This one credential backs both external locations created next.

Unity Catalog Setup

External locations

Create two external locations from the same storage credential — one per container:

-- Source (read)
abfss://<read-container>@<storage-account>.dfs.core.windows.net/

-- Managed storage (write)
abfss://<write-container>@<storage-account>.dfs.core.windows.net/

Grant the Prizm execution identity access on each:

GRANT READ FILES ON EXTERNAL LOCATION <source_location_name> TO `<service_principal_application_id>`;
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION <write_location_name> TO `<service_principal_application_id>`;

For ADLS + Databricks Unity Catalog clients, exception and metadata instruction JSON is staged under the write external location (abfss://…/staging/{run_id}/…) rather than uploaded to Prizm SaaS S3 or via a scheduled Azure Blob upload as in earlier releases. The WRITE FILES grant above already covers this — no separate grant is required. Open-network clients are unaffected and keep the existing S3 HTTPS path.

Exception Staging via a Unity Catalog Volume

As an alternative to staging exception/metadata instruction JSON directly under the write external location, you can stage it in a dedicated Unity Catalog Volume. Choose one of the two options below depending on whether you want the volume backed by a specific path on your existing ADLS container or by Unity Catalog’s own managed storage. Option A: External volume on the existing ADLS path

CREATE SCHEMA IF NOT EXISTS <catalog>.<schema>;
CREATE EXTERNAL VOLUME IF NOT EXISTS <catalog>.<schema>.exception_staging
LOCATION 'abfss://<container>@<account>.dfs.core.windows.net/<path>/staging'
COMMENT 'Staging for exception writeback instruction JSON';

Option B: Managed volume

CREATE VOLUME IF NOT EXISTS <catalog>.<schema>.exception_staging;

Grants on the volume This approach uses two separate identities — the service that uploads the exception JSON, and the identity the notebook job runs as — so each gets only the access it needs:

-- Identity that uploads the JSON via the Files API (your Ray service's SP)
GRANT USE CATALOG ON CATALOG <catalog> TO `<sp-application-id>`;
GRANT USE SCHEMA  ON SCHEMA  <catalog>.<schema> TO `<sp-application-id>`;
GRANT WRITE VOLUME ON VOLUME <catalog>.<schema>.exception_staging TO `<sp-application-id>`;

-- Identity the job runs as (the notebook reads the file)
GRANT USE CATALOG ON CATALOG <catalog> TO `<job-run-as-principal>`;
GRANT USE SCHEMA  ON SCHEMA  <catalog>.<schema> TO `<job-run-as-principal>`;
GRANT READ VOLUME ON VOLUME <catalog>.<schema>.exception_staging TO `<job-run-as-principal>`;

Catalog and schema

CREATE CATALOG IF NOT EXISTS <catalog_name>
  MANAGED LOCATION 'abfss://<write-container>@<storage-account>.dfs.core.windows.net/<managed-subpath>';

CREATE SCHEMA IF NOT EXISTS <catalog_name>.<schema_name>;

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_application_id>`;
GRANT USE SCHEMA, CREATE TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_application_id>`;

If your metastore already has default managed storage configured at the account level, the MANAGED LOCATION clause can be omitted.

Trino / external engine access

Because tables are written with Delta UniForm (Iceberg compatibility), any Iceberg REST catalog client — Trino, external Spark, Snowflake, DuckDB — can read them once pointed at your metastore’s Iceberg REST endpoint (/api/2.1/unity-catalog/iceberg-rest). This is governed independently of the grants above and requires two extra steps:

Enable external data access on the metastore — a one-time, metastore-level setting a metastore admin turns on (Catalog → gear icon → metastore settings), not a SQL statement.

Grant EXTERNAL USE SCHEMA to whichever identity Trino authenticates as:

GRANT EXTERNAL USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<trino-principal>`;

This external-access pathway is currently in Public Preview on Databricks — confirm with your Databricks account team that it’s enabled if you haven’t used it before. For Trino catalog authentication, only client ID/secret/tenant ID are provided, not a pre-generated token; Trino requests its own token per call, so there’s nothing to rotate by hand.

Consolidated grant script

All grants above, in one pass (substitute your real values for every placeholder):

-- 1. Catalog & schema metadata grants
GRANT USE CATALOG ON CATALOG <CATALOG_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;
GRANT USE SCHEMA ON SCHEMA <CATALOG_NAME>.<SCHEMA_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;
GRANT EXTERNAL USE SCHEMA ON SCHEMA <CATALOG_NAME>.<SCHEMA_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;
GRANT SELECT ON SCHEMA <CATALOG_NAME>.<SCHEMA_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;
GRANT CREATE TABLE ON SCHEMA <CATALOG_NAME>.<SCHEMA_NAME> TO `<SERVICE_PRINCIPAL_APP_ID>`;

-- 2. Storage credential grant
GRANT READ FILES ON STORAGE CREDENTIAL `<STORAGE_CREDENTIAL_NAME>` TO `<SERVICE_PRINCIPAL_APP_ID>`;

-- 3. Read grant per source external location (repeat for each additional read container)
GRANT READ FILES ON EXTERNAL LOCATION `<READ_EXTERNAL_LOCATION_NAME_1>` TO `<SERVICE_PRINCIPAL_APP_ID>`;

-- 4. Managed storage (write) external location grant — also covers exception-JSON staging
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION `<WRITE_EXTERNAL_LOCATION_NAME>` TO `<SERVICE_PRINCIPAL_APP_ID>`;

Databricks Workspace Access

Execution identity

Prizm authenticates using an OAuth service principal — a Databricks service principal with a client ID and OAuth secret, scoped only to what Prizm needs. Create it under Account Console → User management → Service principals, generate a secret on its Secrets tab, and store both values in a secrets manager.

Notebook paths

Two notebooks are deployed to workspace paths your team agrees on:

Notebook	Purpose
`adls_uc_iceberg_materializer`	Reads each onboarded source file and creates the corresponding Iceberg-compatible Delta table
`adls_uc_exception_workflow`	Writes exception records to a dedicated exception table (created once, reused every run). Also runs the `STAGE_EXCEPTION_JSON` job type for exception/metadata JSON staging — not a separate notebook

The execution identity needs CAN_RUN (minimum) or CAN_MANAGE permission on both paths.

Compute

Requirement	Minimum	Recommended
Databricks Runtime	14.3 LTS (must support Unity Catalog + Delta UniForm)	16.4 LTS
Node Type	Standard_D4s_v3 / m4.xlarge	Standard_D4s_v3 / m4.xlarge or higher
Min / Max Workers	2 / 16	2 / 16
Memory per Node	8 GB	16 GB
Cores per Node	2	4

Provide either an existing all-purpose/job cluster ID, or permission for Prizm to launch a job cluster automatically with an agreed instance type, DBR version, and pool/AZ constraints.

Approximate load handling: a 100 MB / ~1M row file scans in 1–3 minutes; a 10 GB / ~100M row file takes roughly 30–90 minutes, depending on worker count. Actual timings vary with file format, column count, and configured checks.

Submitting Credentials to Prizm

Return credentials through a secure channel only — never by email, chat, or in a shared notebook cell. Accepted delivery methods: Azure Key Vault (preferred — grant Prizm’s service principal Get/List access on the specific secrets, share only the vault URI and secret names), Prizm’s secure credential intake form, or a password-protected secrets-manager entry with the password relayed out-of-band.

Treat any credential that has ever been pasted into a notebook cell, chat message, or shared document as compromised — rotate it before use, even if submitting through Key Vault going forward.

Setting Up the Connection

Steps 1–3 happen in the Prizm portal; Steps 4–5 happen in Databricks.

Configure Your ADLS Source

On the Source page, click the + icon next to the source list, select Azure Data Lake Storage, and complete:

Field	Required	Notes
Connection Name	Required	e.g. `adls-finance-prod`
Description	Optional
Tenant ID	Required	Your Azure AD tenant ID
Authentication Type	Required	Service Principal (+ Storage Account Name, Client ID, Client Secret) or Storage Access Key (+ Storage Account Name, Storage Access Key)
Use Vault	Optional	Pull credentials from a configured vault instead of entering directly
File Type	Required	csv, parquet, or delta

Prizm validates credentials on save and uses them both to read files and to list containers in the account. If no containers appear, confirm list access on the storage account.

The Azure Data Lake Storage Source Details form — ADLS source details. Service Principal is the default authentication type

The Azure Data Lake Storage credential fields and File Type dropdown — Tenant ID, Client ID, Client Secret and File Type

Choose Your Compute Engine

Option	Result
Prizm Spark (default)	No further setup — skip to Monitoring
Use Databricks Spark Cluster	Continue to Step 3

Keep your Databricks cluster running during setup — a stopped cluster will not appear in the dropdown.

Connect Your Databricks Cluster

Field	Required	Notes
Platform Type	Required	Databricks Spark
Databricks Server	Required	Your workspace URL
Catalog Name	Required	The Unity Catalog catalog created above
Schema Name	Required	The Unity Catalog schema created above
Use Vault	Optional	Pull Client ID/Secret from vault
Client ID and Client Secret	Required	The Databricks execution identity
Notebook Path	Required	Path to the generic notebook (Step 4)
Spark Cluster	Required	Selected from dropdown

Scope the crawl with Containers to Include, Root Directory Path, and Include/Exclude Folder Patterns (glob-style, e.g. gold/*, */_staging/*). Excludes apply after includes are resolved.

The container dropdown lists every container in the account, not just ones your credentials can read — Prizm can’t filter this for a service principal or access token. Selecting an unreadable container shows a toast and auto-removes it from your selection.

Crawl limits: Partition Depth (how many subfolder levels to scan; 0 = root only), Max Files Per Directory, and an Incremental Mode toggle (scan only files changed since the last crawl).Delta Lake tables: when a folder contains a _delta_log, Prizm’s technical catalog step identifies it as a Delta table and catalogs the whole table as one asset (named after the table folder) instead of one asset per part file — reading it via Spark’s Delta format at the table root rather than as standalone Parquet. Updates trigger incremental rematerialization and metric refresh the same way as other ADLS assets. Plain Parquet folders without a _delta_log are unaffected.External storage details (Iceberg tables): if Iceberg tables in this source live in a different storage account than the one configured above, toggle Use External Storage Details and, if needed, Override External Storage Details to enter separate Storage Provider, Tenant ID, Client ID/Secret (or Storage Account Key/SAS Token), Storage Account Name, Resource Group, Container, and Directory Path. This is separate from the Unity Catalog materialization flow — it governs where Prizm reads existing Iceberg tables, not where it writes new ones.Advanced options: define per-asset file-matching rules (Asset Name, File Path, File Pattern) layered on top of Include/Exclude.

Set Up the Generic Notebook

A single generic notebook is used for quality-check jobs (separate from the two Unity Catalog notebooks above — all three run independently).

Import the notebook file (provided by your Prizm account team) into your Databricks workspace via Workspace → right-click → Import.
Generate an API token: Prizm platform → your organization → Tokens → Create Token.
Get your MCP host value from your Prizm account team.

Add the token and MCP host to the notebook’s designated fields — or, better, store them as Databricks secrets:

databricks secrets create-scope "prizm_notebook_secrets"
databricks secrets put-secret --json '{"scope": "prizm_notebook_secrets", "key": "access-token", "string_value": "<your-api-token>"}'
databricks secrets put-secret --json '{"scope": "prizm_notebook_secrets", "key": "mcp-host", "string_value": "<your-mcp-host-value>"}'

and reference them instead of pasting values directly: dbutils.secrets.get(scope="prizm_notebook_secrets", key="access-token").

Save the notebook to the path entered in Step 3.

Pasting the API token and MCP host directly into the notebook stores them in plain text — treat the notebook as a credential and restrict access accordingly.

Service Principal & Databricks CLI Configuration

# Create Databricks secret scope
databricks secrets create-scope <SECRET_SCOPE_NAME> -p <DATABRICKS_PROFILE>

# Store Service Principal Client ID and Secret
databricks secrets put-secret <SECRET_SCOPE_NAME> adls-client-id -p <DATABRICKS_PROFILE>
databricks secrets put-secret <SECRET_SCOPE_NAME> adls-client-secret -p <DATABRICKS_PROFILE>

# Set ACL permission for the Service Principal
databricks secrets put-acl \
  <SECRET_SCOPE_NAME> \
  "<SERVICE_PRINCIPAL_APP_ID>" \
  READ \
  -p <DATABRICKS_PROFILE>

Monitoring Your Jobs

Every job Prizm triggers appears in Databricks under Jobs & Pipelines, named by run date, a static keyword, and the source file name. Job status and logs behave like any other Databricks job. Once a job finishes, results return to Prizm automatically via webhook — no manual step required.

Pre-Go-Live Validation Checklist

Before the first production run of Iceberg materialization, confirm from within Databricks:

SHOW STORAGE CREDENTIALS; lists the credential from setup
SHOW EXTERNAL LOCATIONS; lists both locations with the correct URLs
DESCRIBE EXTERNAL LOCATION <name>; plus Test connection on each confirms read / read-write access
SHOW SCHEMAS IN <catalog_name>; confirms the target catalog/schema exist
SHOW GRANTS ON CATALOG <catalog_name>;, SHOW GRANTS ON SCHEMA ...;, SHOW GRANTS ON EXTERNAL LOCATION ...; confirm the execution identity’s grants
The execution identity can list/attach to the designated cluster (or launch a job cluster per the agreed policy)
A manual test run of a small known file succeeds: the table appears under <catalog>.<schema>, DESCRIBE DETAIL shows Iceberg-compatible properties, and the webhook payload reaches Prizm with a success status
External data access is enabled on the metastore, the Trino principal has EXTERNAL USE SCHEMA, and Trino can list/query the table via the Iceberg REST endpoint

Once confirmed, notify your Prizm implementation contact to schedule production onboarding.

Get Started

Core Concepts

Platform

Architecture

Data Sources

Integrations

Security & Access Control

Deployment

Help

Settings

MCP

Release Notes

Prerequisites Checklist

Authentication Setup (Azure)

Unity Catalog Setup

External locations

Exception Staging via a Unity Catalog Volume

Catalog and schema

Trino / external engine access

Consolidated grant script

Databricks Workspace Access

Execution identity

Notebook paths

Compute

Submitting Credentials to Prizm

Setting Up the Connection

Monitoring Your Jobs

Pre-Go-Live Validation Checklist

Next Steps

What We Collect

FAQ

​Prerequisites Checklist

​Authentication Setup (Azure)

​Unity Catalog Setup

​External locations

​Exception Staging via a Unity Catalog Volume

​Catalog and schema

​Trino / external engine access

​Consolidated grant script

​Databricks Workspace Access

​Execution identity

​Notebook paths

​Compute

​Submitting Credentials to Prizm

​Setting Up the Connection

​Monitoring Your Jobs

​Pre-Go-Live Validation Checklist

​Next Steps

What We Collect

FAQ

Prerequisites Checklist

Authentication Setup (Azure)

Unity Catalog Setup

External locations

Exception Staging via a Unity Catalog Volume

Catalog and schema

Trino / external engine access

Consolidated grant script

Databricks Workspace Access

Execution identity

Notebook paths

Compute

Submitting Credentials to Prizm

Setting Up the Connection

Monitoring Your Jobs

Pre-Go-Live Validation Checklist

Next Steps