What We Collect

Job Types
Compute Metrics
Delta Lake Tables
Exception JSON Staging

Job Type	What’s Collected
Operational	Volume, schema, freshness
Compute metrics	Five supported measure types — Conditional, Query, Lookup, Comparison, and Behavioral (see the Compute Metrics tab)
Profile and Structural	Processed through the same pipeline; specific outputs depend on your configured checks
Iceberg Materialization	Fully qualified table name, row count, and status — returned via webhook
Exception Workflow	Fully qualified table name, row count, and status — returned via webhook

Delta Lake tables are collected the same way as other ADLS assets: each Delta table registers as a single Operational asset, with volume, schema, and freshness refreshed incrementally as the table is updated.

When a compute metric job runs on your Databricks Spark cluster, Prizm supports five measure types — all executed via Spark on your own compute, with results returned through the same webhook callback as other job types.

Measure Type	What it does
Conditional	Evaluates records against rule-based logical conditions (threshold checks, boolean expressions) directly on the ADLS-sourced DataFrame. Supports population scoping — random, top-N, filter query, or incremental rolling window — so checks can run against a representative or targeted subset
Query	Executes parameterized SQL/PySpark queries against files read into Spark from ADLS (Parquet, CSV, JSON, XML), using Spark SQL for aggregation and filtering. Results score and push back through the same pipeline as other metric types
Lookup (API, Table, File, or Query)	Cross-references source records against a reference — an external API call, a Databricks/Unity Catalog table, a flat file in ADLS, or another query’s result. Match/mismatch results carry a match score and issue type (Exact Match, Extra in Ref, Missing in Ref), feeding downstream exception reporting
Comparison	Compares two datasets (or two time-snapshots of the same dataset) field-by-field or record-by-record to detect drift or mismatches, using batched, set-based operations rather than row-by-row comparison
Behavioral	Applies statistical/ML-based scoring (trend, range, directional models) to detect anomalies in metric behavior over time, rather than static rule checks

Prizm natively discovers Delta Lake tables stored in ADLS. When a folder contains a Delta transaction log (_delta_log), the technical catalog step identifies it as a Delta table and catalogs the entire table as one asset — not one asset per part file.

Before	After
Each Parquet part file under a Delta folder cataloged separately	One asset per Delta table directory
Part-file Parquet reads could fail with Delta format errors	Table is read as Delta from the folder root
Delta layouts often appeared corrupted or rejected	Valid Delta tables catalog and materialize successfully

One asset per Delta table, registered under the table folder’s name (e.g. ALL_DATA_TYPES_20COL_SNAPPY)
Correct reads for materialization — read via Spark’s Delta format at the table root, avoiding the errors that occurred when part files were read as standalone Parquet
Feeds the same Iceberg (Unity Catalog UniForm) pipeline as other ADLS file types
Incremental refresh — new commits/files under the table trigger rematerialization and a refresh of volume, schema, and freshness metrics

Plain Parquet folders without a _delta_log are unaffected. Delta support applies when the source’s File Type allows Parquet or Delta — existing CSV, JSON, and XML sources are unaffected. Run (or re-run) technical catalog after enabling this to discover and register any Delta table assets.

For ADLS + Databricks Unity Catalog clients, exception and metadata instruction JSON is staged to your write external location at abfss://…/staging/{run_id}/…, via the STAGE_EXCEPTION_JSON job type on the exception workflow notebook — rather than uploaded to Prizm SaaS S3 or written via a scheduled Azure Blob upload as in earlier releases. This is covered by the same WRITE FILES grant used for Iceberg materialization; no separate grant is needed. Open-network clients are unaffected and keep the existing S3 HTTPS path.

Notebook Parameters (Reference)

For technical review before deployment. Both adls_uc_iceberg_materializer and adls_uc_exception_workflow accept the same widget set — Prizm populates these automatically per job; no manual entry is required at run time.

Parameter	Required	Description
`storage_account`	Yes	ADLS Gen2 storage account name
`container`	Yes	Default source (read) container; can be overridden per file in a batch
`file_path`	Conditional	Path to a single source file (when `file_batch` isn’t supplied)
`file_extension`	No	Overrides file-extension detection (csv, parquet, json, delta)
`access_key` / `client_id` + `client_secret` + `tenant_id`	Conditional	Authentication — mutually exclusive; the notebook prefers `access_key` if both are supplied
`uc_catalog` / `uc_schema`	Yes	Target Unity Catalog catalog/schema
`table_name` / `iceberg_table`	No	Target table name; for the exception workflow notebook this always resolves to the same dedicated exception table (created once, reused every run)
`asset_id`	Conditional	Prizm asset identifier (when `file_batch` isn’t supplied)
`source_id`	Yes	Prizm source identifier, included in the webhook payload
`job_type`	Yes	`MATERIALIZE_ICEBERG`, `EXCEPTIONS_WORKFLOW`, or `STAGE_EXCEPTION_JSON`
`webhook_url`	Yes	Prizm callback endpoint that receives the resulting table metadata
`file_batch`	No	JSON array of file entries for multi-asset runs in a single job

adls_uc_exception_workflow writes to its dedicated exception table rather than creating a new one per run — CREATE TABLE only fires on the first run; every subsequent run appends.

Limitations

The Create At column for ADLS Notebook assets cannot reflect the true source file creation time — Databricks external location listing only exposes the last-modified timestamp, not the original creation timestamp.
File-type Lookup measures are not supported in the ADLS Notebook (Databricks) setup, since the Databricks notebook has no direct access to ADLS storage to upload a file for the lookup. Table- and query-type lookups are unaffected.
Attribute/schema discovery for XML assets is supported only for flat record XML (column names used as tag names, e.g. <EMP_ID>). Application-generated spreadsheet XML formats — such as LibreOffice/OpenOffice ODS XML, where column names sit inside generic tags like <text:p> — are not supported, and no attributes will be discovered for these files.
For ADLS Spark (Iceberg-backed) assets, two system-generated columns — upload_timestamp and prev_upload_timestamp — always appear in the asset’s attribute list, even when they’re not part of the source file. These are added intentionally to support the Freshness metric calculation and aren’t usable for other purposes.

Get Started

Core Concepts

Platform

Architecture

Data Sources

Integrations

Security & Access Control

Deployment

Help

Settings

MCP

Release Notes

API Reference

Notebook Parameters (Reference)

Limitations

​Notebook Parameters (Reference)

​Limitations

Notebook Parameters (Reference)

Limitations