> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# What We Collect
> Job types, compute metric types, and how Prizm handles Delta Lake tables and Iceberg materialization for ADLS.
| Job Type | What's Collected |
| :-------------------------- | :------------------------------------------------------------------------------------------------------------------ |
| **Operational** | Volume, schema, freshness |
| **Compute metrics** | Five supported measure types — Conditional, Query, Lookup, Comparison, and Behavioral (see the Compute Metrics tab) |
| **Profile and Structural** | Processed through the same pipeline; specific outputs depend on your configured checks |
| **Iceberg Materialization** | Fully qualified table name, row count, and status — returned via webhook |
| **Exception Workflow** | Fully qualified table name, row count, and status — returned via webhook |
Delta Lake tables are collected the same way as other ADLS assets: each Delta table registers as a single Operational asset, with volume, schema, and freshness refreshed incrementally as the table is updated.
When a compute metric job runs on your Databricks Spark cluster, Prizm supports five measure types — all executed via Spark on your own compute, with results returned through the same webhook callback as other job types.
| Measure Type | What it does |
| :-------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Conditional** | Evaluates records against rule-based logical conditions (threshold checks, boolean expressions) directly on the ADLS-sourced DataFrame. Supports population scoping — random, top-N, filter query, or incremental rolling window — so checks can run against a representative or targeted subset |
| **Query** | Executes parameterized SQL/PySpark queries against files read into Spark from ADLS (Parquet, CSV, JSON, XML), using Spark SQL for aggregation and filtering. Results score and push back through the same pipeline as other metric types |
| **Lookup** (API, Table, File, or Query) | Cross-references source records against a reference — an external API call, a Databricks/Unity Catalog table, a flat file in ADLS, or another query's result. Match/mismatch results carry a match score and issue type (Exact Match, Extra in Ref, Missing in Ref), feeding downstream exception reporting |
| **Comparison** | Compares two datasets (or two time-snapshots of the same dataset) field-by-field or record-by-record to detect drift or mismatches, using batched, set-based operations rather than row-by-row comparison |
| **Behavioral** | Applies statistical/ML-based scoring (trend, range, directional models) to detect anomalies in metric behavior over time, rather than static rule checks |
Prizm natively discovers Delta Lake tables stored in ADLS. When a folder contains a Delta transaction log (`_delta_log`), the technical catalog step identifies it as a Delta table and catalogs the **entire table as one asset** — not one asset per part file.
| Before | After |
| :--------------------------------------------------------------- | :------------------------------------------------------ |
| Each Parquet part file under a Delta folder cataloged separately | One asset per Delta table directory |
| Part-file Parquet reads could fail with Delta format errors | Table is read as Delta from the folder root |
| Delta layouts often appeared corrupted or rejected | Valid Delta tables catalog and materialize successfully |
* **One asset per Delta table**, registered under the table folder's name (e.g. `ALL_DATA_TYPES_20COL_SNAPPY`)
* **Correct reads for materialization** — read via Spark's Delta format at the table root, avoiding the errors that occurred when part files were read as standalone Parquet
* **Feeds the same Iceberg (Unity Catalog UniForm) pipeline** as other ADLS file types
* **Incremental refresh** — new commits/files under the table trigger rematerialization and a refresh of volume, schema, and freshness metrics
Plain Parquet folders without a `_delta_log` are unaffected. Delta support applies when the source's File Type allows Parquet or Delta — existing CSV, JSON, and XML sources are unaffected. Run (or re-run) technical catalog after enabling this to discover and register any Delta table assets.
For ADLS + Databricks Unity Catalog clients, exception and metadata instruction JSON is staged to your write external location at `abfss://…/staging/{run_id}/…`, via the `STAGE_EXCEPTION_JSON` job type on the exception workflow notebook — rather than uploaded to Prizm SaaS S3 or written via a scheduled Azure Blob upload as in earlier releases. This is covered by the same `WRITE FILES` grant used for Iceberg materialization; no separate grant is needed. Open-network clients are unaffected and keep the existing S3 HTTPS path.
## Notebook Parameters (Reference)
For technical review before deployment. Both `adls_uc_iceberg_materializer` and `adls_uc_exception_workflow` accept the same widget set — Prizm populates these automatically per job; no manual entry is required at run time.
| Parameter | Required | Description |
| :--------------------------------------------------------- | :---------- | :------------------------------------------------------------------------------------------------------------------------------------------------- |
| `storage_account` | Yes | ADLS Gen2 storage account name |
| `container` | Yes | Default source (read) container; can be overridden per file in a batch |
| `file_path` | Conditional | Path to a single source file (when `file_batch` isn't supplied) |
| `file_extension` | No | Overrides file-extension detection (csv, parquet, json, delta) |
| `access_key` / `client_id` + `client_secret` + `tenant_id` | Conditional | Authentication — mutually exclusive; the notebook prefers `access_key` if both are supplied |
| `uc_catalog` / `uc_schema` | Yes | Target Unity Catalog catalog/schema |
| `table_name` / `iceberg_table` | No | Target table name; for the exception workflow notebook this always resolves to the same dedicated exception table (created once, reused every run) |
| `asset_id` | Conditional | Prizm asset identifier (when `file_batch` isn't supplied) |
| `source_id` | Yes | Prizm source identifier, included in the webhook payload |
| `job_type` | Yes | `MATERIALIZE_ICEBERG`, `EXCEPTIONS_WORKFLOW`, or `STAGE_EXCEPTION_JSON` |
| `webhook_url` | Yes | Prizm callback endpoint that receives the resulting table metadata |
| `file_batch` | No | JSON array of file entries for multi-asset runs in a single job |
`adls_uc_exception_workflow` writes to its dedicated exception table rather than creating a new one per run — `CREATE TABLE` only fires on the first run; every subsequent run appends.
## Limitations
* The `Create At` column for ADLS Notebook assets cannot reflect the true source file creation time — Databricks external location listing only exposes the last-modified timestamp, not the original creation timestamp.
* File-type Lookup measures are not supported in the ADLS Notebook (Databricks) setup, since the Databricks notebook has no direct access to ADLS storage to upload a file for the lookup. Table- and query-type lookups are unaffected.
* Attribute/schema discovery for XML assets is supported only for flat record XML (column names used as tag names, e.g. ``). Application-generated spreadsheet XML formats — such as LibreOffice/OpenOffice ODS XML, where column names sit inside generic tags like `` — are not supported, and no attributes will be discovered for these files.
* For ADLS Spark (Iceberg-backed) assets, two system-generated columns — `upload_timestamp` and `prev_upload_timestamp` — always appear in the asset's attribute list, even when they're not part of the source file. These are added intentionally to support the Freshness metric calculation and aren't usable for other purposes.