> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# What Prizm Collects from Databricks

> Complete field-level breakdown of every metadata object, quality metric, and signal Prizm extracts from Databricks Unity Catalog across all platform jobs.

<script type="application/ld+json">
  {`{
            "@context": "https://schema.org",
            "@type": "TechArticle",
            "headline": "What Prizm Collects from Databricks",
            "description": "Complete field-level breakdown of every metadata object, quality metric, and signal Prizm extracts from Databricks Unity Catalog across all platform jobs.",
            "url": "https://docs.dqlabs.ai/sources/databricks/what-we-collect",
            "publisher": {
              "@type": "Organization",
              "name": "DQLabs Inc",
              "logo": "https://media.brand.dev/332adc35-5bc4-4d2b-bf78-256aa4a5e414.svg"
            }
            }`}
</script>

Prizm runs the following platform jobs against each Databricks connector. Each job has a defined type, execution flow, and purpose. Select a job below to see the details.

<Tabs>
  <Tab title="Technical">
    | Property      | Value                                                                   |
    | :------------ | :---------------------------------------------------------------------- |
    | **Type**      | CONTEXT                                                                 |
    | **Execution** | Runs immediately after source is configured — first job in the pipeline |

    **What this job does:** Extracts catalogs, schemas, and top-level structural metadata from the Databricks Unity Catalog. This is the foundation job — all subsequent jobs depend on it completing successfully. Without a successful TECHNICAL run, no other job can start.

    | Object      | Fields Collected                                                                                                                          | Databricks Source                       |
    | :---------- | :---------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------- |
    | **Catalog** | name, owner, comment, metastore\_id, created\_at, updated\_at                                                                             | `GET /api/2.1/unity-catalog/catalogs`   |
    | **Schema**  | schema\_name, catalog\_name, schema\_owner, comment, created, last\_altered                                                               | `<catalog>.information_schema.schemata` |
    | **Table**   | table\_name, table\_type, schema\_name, catalog\_name, table\_owner, comment, created, last\_altered, data\_source\_format, storage\_path | `<catalog>.information_schema.tables`   |
    | **View**    | Same as Table + view\_definition                                                                                                          | `<catalog>.information_schema.views`    |
    | **Column**  | column\_name, data\_type, ordinal\_position, is\_nullable, column\_default, comment, partition\_index, mask\_function\_name               | `<catalog>.information_schema.columns`  |

    <Note>
      Prizm catalogs Managed Tables, External Tables, and Views from Unity Catalog. Delta Live Tables and Jobs/Pipelines are discoverable when the Pipeline asset type is enabled in the connector settings.
    </Note>
  </Tab>

  <Tab title="Operational">
    | Property      | Value                            |
    | :------------ | :------------------------------- |
    | **Type**      | CONTEXT / TRUST                  |
    | **Execution** | Starts after TECHNICAL completes |

    **What this job does:** Extracts volume, freshness, and schema change signals for all in-scope table assets from Databricks metadata. Serves a dual purpose — it feeds the Prizm catalog (CONTEXT) and the alerting and anomaly detection pipeline (TRUST). It is the source of all time-series observability signals in Prizm and the upstream dependency for most parallel jobs.

    | Signal                         | Source                                     | Description                                                        |
    | :----------------------------- | :----------------------------------------- | :----------------------------------------------------------------- |
    | **Volume**                     | `information_schema.tables.row_count`      | Row count trend; anomaly alerts on drops/spikes                    |
    | **Freshness (Managed Table)**  | `DESCRIBE HISTORY <table> LIMIT 1`         | Latest Delta commit timestamp — captures data writes, not just DDL |
    | **Freshness (External Table)** | `DESCRIBE DETAIL <table>` → `lastModified` | Last modification timestamp from the external storage layer        |
    | **Schema Change**              | `information_schema.tables.last_altered`   | Column additions, renames, type changes, removals                  |

    Anomaly detection uses adaptive forecasting models (EWMA, AutoReg, Prophet, SARIMAX) selected automatically based on each metric's historical pattern. Thresholds are recalculated after every run. A minimum of 5 historical values is required before alerts begin firing.
  </Tab>

  <Tab title="Lineage">
    | Property      | Value                                        |
    | :------------ | :------------------------------------------- |
    | **Type**      | CONTEXT                                      |
    | **Execution** | Runs in parallel after OPERATIONAL completes |

    **What this job does:** Extracts table-level and column-level lineage from Databricks Unity Catalog system tables. Lineage extraction is event-driven — triggered only when a DDL change is detected on an asset (i.e., when `LAST_ALTERED` changes), keeping extraction efficient with no redundant per-hour runs.

    | Direction                | Source Table                   | Fields                                                                                                             |
    | :----------------------- | :----------------------------- | :----------------------------------------------------------------------------------------------------------------- |
    | **Table-level lineage**  | `system.access.table_lineage`  | source\_table\_full\_name, target\_table\_full\_name, dependency\_type (DIRECT / TRANSITIVE), event\_time          |
    | **Column-level lineage** | `system.access.column_lineage` | source\_table\_full\_name, source\_column\_name, target\_table\_full\_name, target\_column\_name, dependency\_type |

    | Direction      | What Prizm Shows                                       |
    | :------------- | :----------------------------------------------------- |
    | **Upstream**   | Tables and views this asset reads from                 |
    | **Downstream** | Tables, views, and pipelines that depend on this asset |

    <Note>
      Access to `system.access.table_lineage` and `system.access.column_lineage` requires `SELECT` grants on those system tables. These tables are populated by Unity Catalog's built-in lineage tracking — no additional Databricks configuration is required beyond the permission grant.
    </Note>
  </Tab>

  <Tab title="Performance">
    | Property      | Value                                        |
    | :------------ | :------------------------------------------- |
    | **Type**      | CONTEXT                                      |
    | **Execution** | Runs in parallel after OPERATIONAL completes |

    **What this job does:** Extracts warehouse usage, compute costs, and query statistics for the previous 24 hours. Powers the Usage and Performance views in the Prizm asset detail page. This job is **disabled by default** and must be explicitly enabled in the connector settings. Requires `SELECT` on `system.query.history` and `system.billing.usage`.

    | Category                  | Metrics                                                                                                    | Source                 |
    | :------------------------ | :--------------------------------------------------------------------------------------------------------- | :--------------------- |
    | **Query Performance**     | Execution time, bytes read, bytes written, result cache hit, Photon time, spill to disk, rows read/written | `system.query.history` |
    | **Warehouse Utilization** | Total queries per warehouse, unique users, total execution hours, cache hit rate                           | `system.query.history` |
    | **Billing & Cost**        | Compute cost per query/job, storage cost per table, cost per domain, cost per user/role, cost per pipeline | `system.billing.usage` |

    <Note>
      The Performance job is **disabled by default** and must be explicitly enabled per connector instance. Prizm retains time-series observability data, quality score history, and query usage history for a default window of **90 days**.
    </Note>
  </Tab>

  <Tab title="Usage Summary">
    | Property      | Value                                        |
    | :------------ | :------------------------------------------- |
    | **Type**      | CONTEXT                                      |
    | **Execution** | Runs in parallel after OPERATIONAL completes |

    **What this job does:** Extracts summarized usage information and generates usage-based criticality inputs using 7 days of activity. Analyses how frequently each asset is queried, by whom, and with what patterns — then feeds those signals into the CRITICALITY RECOMMENDATION job. Assets with high query volume or many unique users are more likely to be recommended as critical.

    | What It Produces       | Description                                                        |
    | :--------------------- | :----------------------------------------------------------------- |
    | **Usage frequency**    | Number of queries per asset over the past 7 days                   |
    | **Unique user count**  | Number of distinct users who accessed each asset                   |
    | **Access patterns**    | Read vs write activity, peak usage windows                         |
    | **Criticality inputs** | Usage-based signals consumed by the CRITICALITY RECOMMENDATION job |
  </Tab>

  <Tab title="Governance">
    | Property      | Value                                        |
    | :------------ | :------------------------------------------- |
    | **Type**      | CONTEXT                                      |
    | **Execution** | Runs in parallel after OPERATIONAL completes |

    **What this job does:** Extracts governance metadata including tags and business metadata. Keeps Prizm's governance layer in sync with classifications, ownership, and tagging applied directly in Databricks Unity Catalog.

    | Object                  | Fields Collected                                                                        | Databricks Source                       |
    | :---------------------- | :-------------------------------------------------------------------------------------- | :-------------------------------------- |
    | **Tags (table-level)**  | tag\_name, tag\_value, catalog\_name, schema\_name, table\_name                         | `system.information_schema.table_tags`  |
    | **Tags (column-level)** | tag\_name, tag\_value, table\_name, column\_name                                        | `system.information_schema.column_tags` |
    | **Stored Procedure**    | routine\_name, routine\_type, return\_type, routine\_definition, created, last\_altered | `<catalog>.information_schema.routines` |
    | **Business metadata**   | Owner, description, classification fields                                               | `information_schema` object properties  |
  </Tab>

  <Tab title="Criticality Recommendation">
    | Property      | Value                                        |
    | :------------ | :------------------------------------------- |
    | **Type**      | CONTEXT                                      |
    | **Execution** | Runs in parallel after OPERATIONAL completes |

    **What this job does:** Calculates and recommends criticality scores for assets and attributes. Criticality determines how much observability coverage an asset receives — higher criticality assets get more frequent monitoring, more structural metrics, and tighter alert thresholds. The output is a recommended criticality level (LOW, MEDIUM, HIGH, CRITICAL) per asset, which users can accept or override.

    | Input Signal                   | Source Job              |
    | :----------------------------- | :---------------------- |
    | Usage frequency and user count | USAGE SUMMARY           |
    | Downstream dependency count    | LINEAGE                 |
    | Business metadata completeness | GOVERNANCE              |
    | Asset type and schema position | TECHNICAL / OPERATIONAL |
  </Tab>

  <Tab title="Compute Metric">
    | Property      | Value                                        |
    | :------------ | :------------------------------------------- |
    | **Type**      | TRUST                                        |
    | **Execution** | Runs in parallel after OPERATIONAL completes |

    **What this job does:** Computes non-metadata metrics such as freshness and volume for non-table assets by running direct queries against Databricks. The OPERATIONAL job derives signals from metadata — but for assets like views and external tables, metadata alone is not sufficient. COMPUTE METRIC fills this gap by executing lightweight queries to obtain accurate signals.

    | Asset Type            | What Is Computed                           | How                                                  |
    | :-------------------- | :----------------------------------------- | :--------------------------------------------------- |
    | **View**              | Row count, freshness derived from upstream | Direct `SELECT COUNT(*)` and upstream lineage lookup |
    | **External Table**    | Row count, last modified                   | `DESCRIBE DETAIL` → `lastModified`                   |
    | **Materialized View** | Row count, refresh timestamp               | Query against materialized view metadata             |
  </Tab>

  <Tab title="Profile Recommendation">
    | Property      | Value                                                                 |
    | :------------ | :-------------------------------------------------------------------- |
    | **Type**      | TRUST                                                                 |
    | **Execution** | Runs in parallel after OPERATIONAL; PROFILE job depends on its output |

    **What this job does:** Recommends profiling schedules and configurations based on asset scope definitions. Determines which assets should be profiled, at what frequency, and with what sample settings — taking into account criticality level, asset size, and scope configuration. The PROFILE job will not run until this job has produced its output and created the schedules.
  </Tab>

  <Tab title="Structural Recommendation">
    | Property      | Value                                                                    |
    | :------------ | :----------------------------------------------------------------------- |
    | **Type**      | TRUST                                                                    |
    | **Execution** | Runs in parallel after OPERATIONAL; STRUCTURAL job depends on its output |

    **What this job does:** Recommends structural quality metrics (Essential or Advanced) for each attribute. Analyses the column's data type, profile history, and criticality to determine the appropriate metric set. Essential metrics are always recommended; Advanced metrics are added for columns where deeper analysis is warranted. The STRUCTURAL job executes against this recommendation output.
  </Tab>

  <Tab title="Profile">
    | Property      | Value                                                                      |
    | :------------ | :------------------------------------------------------------------------- |
    | **Type**      | TRUST                                                                      |
    | **Execution** | Triggered after PROFILE RECOMMENDATION completes and schedules are created |

    **What this job does:** Executes profiling jobs according to the schedules generated by PROFILE RECOMMENDATION and stores the profiling results. Can also be triggered manually from the asset page at any time.

    **Table-level metrics:**

    | Metric                  | Description                                                                                      |
    | :---------------------- | :----------------------------------------------------------------------------------------------- |
    | **Row Count**           | Total rows at profile time (from `information_schema.tables.row_count` with `COUNT(*)` fallback) |
    | **Duplicate Row Count** | Rows that are exact duplicates of another row                                                    |
    | **Completeness Score**  | % of columns with non-null values across all rows                                                |

    **Column-level metrics:**

    | Metric                  | Applicable Types        | Description                                    |
    | :---------------------- | :---------------------- | :--------------------------------------------- |
    | **Null Rate**           | All                     | % of null values                               |
    | **Distinct Count**      | All                     | Number of unique non-null values (cardinality) |
    | **Uniqueness Rate**     | All                     | % of values appearing exactly once             |
    | **Min / Max**           | Numeric, Date/Timestamp | Observed range of values                       |
    | **Mean**                | Numeric                 | Arithmetic mean                                |
    | **Standard Deviation**  | Numeric                 | Statistical spread                             |
    | **Median**              | Numeric                 | 50th percentile value                          |
    | **Top N Values**        | String, Boolean         | Most frequent values and their counts          |
    | **Pattern Conformance** | String                  | % matching an expected regex format            |
    | **Type Conformance**    | String                  | % of values castable to the declared data type |

    **Trigger conditions:**

    | Trigger                | Conditions                                                                                                                                    |
    | :--------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------- |
    | **Scheduled run**      | Active schedule, `next_run_at` is due, source is valid and active                                                                             |
    | **Manual run**         | User triggers Profile from the asset UI (TABLE or VIEW)                                                                                       |
    | **Criticality gating** | Only attributes eligible for the asset's criticality level are profiled                                                                       |
    | **Skip (incremental)** | Skipped if the table row count is unchanged since the last run, or if the existing profile is newer than the table's `last_altered` timestamp |
  </Tab>

  <Tab title="Structural">
    | Property      | Value                                                                       |
    | :------------ | :-------------------------------------------------------------------------- |
    | **Type**      | TRUST                                                                       |
    | **Execution** | Triggered after STRUCTURAL RECOMMENDATION completes and metrics are created |

    **What this job does:** Executes structural quality metrics according to the schedules and configurations generated by STRUCTURAL RECOMMENDATION. Computes distribution, frequency, pattern, and statistics metrics at the attribute level for all in-scope columns.

    | Metric Category  | What Is Computed                                                                                          |
    | :--------------- | :-------------------------------------------------------------------------------------------------------- |
    | **Distribution** | Null count, empty, space, distinct, repeating, character composition, leading/trailing/inner space        |
    | **Frequency**    | Enum (value histogram), min/max value, value range, min/max length, length distribution                   |
    | **Pattern**      | Short pattern, long pattern, user-defined regular expressions                                             |
    | **Statistics**   | Mean, median, mode, standard deviation, sum, Q1, Q3, range, skewness, kurtosis, variance, margin of error |
  </Tab>

  <Tab title="Asset Detail Recommendation">
    | Property      | Value                                   |
    | :------------ | :-------------------------------------- |
    | **Type**      | CONTEXT                                 |
    | **Execution** | Independent job; no upstream dependency |

    **What this job does:** Recommends asset definitions, descriptions, intended audience, semantic context, and execution scope settings (Incremental, Sampling with Filter, or None). Uses AI to analyse the asset's name, column names, sample values, and usage patterns to generate human-readable context that data stewards can review and accept.

    | What It Generates    | Description                                                                                 |
    | :------------------- | :------------------------------------------------------------------------------------------ |
    | **Description**      | AI-generated plain-language description of what the asset contains and its purpose          |
    | **Who should use**   | Recommended audience — data analysts, engineers, business users, etc.                       |
    | **Semantic context** | Suggested domain, product, application, and tag classifications                             |
    | **Execution scope**  | Recommended profiling mode — Incremental (only changed rows), Sampling with Filter, or None |
  </Tab>

  <Tab title="Business Metric Recommendation">
    | Property      | Value                                   |
    | :------------ | :-------------------------------------- |
    | **Type**      | TRUST                                   |
    | **Execution** | Independent job; no upstream dependency |

    **What this job does:** Recommends business metrics for assets based on column names, data types, profiling results, and industry patterns. Suggests quality checks that go beyond structural profiling — for example, a not-null check on a primary key column, or a range check on a revenue column. Recommendations appear in the Prizm UI for review before any metric is created.
  </Tab>

  <Tab title="Metric Detail Recommendation">
    | Property      | Value                                   |
    | :------------ | :-------------------------------------- |
    | **Type**      | TRUST                                   |
    | **Execution** | Independent job; no upstream dependency |

    **What this job does:** Recommends metric descriptions, purposes, and rationale for existing metrics. For each metric already defined on an asset, this job generates a human-readable explanation of what the metric checks, why it matters, and what a failure would indicate. These descriptions appear in the Prizm metric detail panel and help data stewards understand the intent behind auto-generated checks.
  </Tab>

  <Tab title="Term Recommendation">
    | Property      | Value                                   |
    | :------------ | :-------------------------------------- |
    | **Type**      | CONTEXT                                 |
    | **Execution** | Independent job; no upstream dependency |

    **What this job does:** Maps assets and attributes to existing glossary terms or generates new business terms where required. Analyses column names, descriptions, and sample values to find matching terms in the Prizm business glossary. Powers the Semantic layer in Prizm — ensuring that columns are aligned to approved business vocabulary.

    | Output               | Description                                                                              |
    | :------------------- | :--------------------------------------------------------------------------------------- |
    | **Term mapping**     | Links a column to an existing glossary term (e.g. `customer_id` → "Customer Identifier") |
    | **Term proposal**    | Suggests a new glossary term where no existing match is found                            |
    | **Confidence score** | Indicates how confident the AI is in the mapping recommendation                          |
  </Tab>

  <Tab title="Alert Cluster">
    | Property      | Value                                   |
    | :------------ | :-------------------------------------- |
    | **Type**      | TRUST                                   |
    | **Execution** | Independent job; no upstream dependency |

    **What this job does:** Clusters and groups related alerts to reduce duplication and improve root-cause analysis. When multiple alerts fire simultaneously on the same asset or related assets — for example, a freshness alert and a volume alert on the same table — this job groups them into a single alert cluster. Reduces alert fatigue by surfacing correlated failures together rather than as separate notifications.
  </Tab>

  <Tab title="Score Trend">
    | Property      | Value                                   |
    | :------------ | :-------------------------------------- |
    | **Type**      | TRUST                                   |
    | **Execution** | Independent job; no upstream dependency |

    **What this job does:** Generates and refreshes daily asset trend data by aggregating DQ scores, alerts, issues, and asset metadata. Powers the score history charts on the asset detail page — the time-series view of how an asset's quality score has evolved over time. Also feeds domain-level and portfolio-level trend views in the Prizm Scorecard and Analytics pages.
  </Tab>

  <Tab title="Purge">
    | Property      | Value                                   |
    | :------------ | :-------------------------------------- |
    | **Type**      | PLATFORM                                |
    | **Execution** | Independent job; no upstream dependency |

    **What this job does:** Cleans and maintains the Prizm repository by removing obsolete data. Deletes stale metric history, expired alert records, orphaned job artifacts, and data from deleted or disconnected sources. Runs on a configurable schedule to keep the Prizm database within the configured retention window. Does not affect live assets or active metric data within the retention period.
  </Tab>

  <Tab title="Pipeline">
    | Property      | Value                                                                                                                           |
    | :------------ | :------------------------------------------------------------------------------------------------------------------------------ |
    | **Type**      | CONTEXT / TRUST                                                                                                                 |
    | **Execution** | Runs on a 15-minute polling interval when **Enable pipelines and jobs** is toggled on in the connector's Advanced Configuration |

    **What this job does:** Discovers and monitors all in-scope Databricks Job, Workflow, Delta Live Table (DLT), and Lakeflow Pipeline objects. On first run it backfills run history for the configured lookback window (default 7 days). On subsequent runs it incrementally ingests new run events and flow-level telemetry, computes observability metrics, and fires alerts on failures, missed schedules, and data quality expectation violations.

    **Pipeline catalog fields** (sourced from `system.lakeflow.job_runs` and the Databricks Jobs REST API):

    | Field                        | Pipeline Types                   | Description                                                                                 |
    | :--------------------------- | :------------------------------- | :------------------------------------------------------------------------------------------ |
    | `workspace_id`               | All                              | Databricks workspace identifier                                                             |
    | `pipeline_id` / `job_id`     | All                              | Unique identifier for the Job or DLT pipeline                                               |
    | `pipeline_name` / `job_name` | All                              | Display name                                                                                |
    | `creator_user_name`          | Jobs, DLT, Procedures, Functions | Identity that created the pipeline                                                          |
    | `run_as_user_name`           | Jobs, Procedures                 | Service principal or user the pipeline runs as                                              |
    | `catalog_name`               | All                              | Unity Catalog catalog the pipeline operates on                                              |
    | `schema_name`                | DLT, Procedures, Functions       | Target schema                                                                               |
    | `start_time`                 | Jobs, DLT, Tasks                 | Run start timestamp                                                                         |
    | `end_time`                   | Jobs, DLT, Tasks                 | Run end timestamp                                                                           |
    | `execution_duration`         | Jobs, DLT, Tasks                 | Total elapsed time in milliseconds                                                          |
    | `result_state`               | Jobs, DLT, Tasks                 | Final run state: `SUCCESS`, `FAILED`, `CANCELLED`, `SKIPPED`, `EXCLUDED`, `STOPPED`, `IDLE` |

    **Flow-level telemetry fields** (sourced from DLT event log — `flow_progress` events):

    | Field                               | Description                                                   |
    | :---------------------------------- | :------------------------------------------------------------ |
    | `table_name` / `flow_name`          | Name of the DLT flow (streaming table or materialized view)   |
    | `update_id`                         | DLT pipeline update identifier                                |
    | `final_status`                      | Flow-level outcome (`COMPLETED`, `FAILED`, `CANCELLED`, etc.) |
    | `total_output_records`              | Rows written to the target in this update                     |
    | `total_upserted_records`            | Rows upserted (merge operations)                              |
    | `total_deleted_records`             | Rows deleted                                                  |
    | `total_expectation_dropped_records` | Rows dropped due to DLT `EXPECT` constraint violations        |
    | `duration_seconds`                  | Flow-level execution time in seconds                          |

    **DLT expectation (data quality) fields** (sourced from DLT event log — `flow_progress.data_quality`):

    | Field              | Description                                             |
    | :----------------- | :------------------------------------------------------ |
    | `expectation_name` | Name of the `EXPECT` constraint                         |
    | `dataset`          | DLT table the expectation applies to                    |
    | `passed_records`   | Records that satisfied the expectation                  |
    | `failed_records`   | Records that violated the expectation                   |
    | `dropped_records`  | Records dropped when the expectation's action is `DROP` |

    <Note>
      Pipeline collection requires **Enable pipelines and jobs** to be turned on in the connector's Advanced Configuration. The lookback window (default 7 days) controls how far back Prizm reaches on the initial sync. Subsequent runs are incremental.
    </Note>
  </Tab>
</Tabs>

## What You See in Prizm

Once Prizm completes its initial catalog and observability runs, every Databricks asset gets a unified detail page in the catalog. The sections below describe what each asset type shows.

## Table & View Asset Pages

Tables and views cataloged from Databricks Unity Catalog each get a dedicated asset detail page with the following tabs.

<Tabs>
  <Tab title="Overview">
    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_overview-1.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=91b0149cab4a3d2ddb068be0011c52b5" alt="Databricks Overview 1" width="2831" height="2495" data-path="images/databricks_overview-1.png" />
    </Frame>

    | **Section**             | **What Prizm Shows**                                                                                                   |
    | :---------------------- | :--------------------------------------------------------------------------------------------------------------------- |
    | **Quality Score**       | Overall data quality percentage computed across all active metrics                                                     |
    | **Status**              | Review state (e.g., READY FOR REVIEW) and criticality badge                                                            |
    | **Key Metrics**         | VOLUME (row count), SCHEMA (column count), FRESHNESS (time since last update), METRICS (total quality metrics defined) |
    | **Description**         | AI-generated or manually verified description of the asset                                                             |
    | **Semantic Context**    | Domain, Application, Product, Tag, and Terms classifications                                                           |
    | **Owners**              | Business owner, Technical owner, and Steward contacts                                                                  |
    | **Audience**            | Intended audience and recommended use cases                                                                            |
    | **Who should use this** | AI-generated guidance on which teams or roles should consume this asset                                                |
  </Tab>

  <Tab title="Attribute">
    The Attribute tab lists every column Prizm has cataloged with columns: **NAME, DATA TYPE, SCORE, ALERT, ISSUE, METRIC COUNT, STATUS**.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_attribute-1.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=85910cb5c492e4ca651e0328c6990d4e" alt="Databricks Attribute 1" width="2831" height="1712" data-path="images/databricks_attribute-1.png" />
    </Frame>

    Clicking any attribute opens its column profile with views across completeness, uniqueness, character distribution, patterns, and frequency.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_profile-1.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=1fad49a47cda6be8245b6ceace7edaed" alt="Databricks Profile 1" width="2831" height="1666" data-path="images/databricks_profile-1.png" />
    </Frame>

    | **Profile Section** | **What Prizm Shows**                                                  |
    | :------------------ | :-------------------------------------------------------------------- |
    | **Completeness**    | Count and % of NULL, EMPTY, SPACE, and NON-EMPTY values               |
    | **Uniqueness**      | Count and % of DISTINCT vs REPEATING values                           |
    | **Character**       | Distribution across DIGIT, ALPHABET, ALPHA NUMERIC, SPECIAL CHARACTER |
    | **Length**          | Minimum and maximum character length                                  |
    | **Pattern**         | Top regex patterns detected across values                             |
    | **Frequency**       | Most and least frequent individual values                             |

    For the full breakdown of profiling views, metric types, and how to interpret profile results, see the [Data Profiling](/data-profiling) section.
  </Tab>

  <Tab title="Metric">
    The Metric tab lists all quality metrics defined on the asset. Each row shows the metric name, asset/column name, domain, current score, alert count, issue count, and action controls.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_Metric-1.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=813f4e67885676f763f6dc68ff02cbf3" alt="Databricks Metric 1" width="2831" height="1712" data-path="images/databricks_Metric-1.png" />
    </Frame>

    Metrics are filterable by category:

    | **Category**     | **What It Contains**                                                                                                                      |
    | :--------------- | :---------------------------------------------------------------------------------------------------------------------------------------- |
    | **Distribution** | Completeness, uniqueness, character, and space metrics (null count, empty, distinct, repeating, whitespace, leading/trailing space, etc.) |
    | **Custom**       | User-defined quality rules, filters, and threshold expressions                                                                            |
    | **Frequency**    | Value frequency histograms, min/max values, min/max length, value range, and length distribution                                          |
    | **Pattern**      | Short pattern, long pattern, and user-defined regular expression metrics                                                                  |
    | **Statistics**   | Numeric summary metrics: mean, median, mode, standard deviation, sum, Q1, Q3, range, skewness, kurtosis, variance, margin of error        |
    | **Availability** | Data availability and freshness signals                                                                                                   |
    | **Conditional**  | Metrics applying different logic or thresholds based on a column condition                                                                |
  </Tab>

  <Tab title="Usage">
    The Usage tab surfaces query history for the asset — who queried it, when, and at what cost. It has two sub-views: **Queries** and **Users**. Powered by the PERFORMANCE job (`system.query.history`).

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_Usage-1.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=341aaf03382de7a22d3020290d566904" alt="Databricks Usage 1" width="2831" height="2085" data-path="images/databricks_Usage-1.png" />
    </Frame>

    **Queries** — filterable by ALL, EXPENSIVE, SLOWER, POPULAR:

    | **Column**             | **What Prizm Shows**                          |
    | :--------------------- | :-------------------------------------------- |
    | **Query**              | The SQL statement executed against this asset |
    | **Query Date**         | Date the query was run                        |
    | **User**               | User or service principal that ran the query  |
    | **Start Time**         | Query start timestamp                         |
    | **End Time**           | Query end timestamp                           |
    | **Status**             | Query outcome (success, failed, etc.)         |
    | **Execution Time (s)** | Total execution time in seconds               |
    | **Credits Used**       | Compute credits consumed by this query        |

    **Users** — shows distinct users who have queried this asset, their query counts, and execution patterns.

    <Note>
      Usage data requires the Performance job to be enabled and the service principal to have `SELECT` on `system.query.history`.
    </Note>
  </Tab>

  <Tab title="Lineage">
    The Lineage tab shows upstream sources and downstream consumers as a directed graph. Each node displays the asset name, object type, row count, and column count.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/Databricks_customer-1.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=d6ba778cf967a8aa5e0492ef541780ad" alt="Databricks Customer 1" width="2831" height="1821" data-path="images/Databricks_customer-1.png" />
    </Frame>

    | **Direction**    | **What Prizm Shows**                                               |
    | :--------------- | :----------------------------------------------------------------- |
    | **Upstream**     | Tables, views, and external sources this asset reads from          |
    | **Downstream**   | Tables, views, and pipelines that depend on this asset             |
    | **Column-level** | Which specific columns flow between upstream and downstream assets |

    Lineage is extracted from `system.access.table_lineage` (table-level) and `system.access.column_lineage` (column-level) and is triggered only when a DDL change is detected.
  </Tab>

  <Tab title="Audit">
    The Audit tab is a tamper-evident log of every action taken on the asset — who changed what, and when. It provides a complete change history for governance, compliance, and incident investigation.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_audit-1.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=3ddeae7ad68506a764a778ba56bdf18b" alt="Databricks Audit 1" width="2831" height="1712" data-path="images/databricks_audit-1.png" />
    </Frame>

    | **Column**   | **What Prizm Shows**                                                                  |
    | :----------- | :------------------------------------------------------------------------------------ |
    | **Log Date** | Timestamp of the recorded action                                                      |
    | **Action**   | Type of change (e.g., metadata update, description edit, owner change, status update) |
    | **Message**  | Human-readable detail of what changed and by whom                                     |

    The log is filterable by action type. All records are immutable — entries cannot be edited or deleted from the UI.
  </Tab>
</Tabs>

## Pipeline Asset Pages

When **Enable pipelines and jobs** is turned on, each Databricks Job, Workflow, and Delta Live Table gets its own pipeline asset page in Prizm. The pipeline asset page has a different tab set from table and view assets, reflecting the operational and run-level nature of pipeline observability.

<Tabs>
  <Tab title="Overview">
    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_piplien_overview.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=0f1088060998622315152d10180ad715" alt="Databricks Piplien Overview" width="2831" height="2415" data-path="images/databricks_piplien_overview.png" />
    </Frame>

    | **Section**          | **What Prizm Shows**                                                                                                                                                                                                            |
    | :------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
    | **Quality Score**    | Overall quality percentage across all active pipeline metrics                                                                                                                                                                   |
    | **Status**           | Review state and criticality badge                                                                                                                                                                                              |
    | **Key Metrics**      | ROWS AFFECTED (rows written per run), SCHEMA (output schema column count), FRESHNESS (time since last successful run), METRICS (active metric count), TOTAL RUNS (total historical runs), FAILURE RATE % (rolling failure rate) |
    | **Description**      | AI-generated or manually verified description of the pipeline                                                                                                                                                                   |
    | **Semantic Context** | Domain, Application, Product, Tag, and Terms classifications                                                                                                                                                                    |
    | **Owners**           | Business owner, Technical owner, and Steward contacts                                                                                                                                                                           |
  </Tab>

  <Tab title="Metric">
    The Metric tab lists all quality metrics defined on the pipeline asset — including observability metrics such as Execution Status, Freshness, Failure Rate, Rows Affected, and Dropped Records.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_pipeline_metrics.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=f423a10c738b783f2a49ef9b496b735e" alt="Databricks Pipeline Metrics" width="2829" height="1485" data-path="images/databricks_pipeline_metrics.png" />
    </Frame>

    | **Metric**           | **What It Measures**                                                     |
    | :------------------- | :----------------------------------------------------------------------- |
    | **Execution Status** | Whether the most recent pipeline run succeeded, failed, or was cancelled |
    | **Freshness**        | Time elapsed since the last successful pipeline completion               |
    | **Rows Affected**    | Number of rows written, updated, or deleted by the pipeline run          |
    | **Failure Rate**     | Percentage of pipeline runs that have failed over a rolling window       |
    | **Dropped Records**  | Records discarded due to DLT `EXPECT` constraint violations              |
    | **Failed Records**   | Records that failed validation or processing rules                       |
  </Tab>

  <Tab title="Lineage">
    The Lineage tab renders a directed graph showing the upstream source tables the pipeline reads from and the downstream tables it produces.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/Databricks_customer-2.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=d8e27fa36389baa34da8fb2ac709cb52" alt="Databricks Customer 2" width="2831" height="1821" data-path="images/Databricks_customer-2.png" />
    </Frame>

    | **Direction**  | **What Prizm Shows**                              |
    | :------------- | :------------------------------------------------ |
    | **Upstream**   | Source tables and views the pipeline reads from   |
    | **Downstream** | Output tables and assets produced by the pipeline |

    Lineage for pipeline assets is extracted from the DLT `flow_definition` event log and the Databricks Jobs REST API.
  </Tab>

  <Tab title="Run">
    The Run tab shows the full execution history for the pipeline — every run Prizm has ingested within the lookback window, with status, duration, trigger, and environment context.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_pipeline_Run.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=794f7fc08ed75ecd0dc1a2c20c38de2f" alt="Databricks Pipeline Run" width="2831" height="1659" data-path="images/databricks_pipeline_Run.png" />
    </Frame>

    | **Column**      | **What Prizm Shows**                                                            |
    | :-------------- | :------------------------------------------------------------------------------ |
    | **Run ID**      | Unique identifier for this pipeline execution                                   |
    | **Status**      | Run outcome: SUCCESS, FAILED, CANCELLED, SKIPPED, RUNNING                       |
    | **Error**       | Error message or failure reason if the run did not succeed                      |
    | **Duration**    | Total elapsed time for the run                                                  |
    | **Trigger**     | What initiated the run — JOB\_TASK, RETRY\_ON\_FAILURE, MANUAL, SCHEDULED, etc. |
    | **Environment** | Databricks environment context (workspace, cluster)                             |
    | **Started At**  | Run start timestamp                                                             |

    Runs are filterable by status. Click any run row to expand the detail view showing task-level breakdown and error logs.
  </Tab>

  <Tab title="Table">
    The Table tab lists the data source tables that this pipeline reads from or writes to — providing direct traceability between the pipeline and the warehouse assets it touches.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_pipeline_table.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=cb7a806baa40efaf5d1d7a74d069a54a" alt="Databricks Pipeline Table" width="2829" height="1485" data-path="images/databricks_pipeline_table.png" />
    </Frame>

    | **Column** | **What Prizm Shows**                              |
    | :--------- | :------------------------------------------------ |
    | **Name**   | Fully qualified table name (catalog.schema.table) |

    Tables listed here are the report data source tables associated with this pipeline run context. This view complements the Lineage tab with a flat list format for quick reference.
  </Tab>

  <Tab title="Audit">
    The Audit tab records every metadata change made to the pipeline asset in Prizm — owner updates, description edits, status changes, and configuration modifications.

    <Frame>
      <img src="https://mintcdn.com/dqlabs/34mJ3e0hKylvX1jz/images/databricks_pipeline_Audit.png?fit=max&auto=format&n=34mJ3e0hKylvX1jz&q=85&s=e03faf6d03628ccf1e978f7aa219b409" alt="Databricks Pipeline Audit" width="2831" height="1659" data-path="images/databricks_pipeline_Audit.png" />
    </Frame>

    | **Column**   | **What Prizm Shows**                                                |
    | :----------- | :------------------------------------------------------------------ |
    | **Log Date** | Timestamp of the recorded action                                    |
    | **Action**   | Type of change (metadata update, owner change, status update, etc.) |
    | **Message**  | Human-readable detail of what changed and by whom                   |

    All records are immutable — entries cannot be edited or deleted from the UI.
  </Tab>
</Tabs>

## Next Steps

<CardGroup cols={2}>
  <Card title="Setup" icon="plug" href="/sources/databricks/setup">
    Connect Databricks to Prizm — prerequisites, authentication, and configuration.
  </Card>

  <Card title="FAQ" icon="circle-question" href="/sources/databricks/faq">
    Common questions about the Databricks connector.
  </Card>
</CardGroup>