> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Lineage Extraction
> How Prizm extracts, models, and uses table and column-level lineage, including multi-hop lineage, the lineage data model, and derived metrics.
## Implementation Approaches
Prizm offers multiple methods to capture and maintain lineage:
Connectors parse transformation code, extract query logs, and analyze job metadata automatically.
Direct integration with ETL tools, orchestration platforms, and transformation engines (dbt, Airflow, ADF).
Programmatic lineage updates from custom applications and processes via the Prizm API.
Tools for data stewards to document and verify lineage relationships where automation cannot reach.
Prizm uses extracted lineage data for various purposes as listed below but not limited to
### Context Enrichment
Lineage data enhances metadata with contextual information:
* **Business Context** — Connects technical lineage with business processes, domains, and data products
* **Usage Patterns** — Overlays access statistics and query patterns onto lineage paths and to understand how data flows changed over time.
* **Pipeline Metadata** — Enriches lineage with job execution metrics, refresh frequency, and processing duration
### Cataloging Integration
* **Discoverability** — Enhances search and discovery by revealing related assets through lineage connections
* **Criticality Scoring** — Understand which assets are most critical to downstream business processes.
* **Data Asset Graph** — Builds a comprehensive knowledge graph of all data relationships
* **Impact Documentation** — Automatically documents dependencies for governance and compliance
### Observability Enhancement
* **Anomaly Correlation** — Links anomalies across related datasets to identify common root causes
* **SLA Monitoring** — Traces cascading delays through pipeline dependencies
* **Impact Analysis** — Compute blast radius before making changes to any upstream asset.
* **Freshness Tracking** — Monitors data currency across transformation stages
### Quality Recommendations
Lineage intelligence drives proactive quality management:
* **Targeted Testing** — Suggests where to implement quality checks based on criticality and impact analysis
* **Preventive Monitoring** — Identifies upstream assets requiring heightened monitoring to prevent downstream issues
* **Pattern Recognition** — Detects recurring quality issues across lineage paths to recommend systemic improvements
* **Risk Prioritization** — Focuses quality efforts on high-impact, high-risk data assets
***
### Use API first when available
| Asset type | Extraction method |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Tables, Views and materialized views | **Get Lineage** — queries the source system's access history or query log (e.g. `snowflake.account_usage.access_history`) to identify which tables were read and written together |
| When not available via API | **Object Dependency** — reads view definitions directly from the source system's metadata (e.g. `SHOW VIEWS`, `information_schema.view_definition`) |
### Attribute lineage (Snowflake as an example)
For Snowflake sources, Prizm extracts column-level lineage by using the API. If needed when not availble via API, the platform has the ability to read `access_history` — a system table that records which columns were accessed and modified by each query. The extraction job:
1. Queries `access_history` for a defined time window, filtering to assets within the configured scope
2. Flattens `objects_modified` to find target columns and their `directSources` / `baseSources`
3. Builds a row per source-column → target-column pair with source and target asset context
4. Deduplicates and stores the edges in the Prizm lineage table
5. Runs both **downstream** (source → target) and **upstream** (target ← source) recursive CTEs to depth 10 to build the full multi-hop graph
However this process can be time consuming and expensive so Prizm by default prefers using API and pull only when needed based on DDL operation. For DDL operations (CREATE VIEW, ALTER VIEW), Prizm additionally parses `object_modified_by_ddl` to capture view-level column lineage that is not recorded in `objects_modified`.
***
## The lineage data model
Every lineage relationship is stored as an **edge** in the `lineage` table with references to the source and target asset IDs.
### Core lineage table
| Field | Type | Description |
| ----------------- | --------- | -------------------------------------------------------------------------------------- |
| `lineage_id` | uuid | Primary key — uniquely identifies each lineage relationship |
| `from_id` | uuid | Source asset or attribute ID (upstream) |
| `to_id` | uuid | Target asset or attribute ID (downstream) |
| `from_asset_type` | string | Type of the source — `table`, `view`, `attribute`, `dbt_attribute`, `dashboard`, etc. |
| `to_asset_type` | string | Type of the target |
| `relation` | enum | Relationship type — `depends_on`, `derived_from`, `transforms`, `copies`, `aggregates` |
| `transform_type` | enum | Language of the transformation logic — `SQL`, `Python`, `Procedure`, `Function` |
| `transform_logic` | text | The actual transformation expression — e.g. `LOWER(TRIM(email))` |
| `version` | float | Version of the lineage relationship — incremented when the relationship changes |
| `created_at` | timestamp | When this lineage edge was first recorded |
| `modified_at` | timestamp | When this lineage edge was last updated |
| `created_by` | uuid | User who created the record |
| `modified_by` | uuid | User who last modified the record |
### Supported asset types
| Asset type | Description |
| ----------- | ----------------------------------------------- |
| `table` | Database tables in any connected source |
| `view` | Database views and materialized views |
| `pipeline` | ETL or ELT jobs, Airflow DAGs, dbt models |
| `model` | dbt models, ML models, semantic models |
| `dashboard` | BI dashboards (Tableau, Looker, Power BI, etc.) |
| `report` | Business reports and exports |
| `api` | API endpoints that read or write data |
| `file` | CSV, Parquet, and other file-based data assets |
### Column lineage view
Prizm builds a `column_lineage_with_assets_view` that joins the lineage edges to attribute metadata, producing human-readable source/target column and table names:
| Field | Description |
| ---------------- | ---------------------------------------------- |
| `source_column` | Name of the source column |
| `source_asset` | Name of the source table or model |
| `source_type` | Asset type of the source |
| `target_column` | Name of the target column |
| `target_asset` | Name of the target table or model |
| `target_type` | Asset type of the target |
| `transformation` | The relation type or transformation expression |
***
## Multi-hop lineage
Prizm builds a **recursive materialized view** (`column_lineage_recursive_mview`) that resolves chains of any depth — not just direct one-hop relationships. Starting from a base set of direct edges, the recursive CTE follows each `to_id` as a new `from_id` until all paths are exhausted or a cycle is detected.
The output includes a `level` field (the hop count) and a `path` array (the UUIDs traversed), making it possible to:
* Find all columns ultimately derived from a given source column, regardless of how many transformations sit between them
* Identify the full blast radius of removing or changing a source column
* Understand the complete provenance chain of a target column with one query
**Example — three-hop chain:**
| Level | Source column | Source asset | Target column | Target asset | Transformation |
| ----- | ------------- | ------------ | ------------- | ------------ | --------------- |
| 1 | customer\_id | raw\_orders | customer\_id | stg\_orders | direct mapping |
| 2 | customer\_id | raw\_orders | customer\_id | fct\_orders | propagate |
| 3 | customer\_id | raw\_orders | customer\_id | orders\_view | final transform |
***
## Lineage metrics
Prizm derives the following metrics directly from the lineage graph, surfaced on every asset's overview page:
| Metric | Level | Description |
| -------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Upstream Count | Asset | Number of direct upstream assets feeding this asset |
| Downstream Count | Asset | Number of direct downstream assets consuming this asset |
| Impact Score | Asset | A weighted score of how many assets would be affected if this asset changes — higher fan-out and deeper downstream depth increase the score |
| Dependency Depth | Asset | The maximum number of transformation hops between the deepest upstream source and this asset |
| Lineage Completeness | Global | Percentage of all registered assets that have at least one lineage relationship recorded |
***
## Business Value
| Benefit | Description |
| :------------------------ | :----------------------------------------------------------------------------------- |
| **Reduced MTTR** | Faster incident resolution through precise impact and root cause identification |
| **Enhanced Governance** | Clear visibility into data flow for regulatory compliance and privacy management |
| **Accelerated Migration** | Comprehensive dependency mapping to support cloud migration and system modernization |
| **Trust Building** | Increased confidence in data through transparent provenance tracking |