Lineage Extraction

​

Implementation Approaches

Prizm offers multiple methods to capture and maintain lineage: Prizm uses extracted lineage data for various purposes as listed below but not limited to

​

Context Enrichment

Lineage data enhances metadata with contextual information:

• Business Context — Connects technical lineage with business processes, domains, and data products
• Usage Patterns — Overlays access statistics and query patterns onto lineage paths and to understand how data flows changed over time.
• Pipeline Metadata — Enriches lineage with job execution metrics, refresh frequency, and processing duration

​

Cataloging Integration

• Discoverability — Enhances search and discovery by revealing related assets through lineage connections
• Criticality Scoring — Understand which assets are most critical to downstream business processes.
• Data Asset Graph — Builds a comprehensive knowledge graph of all data relationships
• Impact Documentation — Automatically documents dependencies for governance and compliance

​

Observability Enhancement

• Anomaly Correlation — Links anomalies across related datasets to identify common root causes
• SLA Monitoring — Traces cascading delays through pipeline dependencies
• Impact Analysis — Compute blast radius before making changes to any upstream asset.
• Freshness Tracking — Monitors data currency across transformation stages

​

Quality Recommendations

Lineage intelligence drives proactive quality management:

• Targeted Testing — Suggests where to implement quality checks based on criticality and impact analysis
• Preventive Monitoring — Identifies upstream assets requiring heightened monitoring to prevent downstream issues
• Pattern Recognition — Detects recurring quality issues across lineage paths to recommend systemic improvements
• Risk Prioritization — Focuses quality efforts on high-impact, high-risk data assets

---

​

Use API first when available

​

Attribute lineage (Snowflake as an example)

For Snowflake sources, Prizm extracts column-level lineage by using the API. If needed when not availble via API, the platform has the ability to read access_history — a system table that records which columns were accessed and modified by each query. The extraction job:

1. Queries access_history for a defined time window, filtering to assets within the configured scope
2. Flattens objects_modified to find target columns and their directSources / baseSources
3. Builds a row per source-column → target-column pair with source and target asset context
4. Deduplicates and stores the edges in the Prizm lineage table
5. Runs both downstream (source → target) and upstream (target ← source) recursive CTEs to depth 10 to build the full multi-hop graph

However this process can be time consuming and expensive so Prizm by default prefers using API and pull only when needed based on DDL operation. For DDL operations (CREATE VIEW, ALTER VIEW), Prizm additionally parses object_modified_by_ddl to capture view-level column lineage that is not recorded in objects_modified.

---

​

The lineage data model

Every lineage relationship is stored as an edge in the lineage table with references to the source and target asset IDs.

​

Core lineage table

​

Supported asset types

​

Column lineage view

Prizm builds a column_lineage_with_assets_view that joins the lineage edges to attribute metadata, producing human-readable source/target column and table names:

---

​

Multi-hop lineage

Prizm builds a recursive materialized view (column_lineage_recursive_mview) that resolves chains of any depth — not just direct one-hop relationships. Starting from a base set of direct edges, the recursive CTE follows each to_id as a new from_id until all paths are exhausted or a cycle is detected. The output includes a level field (the hop count) and a path array (the UUIDs traversed), making it possible to:

• Find all columns ultimately derived from a given source column, regardless of how many transformations sit between them
• Identify the full blast radius of removing or changing a source column
• Understand the complete provenance chain of a target column with one query

Example — three-hop chain:

---

​

Lineage metrics

Prizm derives the following metrics directly from the lineage graph, surfaced on every asset’s overview page:

---

​

Business Value