Skip to main content

​
Overview

Prizm’s autonomous intelligence is built on a 5-level architecture that progresses from basic metadata collection to sophisticated, self-directed action management. The system is designed to maximize automation while preserving meaningful human oversight at the right decision points.

​
The 5-Level Architecture

​
Level 1: Data Foundation Layer

The base layer stores all essential metadata components in the MetaStore:

​
Level 2: Data Intelligence Layer

  • Profile Snapshot — Attribute-level profiling using a default 7-day window or percentage-based sampling to establish baseline data characteristics
  • Semantic Classification — Automatically identifies and assigns business terms to data elements based on content and context analysis

​
Level 3: Autonomous Decision Layer

  • Criticality Scoring — Analyzes data assets to determine business importance and assigns priority levels for monitoring and governance
  • Schedule Intelligence — Optimizes profiling frequency and resource allocation based on data criticality, change patterns, and system load

​
Level 4: Recommendation Layer

  • Metric Recommendations — Suggests appropriate quality metrics (standard and custom) based on data characteristics and usage patterns
  • Quality Engine (Q & CQ) — Powers the recommendation system for both standard quality and custom quality metrics
  • AI Stewardship Queue — The task queue and intelligence engine that orchestrates autonomous operations across the platform

​
Level 5: Action Layer

Actions are categorized into three states based on confidence and risk:

AI Completed

Fully automated resolution — Prizm takes action without human intervention based on high-confidence signals.

Human Assisted

Partial automation with human guidance — Prizm surfaces a recommendation and waits for steward approval.

Action Needed

Requires manual intervention — Issue is flagged for human investigation and resolution.

​
Autonomous Intelligence Capabilities

​
Data Quality Intelligence

  • Automated profiling — Continuously scan data to identify patterns, anomalies, and statistical properties without manual intervention
  • Self-healing pipelines — Detect and correct data quality issues based on predefined rules and ML models
  • Smart validation — Apply contextual rules that adapt to changing data patterns and automatically flag inconsistencies
  • Drift detection — Monitor and alert on changes in data distributions that might indicate quality issues
  • Anomaly detection — Identify outliers and unusual patterns that may represent data quality problems

​
Data Catalog Intelligence

  • Automated metadata extraction — Extract technical metadata from data sources without human intervention
  • Business glossary suggestions — Use NLP to suggest business terms and definitions based on data context
  • Auto-classification — Categorize and tag datasets based on content analysis
  • Lineage inference — Automatically trace data flows and dependencies across systems
  • Usage analytics — Track how data assets are used and surface popular or related datasets

​
Data Observability Intelligence

  • Predictive monitoring — Forecast potential data pipeline failures before they occur
  • Root cause analysis — Automatically identify the source of data incidents
  • Impact assessment — Determine downstream effects of data issues without manual tracing
  • Intelligent alerting — Prioritize notifications based on business impact and urgency
  • Self-optimizing thresholds — Adjust monitoring parameters based on historical patterns and seasonality

​
Semantic Intelligence

  • Relationship discovery — Identify meaningful connections between data entities across sources
  • Context enrichment — Automatically add business context to technical data elements
  • Semantic layer generation — Create business-friendly views that abstract technical complexity
  • Knowledge graph maintenance — Update entity relationships as data evolves
  • Natural language interfaces — Enable data interaction through conversational queries

​
Profile Scheduling Intelligence

  • Dynamic scheduling — Automatically determine optimal profiling frequency based on data change rates and business criticality
  • Resource-aware execution — Schedule profiling jobs during system low-usage periods to minimize performance impact
  • Change-triggered profiling — Automatically initiate profiling when significant schema or data volume changes are detected
  • Intelligent batching — Group related tables for concurrent profiling to optimize system resources
  • Adaptive time windows — Adjust profiling schedules based on historical processing times and data volumes

​
Further Optimization Strategies

Prizm’s autonomous intelligence engine continuously improves its efficiency through:
Column prioritization — high-risk columns (keys, critical business fields) profile daily; low-risk columns profile weekly or monthly. Adaptive cadence increases frequency when recent drift is detected.
Mergeable sketches (HLL, KLL/TDigest, Top-k) are used throughout so historical data is never rescanned. Rolling baselines use a windowed merge strategy for O(log N) complexity.
Robust statistics (median + MAD instead of mean + stdev) reduce false positives. Multi-window alerting fires only when anomalies breach both short (1h) and long (24h) windows.
Intra-batch duplicates use exact matching. Cross-batch duplicates use HLL overlap estimation, with targeted sample queries triggered only when suspicion thresholds are crossed.
Schema snapshots are recorded daily. Alerts fire on new columns, type changes, and nullability flips, and are automatically routed to upstream pipeline owners via lineage.
Dynamic baselining: if a drift persists beyond a configurable threshold without measured business impact, the baseline is automatically updated to prevent permanent alert states.

​
Sub-Features

Quality Metric Recommendation Agent

AI-driven metric recommendations based on asset characteristics.

Business Quality Recommendations

Business-context-aware quality recommendations mapped to KPIs.

Glossary Creation

Automated business glossary generation using organizational context.

Autonomous Mode

Configure fully autonomous execution for trusted workflows.

​
title: “Autonomous Intelligence” description: “The five-layer architecture that takes Prizm from raw metadata to fully autonomous action”

​
Overview

Autonomous Intelligence is the architecture that lets Prizm progress from simply collecting metadata to actually deciding and acting — with human involvement concentrated at the points where it matters most, and routine operations handled entirely by the system.
Prizm five-layer autonomy architecture

​
The five-layer autonomy architecture

Prizm’s autonomy is built as progressive layers of intelligence — each layer depends on the one below it, and human interaction is concentrated at the top, while the system handles lower-level operations on its own.
1

Level 1 — Data Foundation Layer

The MetaStore is the core repository storing every essential metadata component: Tables, Objects, Lineage, Usage statistics, Performance, Cost, and Business terms. An Extract Process continuously feeds this store from connected sources.
2

Level 2 — Data Intelligence Layer

  • Profile Snapshot — builds attribute-level profiles using either a default 7-day window or percentage-based sampling to establish baseline data characteristics
  • Semantic Classification — automatically assigns business terms to data elements based on content and context analysis
3

Level 3 — Autonomous Decision Layer

  • Criticality Scoring — analyzes assets to determine business importance and assigns monitoring/governance priority
  • Schedule Intelligence — optimizes profiling frequency and resource allocation based on criticality, change patterns, and system load
4

Level 4 — Recommendation Layer

  • Metric Recommendations — suggests standard and custom quality metrics based on data characteristics and usage patterns
  • Q and CQ Engines — the quality (Q) and custom quality (CQ) metric engines that power recommendations
  • AI Stewardship — the task queue and intelligence engine orchestrating autonomous operations across the platform (see AI Stewardship)
5

Level 5 — Action Layer

A three-state system categorizes every outcome:
  • AI Completed — fully automated resolution
  • Human Assisted — partial automation with human guidance
  • Action Needed — requires manual intervention
User interaction (the “Actor” in Prizm’s architecture) is concentrated at the higher levels — reviewing recommendations and approving actions — while the system autonomously handles metadata extraction, profiling, and scoring underneath.

​
Where autonomous intelligence shows up

GPT Experience

Natural-language-driven metadata generation, recommendations, and workflows

Organization Profile

Company and business context used to ground AI-generated content

Criticality Score

Determines which assets deserve the most monitoring depth and remediation priority

Data Profiling

Continuous statistical baselining of every attribute

Semantic Term

Automatic business term assignment to technical data elements

Autonomous Mode

Configuration for how much of the platform runs hands-off

​
Autonomous AI Stewardship in data management

Autonomous intelligence expresses itself differently depending on which part of the platform it’s operating in:
  • Automated profiling — continuously scans data to identify patterns, anomalies, and statistical properties without manual intervention
  • Self-healing pipelines — detects and corrects data quality issues in real time based on rules and ML models
  • Smart validation — applies contextual rules that adapt to changing data patterns
  • Drift detection — monitors and alerts on distribution changes that may signal quality issues
  • Anomaly detection — identifies outliers and unusual patterns
  • Automated metadata extraction — pulls technical metadata from sources without human intervention
  • Business glossary suggestions — uses NLP to suggest terms and definitions from data context
  • Auto-classification — categorizes and tags datasets based on content analysis
  • Lineage inference — automatically traces data flows and dependencies
  • Usage analytics — tracks how assets are used and suggests related or popular datasets
  • Predictive monitoring — forecasts potential pipeline failures before they occur
  • Root cause analysis — automatically identifies the source of incidents
  • Impact assessment — determines downstream effects of data issues without manual tracing
  • Intelligent alerting — prioritizes notifications by business impact and urgency
  • Self-optimizing thresholds — adjusts monitoring parameters based on historical patterns and seasonality
  • Relationship discovery — identifies meaningful connections between data entities across sources
  • Context enrichment — automatically adds business context to technical elements
  • Semantic layer generation — creates business-friendly views that abstract technical complexity
  • Knowledge graph maintenance — updates entity relationships as data evolves
  • Natural language interfaces — enables data interaction through conversational queries
  • Dynamic scheduling — determines optimal profiling frequency from change rates and business criticality
  • Resource-aware execution — schedules jobs during low-usage windows to minimize impact
  • Change-triggered profiling — kicks off profiling automatically on significant schema/volume changes
  • Intelligent batching — groups related tables for concurrent profiling
  • Adaptive time windows — adjusts schedules based on historical processing times and volumes
  • Priority-based scheduling — prioritizes critical assets based on usage and business impact
  • Dependency-aware sequencing — schedules profiles in lineage-aware order
  • Failure recovery orchestration — automatically reschedules failed jobs with intelligent retry logic
  • SLA monitoring — tracks job performance against SLAs and adjusts schedules to stay compliant
  • Cross-system coordination — synchronizes profiling with ETL processes and downstream consumption

​
Continuous optimization roadmap

Autonomous Intelligence is designed to keep getting smarter and cheaper to run over time. Key optimization directions include:

AI Stewardship

How human oversight governs autonomous decisions at every level

Multi-Agent Architecture

The agent swarm that executes this autonomy architecture