> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Autonomous Intelligence
> Prizm's 5-level autonomous intelligence architecture that powers continuous, self-improving data management.
## Overview
Prizm's autonomous intelligence is built on a **5-level architecture** that progresses from basic metadata collection to sophisticated, self-directed action management. The system is designed to maximize automation while preserving meaningful human oversight at the right decision points.
***
## The 5-Level Architecture
### Level 1: Data Foundation Layer
The base layer stores all essential metadata components in the **MetaStore**:
| Component | Description |
| ---------------------- | ----------------------------------- |
| **T** (Tables) | Core table and view metadata |
| **O** (Objects) | Database objects and schemas |
| **L** (Lineage) | Data flow and dependency tracking |
| **U** (Usage) | Query frequency and access patterns |
| **P** (Performance) | Job execution and cost metrics |
| **C** (Cost) | Compute and storage cost signals |
| **B** (Business terms) | Semantic business vocabulary |
### Level 2: Data Intelligence Layer
* **Profile Snapshot** — Attribute-level profiling using a default 7-day window or percentage-based sampling to establish baseline data characteristics
* **Semantic Classification** — Automatically identifies and assigns business terms to data elements based on content and context analysis
### Level 3: Autonomous Decision Layer
* **Criticality Scoring** — Analyzes data assets to determine business importance and assigns priority levels for monitoring and governance
* **Schedule Intelligence** — Optimizes profiling frequency and resource allocation based on data criticality, change patterns, and system load
### Level 4: Recommendation Layer
* **Metric Recommendations** — Suggests appropriate quality metrics (standard and custom) based on data characteristics and usage patterns
* **Quality Engine (Q & CQ)** — Powers the recommendation system for both standard quality and custom quality metrics
* **AI Stewardship Queue** — The task queue and intelligence engine that orchestrates autonomous operations across the platform
### Level 5: Action Layer
Actions are categorized into three states based on confidence and risk:
Fully automated resolution — Prizm takes action without human intervention based on high-confidence signals.
Partial automation with human guidance — Prizm surfaces a recommendation and waits for steward approval.
Requires manual intervention — Issue is flagged for human investigation and resolution.
***
## Autonomous Intelligence Capabilities
### Data Quality Intelligence
* **Automated profiling** — Continuously scan data to identify patterns, anomalies, and statistical properties without manual intervention
* **Self-healing pipelines** — Detect and correct data quality issues based on predefined rules and ML models
* **Smart validation** — Apply contextual rules that adapt to changing data patterns and automatically flag inconsistencies
* **Drift detection** — Monitor and alert on changes in data distributions that might indicate quality issues
* **Anomaly detection** — Identify outliers and unusual patterns that may represent data quality problems
### Data Catalog Intelligence
* **Automated metadata extraction** — Extract technical metadata from data sources without human intervention
* **Business glossary suggestions** — Use NLP to suggest business terms and definitions based on data context
* **Auto-classification** — Categorize and tag datasets based on content analysis
* **Lineage inference** — Automatically trace data flows and dependencies across systems
* **Usage analytics** — Track how data assets are used and surface popular or related datasets
### Data Observability Intelligence
* **Predictive monitoring** — Forecast potential data pipeline failures before they occur
* **Root cause analysis** — Automatically identify the source of data incidents
* **Impact assessment** — Determine downstream effects of data issues without manual tracing
* **Intelligent alerting** — Prioritize notifications based on business impact and urgency
* **Self-optimizing thresholds** — Adjust monitoring parameters based on historical patterns and seasonality
### Semantic Intelligence
* **Relationship discovery** — Identify meaningful connections between data entities across sources
* **Context enrichment** — Automatically add business context to technical data elements
* **Semantic layer generation** — Create business-friendly views that abstract technical complexity
* **Knowledge graph maintenance** — Update entity relationships as data evolves
* **Natural language interfaces** — Enable data interaction through conversational queries
### Profile Scheduling Intelligence
* **Dynamic scheduling** — Automatically determine optimal profiling frequency based on data change rates and business criticality
* **Resource-aware execution** — Schedule profiling jobs during system low-usage periods to minimize performance impact
* **Change-triggered profiling** — Automatically initiate profiling when significant schema or data volume changes are detected
* **Intelligent batching** — Group related tables for concurrent profiling to optimize system resources
* **Adaptive time windows** — Adjust profiling schedules based on historical processing times and data volumes
***
## Further Optimization Strategies
Prizm's autonomous intelligence engine continuously improves its efficiency through:
Column prioritization — high-risk columns (keys, critical business fields) profile daily; low-risk columns profile weekly or monthly. Adaptive cadence increases frequency when recent drift is detected.
Mergeable sketches (HLL, KLL/TDigest, Top-k) are used throughout so historical data is never rescanned. Rolling baselines use a windowed merge strategy for O(log N) complexity.
Robust statistics (median + MAD instead of mean + stdev) reduce false positives. Multi-window alerting fires only when anomalies breach both short (1h) and long (24h) windows.
Intra-batch duplicates use exact matching. Cross-batch duplicates use HLL overlap estimation, with targeted sample queries triggered only when suspicion thresholds are crossed.
Schema snapshots are recorded daily. Alerts fire on new columns, type changes, and nullability flips, and are automatically routed to upstream pipeline owners via lineage.
Dynamic baselining: if a drift persists beyond a configurable threshold without measured business impact, the baseline is automatically updated to prevent permanent alert states.
***
## Sub-Features
AI-driven metric recommendations based on asset characteristics.
Business-context-aware quality recommendations mapped to KPIs.
Automated business glossary generation using organizational context.
Configure fully autonomous execution for trusted workflows.
***
## title: "Autonomous Intelligence" description: "The five-layer architecture that takes Prizm from raw metadata to fully autonomous action"
## Overview
Autonomous Intelligence is the architecture that lets Prizm progress from simply collecting metadata to actually **deciding and acting** — with human involvement concentrated at the points where it matters most, and routine operations handled entirely by the system.
## The five-layer autonomy architecture
Prizm's autonomy is built as progressive layers of intelligence — each layer depends on the one below it, and human interaction is concentrated at the top, while the system handles lower-level operations on its own.
The **MetaStore** is the core repository storing every essential metadata component: **T**ables, **O**bjects, **L**ineage, **U**sage statistics, **P**erformance, **C**ost, and **B**usiness terms. An **Extract Process** continuously feeds this store from connected sources.
* **Profile Snapshot** — builds attribute-level profiles using either a default 7-day window or percentage-based sampling to establish baseline data characteristics
* **Semantic Classification** — automatically assigns business terms to data elements based on content and context analysis
* **Criticality Scoring** — analyzes assets to determine business importance and assigns monitoring/governance priority
* **Schedule Intelligence** — optimizes profiling frequency and resource allocation based on criticality, change patterns, and system load
* **Metric Recommendations** — suggests standard and custom quality metrics based on data characteristics and usage patterns
* **Q and CQ Engines** — the quality (Q) and custom quality (CQ) metric engines that power recommendations
* **AI Stewardship** — the task queue and intelligence engine orchestrating autonomous operations across the platform (see [AI Stewardship](/platform/ai-stewardship))
A three-state system categorizes every outcome:
* **AI Completed** — fully automated resolution
* **Human Assisted** — partial automation with human guidance
* **Action Needed** — requires manual intervention
User interaction (the "Actor" in Prizm's architecture) is concentrated at the higher levels — reviewing recommendations and approving actions — while the system autonomously handles metadata extraction, profiling, and scoring underneath.
## Where autonomous intelligence shows up
Natural-language-driven metadata generation, recommendations, and workflows
Company and business context used to ground AI-generated content
Determines which assets deserve the most monitoring depth and remediation priority
Continuous statistical baselining of every attribute
Automatic business term assignment to technical data elements
Configuration for how much of the platform runs hands-off
## Autonomous AI Stewardship in data management
Autonomous intelligence expresses itself differently depending on which part of the platform it's operating in:
* **Automated profiling** — continuously scans data to identify patterns, anomalies, and statistical properties without manual intervention
* **Self-healing pipelines** — detects and corrects data quality issues in real time based on rules and ML models
* **Smart validation** — applies contextual rules that adapt to changing data patterns
* **Drift detection** — monitors and alerts on distribution changes that may signal quality issues
* **Anomaly detection** — identifies outliers and unusual patterns
* **Automated metadata extraction** — pulls technical metadata from sources without human intervention
* **Business glossary suggestions** — uses NLP to suggest terms and definitions from data context
* **Auto-classification** — categorizes and tags datasets based on content analysis
* **Lineage inference** — automatically traces data flows and dependencies
* **Usage analytics** — tracks how assets are used and suggests related or popular datasets
* **Predictive monitoring** — forecasts potential pipeline failures before they occur
* **Root cause analysis** — automatically identifies the source of incidents
* **Impact assessment** — determines downstream effects of data issues without manual tracing
* **Intelligent alerting** — prioritizes notifications by business impact and urgency
* **Self-optimizing thresholds** — adjusts monitoring parameters based on historical patterns and seasonality
* **Relationship discovery** — identifies meaningful connections between data entities across sources
* **Context enrichment** — automatically adds business context to technical elements
* **Semantic layer generation** — creates business-friendly views that abstract technical complexity
* **Knowledge graph maintenance** — updates entity relationships as data evolves
* **Natural language interfaces** — enables data interaction through conversational queries
* **Dynamic scheduling** — determines optimal profiling frequency from change rates and business criticality
* **Resource-aware execution** — schedules jobs during low-usage windows to minimize impact
* **Change-triggered profiling** — kicks off profiling automatically on significant schema/volume changes
* **Intelligent batching** — groups related tables for concurrent profiling
* **Adaptive time windows** — adjusts schedules based on historical processing times and volumes
* **Priority-based scheduling** — prioritizes critical assets based on usage and business impact
* **Dependency-aware sequencing** — schedules profiles in lineage-aware order
* **Failure recovery orchestration** — automatically reschedules failed jobs with intelligent retry logic
* **SLA monitoring** — tracks job performance against SLAs and adjusts schedules to stay compliant
* **Cross-system coordination** — synchronizes profiling with ETL processes and downstream consumption
## Continuous optimization roadmap
Autonomous Intelligence is designed to keep getting smarter and cheaper to run over time. Key optimization directions include:
| Area | Approach |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Smarter profiling** | Prioritize high-risk columns (keys, critical fields) for daily profiling; low-risk columns weekly/monthly. Adaptive cadence increases frequency after recent drift. |
| **Incremental sketches** | Use mergeable sketches (HLL, KLL/TDigest, Top-k) so historical data is never rescanned; windowed merges for rolling baselines in O(log N) instead of re-merging every blob. |
| **Noise reduction** | Base thresholds on median + MAD instead of mean + stdev; require anomalies to breach both a short (1h) and long (24h) window before alerting; use one-sided checks where appropriate (e.g. alert only if nulls increase). |
| **Smarter duplicate checks** | Exact matching for intra-batch duplicates; HLL-based overlap estimation for cross-batch duplicates, escalating to a targeted sample query only when suspicious. |
| **Schema & metadata awareness** | Daily schema snapshots feed new columns, type changes, and nullability flips into the same statistical baseline used for other observability signals — a change doesn't fire an alert automatically just by occurring; lineage-linked alert routing to the correct owners applies when an alert does fire. |
| **Cost & performance observability** | Track job duration, bytes scanned, and warehouse cost alongside data metrics to answer "data loaded fine — so why did the bill spike?" |
| **Partition awareness** | Profile at the partition level to catch localized skew (e.g. one region suddenly showing 90% nulls), while still rolling up into a single baseline. |
| **Storage optimization** | Keep raw daily profiles for 90 days, then compact into monthly rollups; keep full-resolution metrics for 30–60 days, then downsample for long-term trends. |
| **Explainability** | Attach "expected vs. actual" fields and sample records to every incident, reducing back-and-forth with data owners. |
| **Self-healing rules** | Dynamically update a baseline if drift persists beyond a threshold without incident impact — preventing a "permanent red light" state. |
## Related pages
How human oversight governs autonomous decisions at every level
The agent swarm that executes this autonomy architecture