> ## Documentation Index > Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt > Use this file to discover all available pages before exploring further. # Multi-Agent Architecture > How Prizm's specialized AI agents coordinate to handle data management at scale. ## Overview Prizm is DQLabs' AI-native control plane for data observability, data quality, and enterprise context. Rather than layering AI on top of a static monitoring tool, Prizm embeds AI directly into how context is gathered, how signals are interpreted, how priorities are set, and how remediation is orchestrated. The platform operates as a continuous loop — not a point-in-time scan — and every automated action is routed through a governance layer that matches the enterprise's risk posture. Rather than relying on a single AI model, Prizm deploys a **swarm of specialized agents** that work collaboratively, each with a distinct role — mimicking a specialized data team operating 24/7. These agents communicate and coordinate through a shared orchestration layer, creating a system that's greater than the sum of its parts. Image *Figure 1. Prizm architecture: data sources feed the context layer, which feeds four role-driven agents running a closed-loop operating cycle, gated by a governance layer and wrapped in a deployment/security boundary.* ## Sources Prizm connects to the enterprise data estate without requiring wholesale relocation of data: | Source type | Examples | | :---------------------- | :------------------------------------- | | Warehouses / lakehouses | Snowflake, Databricks, BigQuery | | Pipelines | Airflow, dbt | | Catalogs | Collibra, Alation | | BI tools | Tableau, Power BI | | Unstructured content | Documents used for RAG/GenAI grounding | **Processing note:** Prizm's agents operate primarily on metadata and lineage extracted from these sources. Source data is not required to be centralized or copied into a DQLabs-managed store, which is what allows the platform to support on-premises, VPC-resident, and hybrid-cloud sources. ## Context Layer The Semantic & Context Engine is the layer every downstream agent shares. It converts technical metadata into business meaning by: * Mapping fields and tables to business terms * Establishing data ownership * Scoring criticality * Linking assets to business KPIs This layer updates continuously, learning from metadata, lineage, usage patterns, and the outcomes of prior actions. It is a persistent, shared context store — not a static configuration set up once at onboarding. ## Agent Taxonomy Prizm deploys nine specialized agents, each owning a distinct domain of the data trust lifecycle. A shared common layer (context loading, SQL execution, vector search, handoffs, etc.) is available to all agents but is not itself an agent. | Agent | Responsibility | Key tools | | :------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Alerts** | Issue management — links alerts to tickets and downstream notification channels | `manage_issue_links` | | **Asset** | Asset discovery and execution — scopes, searches, and triggers runs against assets | `list_assets_by_scope`, `search_assets`, `run_asset` | | **Chart** | Visualization — creates dashboards and data views from query results | `create_dashboard` | | **File** | Document intelligence — extracts governance entities and answers questions from uploaded documents | `extract_governance_entities`, `query_document` | | **Governance** | Semantic layer management — creates and recommends domains, glossaries, terms, products, tags, and categories; links or unlinks semantic context to assets | `create_domain`, `create_glossary`, `create_terms`, `recommend_terms`, `link_semantic_to_assets`, `unlink_semantic_from_assets`, `search_governance`, + 7 more | | **Lineage** | Data lineage — traces asset relationships upstream and downstream | `get_lineage` | | **Metrics** | Data quality metrics — creates, collects, runs, recommends, and mutes metric alerts | `create_metrics`, `collect_metrics`, `run_metric`, `recommend_business_metrics`, `manage_metric_alert_mute` | | **Schedule** | Job scheduling — manages execution schedules and controls schedule state | `manage_schedule`, `change_schedule_state` | | **Source** | Data source connectivity — creates and validates source connections | `create_source` | Separating responsibilities by agent allows automation to be configured independently per domain — for example, full automation for routine metrics runs while governance actions require human approval. ## Closed-Loop Operating Cycle The four agents execute a continuous four-stage cycle rather than a linear, one-way pipeline: 1. **Understand & Learn** — build and update context from metadata, lineage, and usage signals. 2. **Evaluate Trust** — measure data health in real time (detection). 3. **Prioritize Impact** — rank issues by business criticality and downstream reach to reduce noise. 4. **Resolve & Act** — drive remediation rather than stopping at alerting. Output from **Resolve & Act** feeds back into **Understand & Learn**, so outcomes continuously refine the context layer. This feedback loop is the mechanism behind Prizm's "AI-native" design, as distinct from static, rule-only tooling. ## Governance and Human Oversight Every action generated by the agent layer is routed into one of three operating modes based on confidence and risk: | Mode | Behavior | | :------------------ | :------------------------------------------------------------------------------- | | Autonomous | Executed automatically — reserved for low-risk, high-confidence actions | | Human / AI Assisted | AI proposes an action; a human approves before execution | | Action Needed | Routed for fully manual review, typically where policy or regulation requires it | This tiered model allows an enterprise to adopt automation incrementally, rather than choosing between fully manual and fully automated operation. ## Deployment and Security | Aspect | Detail | | :----------------- | :---------------------------------------------------------------------------------------------------- | | Deployment models | Cloud (available via AWS Marketplace), hybrid, or on-premises | | Access control | Role-based access control (Admin, Owner, User) | | Compliance posture | SOC 2– and HIPAA-aligned controls | | Data handling | Prioritizes processing of metadata and lineage over movement of raw records | | Connectors | Snowflake, Databricks, AWS, Azure, GCP, SAP to name a few. See the Data Sources page for more details | | | | ## Outcome Together, these layers are designed to produce trusted, AI-ready data: fewer false alerts, faster mean-time-to-resolution, clear data ownership, safer grounding for downstream GenAI/ML use cases, and an auditable record of automated actions. ## System architecture ### Agent orchestration layer * **Prizm Agent (Master Controller)** — the central conversational orchestrator, built on LangChain and LangGraph, responsible for context-aware processing, dynamic tool selection, and session management * **Q\&A Summarize Agent** — a specialized agent that transforms raw tool output into user-friendly, formatted responses * **Context Engine** — advanced context management: relevance scoring, token optimization, and dynamic context loading ### Control plane * **Orchestrator** — plans, routes, and coordinates work across the agent swarm * **Policy Engine** — enforces SLAs and guardrails (RBAC/ABAC) * **Prioritized Task Queue** — risk-based scheduling so the highest-impact work runs first Here's how Prizm actually places compute relative to data. Image **Push-down.** Wherever a source has native query compute, Prizm keeps execution there instead of pulling data out. For e.g., The Databricks connector runs parameterized SQL directly against `information_schema` and `system.*` tables on the customer's own SQL Warehouse/cluster; the Converse natural-language-to-SQL feature explicitly generates SQL that's "executed as a push-down operation into your warehouse (e.g. Snowflake) through existing connector — all AI reasoning happens within Prizm, nothing is pushed to the warehouse beyond the resulting query." The Cost agent takes this further with cost-aware push-down choices: `DESCRIBE HISTORY` reads the Delta transaction log (pure metadata, \~50–200ms, no executors) instead of `SELECT COUNT(*)` (a full Spark table scan), and Databricks customers can route conversion/observability jobs to run entirely as notebook Jobs on their own Databricks cluster rather than Prizm's compute. Nothing is copied out; only credentials and job parameters cross the boundary. **Sidecar.** Each data source gets its own connector agent (`prizm-adls-agent`, a Snowflake agent, a Databricks agent, etc.) — small, independently deployed MCP services that own credential resolution and protocol translation for exactly one source type. They're outbound-only over NATS JetStream (no inbound HTTP endpoint), registering with `prizm-core`/`prizm-schedule` via tool-call subjects and heartbeats. This is a sidecar pattern in the literal sense: in the hybrid deployment model, these agent processes can run inside the customer's network, close to the data, while the control plane (UI, API, scheduling, policy) stays centralized — Control plane and UI can be centralized while execution plane runs closer to customer data. **Centralized engine.** When a source has no native query layer (flat files on ADLS/S3), Prizm falls back to its own shared compute: files get converted to Iceberg tables via a platform Spark cluster (`prizm_spark_trino` engine id), cataloged in Nessie, and queried through a centrally-hosted Trino layer for metric reads — one engine shared across tenants for that class of source, as an alternative to push-down when push-down isn't available. **Distributed.** The AI/analytics tier that powers profiling, anomaly detection, and the wider agent swarm (asset discovery, DQ rules, observability, root cause, remediation agents, etc., coordinated by an Agent Orchestrator + event bus + prioritized task queue) runs on Ray/KubeRay, horizontally distributed across per-tenant node pools (`ray-primary`, `ray-secondary`, `ray-third`) inside Prizm's AKS clusters. This scales elastically per tenant rather than as one monolithic process. These four aren't mutually exclusive — they're options the platform picks per source and per deployment model (Prizm-managed SaaS, customer-managed VPC/on-prem, or hybrid), all coordinated by the same control plane over NATS. ### Knowledge & state layer Every agent reads from and writes to a shared knowledge substrate, which is what allows the swarm to act coherently rather than as isolated bots: | Component | Purpose | | --------------------------------- | ------------------------------------------------------------ | | **Unified Catalog** | Assets, owners, tags | | **Semantic Layer** | Business terms, KPIs, business context | | **Knowledge Graph** | Relationships and lineage across entities | | **Vector Store** | Runbooks, incident history, documentation for retrieval | | **Config Store** | Tenant settings, rules, thresholds | | **Operational Observability Log** | Single unified table — the source of truth for all telemetry | ### LLM & reasoning layer * **LLM Gateway** — manages models, prompts, and evaluations * **RAG Layer** — retrieves context by combining the Knowledge Graph, Vector Store, and Catalog before an agent reasons or acts ## Agent specifications The central orchestrator for all system interactions. * **LangChain integration** for proven conversational AI orchestration * **Context-aware processing** — maintains conversation history and current-asset context * **Dynamic tool selection** — intelligently routes requests to the right tool * **Session management** — complete session isolation and state tracking * **Model**: GPT-4o-nano · Temperature 0.1 · Max tokens 4096 · Streaming enabled · 30s timeout * **Performance**: Less than 2s response time for simple queries, 15+ integrated tools, 100% session isolation Converts raw tool output into concise, user-friendly responses. * **Dynamic table formatting** — LLM-driven selection of the 3–6 most relevant fields * **Context-aware summarization** using shared session memory * **Model**: GPT-4.1-nano · Temperature 0.1 · Max tokens 2048 * Truncates long values to 50 characters and falls back gracefully to raw data on formatting errors Manages context strategy and optimization across the agent swarm. * Context type analysis to determine the optimal retrieval strategy * Token optimization via intelligent pruning * Context-aware prompt engineering * Deep integration with the hybrid memory system ## Tool ecosystem All tools are built on **FastMCP** (Model Context Protocol), giving every tool a standardized interface, async-first design, robust error handling, and Pydantic-based type safety. | Category | Example Tools | Purpose | | ---------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | **Asset Management** | `list_assets`, `lookup_assets`, `get_asset_info`, `query_table_info`, `query_attribute_info` | Discovery, fuzzy search, and metadata enrichment across Snowflake, Databricks, PostgreSQL | | **Metrics Collection** | `query_metrics`, `collect_metrics` | Query-first data quality metrics (volume, freshness, schema, profile) | | **Source Management** | `validate_source`, `create_source`, `list_sources` | Guided, validated connection setup for Snowflake, Databricks, Oracle | | **Workflow Tools** | `start_connection_workflow`, `execute_query` | Multi-step process automation with parallel processing and fault tolerance | | **Web Search** | `web_search` | External knowledge lookup for questions unrelated to Prizm's own data — via Google Serper API | Asset and metrics tools support 15–20+ filter parameters each (source, asset type, database, schema, status, and more), with query response times typically under 500ms. ## Memory & context management Prizm uses a **4-layer hybrid memory system** to give agents both short-term conversational context and long-term learned context: | Layer | Type | Purpose | | --------------------- | ---------- | -------------------------------------------------------------------------------- | | **Buffer Memory** | Short-term | Last 10 conversation turns, \~2000 token limit | | **Entity Memory** | Structured | Automatic entity extraction and relationship mapping | | **Persistent Memory** | Long-term | Session-isolated PostgreSQL storage for entities, parameters, and workflow state | | **Vector Memory** | Semantic | Embedding-based retrieval for runbooks, incidents, and documentation | **Session isolation** is strict — every session requires both a `session_id` and `user_id`, with zero cross-session data leakage, supporting GDPR/CCPA-compliant multi-tenant operation. **Performance benchmarks**: Less than 200ms context load time, Less than 100ms semantic search, \~60% reduction in token usage from intelligent context pruning, \~90% context relevance score. ## Workflow orchestration patterns Agents operate in three interaction modes depending on the task: **Purpose**: Natural dialogue for explanations and insights **Flow**: User Query → Context Analysis → Direct Response **Use cases**: General questions, clarifications, data insights **Purpose**: Structured tool execution with parameter collection **Flow**: User Intent → Parameter Extraction → Validation → Tool Execution → Response **Use cases**: Data collection, source configuration, metrics gathering **Purpose**: Combines conversational and workflow elements **Flow**: Context-Aware Processing → Dynamic Tool Selection → Intelligent Response **Use cases**: Complex multi-step operations, context-dependent actions ### Example: source management workflow 1. **Source name collection** — natural language input 2. **Validation** — mandatory duplicate and validity checking (cannot be skipped) 3. **Source type selection** — Snowflake, Databricks, or Oracle 4. **Authentication configuration** — type-specific auth setup 5. **Source creation** — final creation with all validated parameters ### Example: metrics collection workflow 1. **Query existing data first** — never collect without checking what already exists 2. **Parameter extraction** from user input 3. **Asset validation** — confirm the asset exists and is accessible 4. **Source validation** — confirm connectivity and permissions 5. **Metrics collection** — execute with validated parameters ## Technology stack | Layer | Technologies | | ---------------------- | --------------------------------------------------------------------------- | | **Core framework** | LangChain, FastAPI, Pydantic, SQLAlchemy | | **AI/ML** | OpenAI API/ Anthropic Claude, text-embedding-3-small, LangChain Memory, MCP | | **Database & storage** | PostgreSQL (metadata), Snowflake, Databricks, session storage | | **External services** | Google Serper API (web search), OpenAI API (LLM services) | The same orchestration and tool infrastructure that powers the agent swarm also powers [Converse](/platform/converse) — Prizm's conversational interface is the Prizm Agent talking directly to you. ## Related pages See how these agents fit into Prizm's five-layer autonomy architecture How human oversight governs what these agents are allowed to do