Multi-Agent Architecture

​

Overview

Prizm is DQLabs’ AI-native control plane for data observability, data quality, and enterprise context. Rather than layering AI on top of a static monitoring tool, Prizm embeds AI directly into how context is gathered, how signals are interpreted, how priorities are set, and how remediation is orchestrated. The platform operates as a continuous loop — not a point-in-time scan — and every automated action is routed through a governance layer that matches the enterprise’s risk posture. Rather than relying on a single AI model, Prizm deploys a swarm of specialized agents that work collaboratively, each with a distinct role — mimicking a specialized data team operating 24/7. These agents communicate and coordinate through a shared orchestration layer, creating a system that’s greater than the sum of its parts. Figure 1. Prizm architecture: data sources feed the context layer, which feeds four role-driven agents running a closed-loop operating cycle, gated by a governance layer and wrapped in a deployment/security boundary.

​

Sources

Prizm connects to the enterprise data estate without requiring wholesale relocation of data: Processing note: Prizm’s agents operate primarily on metadata and lineage extracted from these sources. Source data is not required to be centralized or copied into a DQLabs-managed store, which is what allows the platform to support on-premises, VPC-resident, and hybrid-cloud sources.

​

Context Layer

The Semantic & Context Engine is the layer every downstream agent shares. It converts technical metadata into business meaning by:

• Mapping fields and tables to business terms
• Establishing data ownership
• Scoring criticality
• Linking assets to business KPIs

This layer updates continuously, learning from metadata, lineage, usage patterns, and the outcomes of prior actions. It is a persistent, shared context store — not a static configuration set up once at onboarding.

​

Agent Taxonomy

Prizm deploys nine specialized agents, each owning a distinct domain of the data trust lifecycle. A shared common layer (context loading, SQL execution, vector search, handoffs, etc.) is available to all agents but is not itself an agent. Separating responsibilities by agent allows automation to be configured independently per domain — for example, full automation for routine metrics runs while governance actions require human approval.

​

Closed-Loop Operating Cycle

The four agents execute a continuous four-stage cycle rather than a linear, one-way pipeline:

1. Understand & Learn — build and update context from metadata, lineage, and usage signals.
2. Evaluate Trust — measure data health in real time (detection).
3. Prioritize Impact — rank issues by business criticality and downstream reach to reduce noise.
4. Resolve & Act — drive remediation rather than stopping at alerting.

Output from Resolve & Act feeds back into Understand & Learn, so outcomes continuously refine the context layer. This feedback loop is the mechanism behind Prizm’s “AI-native” design, as distinct from static, rule-only tooling.

​

Governance and Human Oversight

Every action generated by the agent layer is routed into one of three operating modes based on confidence and risk: This tiered model allows an enterprise to adopt automation incrementally, rather than choosing between fully manual and fully automated operation.

​

Deployment and Security

​

Outcome

Together, these layers are designed to produce trusted, AI-ready data: fewer false alerts, faster mean-time-to-resolution, clear data ownership, safer grounding for downstream GenAI/ML use cases, and an auditable record of automated actions.

​

System architecture

​

Agent orchestration layer

• Prizm Agent (Master Controller) — the central conversational orchestrator, built on LangChain and LangGraph, responsible for context-aware processing, dynamic tool selection, and session management
• Q&A Summarize Agent — a specialized agent that transforms raw tool output into user-friendly, formatted responses
• Context Engine — advanced context management: relevance scoring, token optimization, and dynamic context loading

​

Control plane

• Orchestrator — plans, routes, and coordinates work across the agent swarm
• Policy Engine — enforces SLAs and guardrails (RBAC/ABAC)
• Prioritized Task Queue — risk-based scheduling so the highest-impact work runs first

Here’s how Prizm actually places compute relative to data. Push-down. Wherever a source has native query compute, Prizm keeps execution there instead of pulling data out. For e.g., The Databricks connector runs parameterized SQL directly against information_schema and system.* tables on the customer’s own SQL Warehouse/cluster; the Converse natural-language-to-SQL feature explicitly generates SQL that’s “executed as a push-down operation into your warehouse (e.g. Snowflake) through existing connector — all AI reasoning happens within Prizm, nothing is pushed to the warehouse beyond the resulting query.” The Cost agent takes this further with cost-aware push-down choices: DESCRIBE HISTORY reads the Delta transaction log (pure metadata, ~50–200ms, no executors) instead of SELECT COUNT(*) (a full Spark table scan), and Databricks customers can route conversion/observability jobs to run entirely as notebook Jobs on their own Databricks cluster rather than Prizm’s compute. Nothing is copied out; only credentials and job parameters cross the boundary. Sidecar. Each data source gets its own connector agent (prizm-adls-agent, a Snowflake agent, a Databricks agent, etc.) — small, independently deployed MCP services that own credential resolution and protocol translation for exactly one source type. They’re outbound-only over NATS JetStream (no inbound HTTP endpoint), registering with prizm-core/prizm-schedule via tool-call subjects and heartbeats. This is a sidecar pattern in the literal sense: in the hybrid deployment model, these agent processes can run inside the customer’s network, close to the data, while the control plane (UI, API, scheduling, policy) stays centralized — Control plane and UI can be centralized while execution plane runs closer to customer data. Centralized engine. When a source has no native query layer (flat files on ADLS/S3), Prizm falls back to its own shared compute: files get converted to Iceberg tables via a platform Spark cluster (prizm_spark_trino engine id), cataloged in Nessie, and queried through a centrally-hosted Trino layer for metric reads — one engine shared across tenants for that class of source, as an alternative to push-down when push-down isn’t available. Distributed. The AI/analytics tier that powers profiling, anomaly detection, and the wider agent swarm (asset discovery, DQ rules, observability, root cause, remediation agents, etc., coordinated by an Agent Orchestrator + event bus + prioritized task queue) runs on Ray/KubeRay, horizontally distributed across per-tenant node pools (ray-primary, ray-secondary, ray-third) inside Prizm’s AKS clusters. This scales elastically per tenant rather than as one monolithic process. These four aren’t mutually exclusive — they’re options the platform picks per source and per deployment model (Prizm-managed SaaS, customer-managed VPC/on-prem, or hybrid), all coordinated by the same control plane over NATS.

​

Knowledge & state layer

Every agent reads from and writes to a shared knowledge substrate, which is what allows the swarm to act coherently rather than as isolated bots:

​

LLM & reasoning layer

• LLM Gateway — manages models, prompts, and evaluations
• RAG Layer — retrieves context by combining the Knowledge Graph, Vector Store, and Catalog before an agent reasons or acts

​

Agent specifications

​

Tool ecosystem

All tools are built on FastMCP (Model Context Protocol), giving every tool a standardized interface, async-first design, robust error handling, and Pydantic-based type safety.

​

Memory & context management

Prizm uses a 4-layer hybrid memory system to give agents both short-term conversational context and long-term learned context: Session isolation is strict — every session requires both a session_id and user_id, with zero cross-session data leakage, supporting GDPR/CCPA-compliant multi-tenant operation. Performance benchmarks: Less than 200ms context load time, Less than 100ms semantic search, ~60% reduction in token usage from intelligent context pruning, ~90% context relevance score.

​

Workflow orchestration patterns

Agents operate in three interaction modes depending on the task:

​

Example: source management workflow

1. Source name collection — natural language input
2. Validation — mandatory duplicate and validity checking (cannot be skipped)
3. Source type selection — Snowflake, Databricks, or Oracle
4. Authentication configuration — type-specific auth setup
5. Source creation — final creation with all validated parameters

​

Example: metrics collection workflow

1. Query existing data first — never collect without checking what already exists
2. Parameter extraction from user input
3. Asset validation — confirm the asset exists and is accessible
4. Source validation — confirm connectivity and permissions
5. Metrics collection — execute with validated parameters

​

Technology stack

​

Related pages