Why Connect Databricks to Prizm?
Connecting Databricks gives Prizm access to three layers of intelligence: Catalog & Context Prizm discovers every catalog, schema, table, view, and column in your Unity Catalog workspace and indexes them in the Prizm catalog. Tags defined in Databricks are imported automatically. Descriptions, owners, and classifications can be managed in Prizm. Data Quality & Profiling Prizm runs profile scans on your Delta tables to compute null rates, cardinality, min/max, distribution, and completeness scores at the column level. Quality scores are tracked over time so you can see trends and catch degradation before it reaches consumers. Observability Prizm monitors every in-scope table for freshness (latest Delta commit timestamp), volume (row count changes), and schema drift (added, removed, or renamed columns). Machine-learning anomaly detection sets adaptive thresholds so alerts fire on real deviations — not noise.Metric Support
Every metric in Prizm belongs to a context — the stakeholder lens that defines who the metric serves and why it matters. All seven contexts are supported for Databricks.- Operational
- Performance
- Structural
- Business
- Reconciliation
- Semantics
- Pipeline
Operational metrics monitor the day-to-day health of data assets — whether data is arriving on time, in the expected volume, and with the correct structure. They run at the Asset level and are the primary driver of alerting and anomaly detection in Prizm.
All Operational metrics feed directly into the Alerts dashboard. They are the most actively monitored metrics in a typical Prizm deployment, with Execution Status and Freshness generating the highest alert volumes in production.
| Metric | Subcategory | Dimension | Level | Monitor | Description |
|---|---|---|---|---|---|
| Row | Volume | Completeness | Asset | Yes | Total number of rows at the time of last scan. Tracks volume over time and triggers alerts on drops or spikes. |
| Volume | Volume | Completeness | Asset | Yes | Higher-level measure combining row count and data size signals. Used to detect significant changes in the amount of data arriving. |
| Data size | Volume | Completeness | Asset | Yes | Physical size of the asset in bytes. Complements row count — a large size increase with no row increase may indicate wide or bloated records. |
| Freshness | Freshness | Availability | Asset | Yes | Time elapsed since the asset was last updated. Fires an alert when data has not been refreshed within the expected window. |
| Last updated | Freshness | Timeliness | Asset | Yes | Timestamp of the most recent data load or write. Verifies pipelines are running on schedule and consumers are not working with stale records. |
| Column | Schema | Validity | Asset | Yes | Current number of columns. Detects column additions or removals indicating an unannounced schema change at the source. |
| Schema | Schema | Validity | Asset | Yes | Snapshot of the full column schema — names, types, and order. Alerts when any column name or type changes, which can silently break downstream transformations. |
| Schema name | Schema | Validity | Asset | Yes | Name of the schema the asset belongs to. Tracks unexpected schema renames that would break hard-coded references. |
| Duplicates | Duplicate | Uniqueness | — | Yes | Count of duplicate rows based on a defined key or all columns. Unexpected duplicates often indicate a pipeline re-run without deduplication or broken merge logic. |
Operational metrics do not contribute to trust scoring (
Score: No). They are observability signals — designed to trigger alerts, not to roll up into a quality score.Performance metrics measure how efficiently the data platform itself is running — query execution time, compute utilization, job duration, and credit consumption. Unlike other contexts, they operate at the Source and System level (Warehouse, Database, Account) rather than the individual asset level.
| Metric | Level | Profile Type | Description |
|---|---|---|---|
| Total queries per day | Warehouse, Database | Performance | Total queries executed per day. Tracks usage trends and surfaces unexpected spikes from runaway jobs or inefficient query patterns. |
| Query success rate | Warehouse, Database | Performance | Percentage of queries completing without error. A falling rate signals pipeline instability, permission issues, or timeouts. |
| Execution time | Warehouse, Database | Performance | Average query execution time. Rising times indicate compute pressure, missing indexes, or poorly optimised SQL. |
| Job duration | Warehouse, Database | Performance | Total elapsed time for a data job or pipeline run. Monitors whether batch jobs complete within their SLA windows. |
| Test execution time | Warehouse, Database | Performance | Time to run data quality test suites. Tracks whether test runtime grows as asset counts increase. |
| Avg blocked | Warehouse, Database | Performance | Average queries in a blocked state — waiting on locks or resource conflicts. High counts indicate concurrency issues or long-running transactions. |
| Avg queue load | Warehouse, Database | Performance | Average queries waiting in the execution queue. A persistently high value means the warehouse is under-provisioned for current workload. |
| Avg running | Warehouse, Database | Performance | Average queries actively running at any given time. Combined with queue load, gives a full picture of warehouse throughput. |
| Database storage | Database | Performance | Total storage consumed by the database including tables, indexes, and staging data. Tracks growth trends and flags databases approaching capacity thresholds. |
| Total credits per day | Warehouse, Database | Cost | Total compute credits consumed per day. Primary metric for cloud cost management — used to set budgets and trigger cost alerts. |
| Credits used | Warehouse, Database | Cost | Cumulative credits over a defined period. Used for chargeback reporting and team-level cost allocation. |
Performance metrics do not contribute to trust scoring (
Score: No). They are platform observability signals for infrastructure and platform teams.Structural metrics profile the internal shape of data at the column level — how values are distributed, how often patterns repeat, and what the numbers say. This is the largest context with 33K+ metrics. All structural metrics are auto-generated when an asset is profiled — no configuration required.Distribution — How values are composed and spread:
Frequency — How often values, lengths, and ranges repeat:
Pattern — Structural format templates:
Statistics — Numeric summaries:
| Metric | Subcategory | Dimension | Profile Type | Description |
|---|---|---|---|---|
| Null count | Completeness | Completeness | Essential | Total null values. High null count signals missing data that may break downstream logic or skew analysis. |
| Empty | Completeness | Completeness | Essential | Count of empty strings (""). Distinct from null — present but containing no meaningful content. |
| Non-empty | Completeness | Completeness | Advanced | Count of values that are neither null nor empty. Baseline for how much of the column actually has content. |
| Space | Completeness | Completeness | Essential | Values containing only whitespace. Appear non-empty but carry no data — a common data entry artefact. |
| Zero value | Numeric | Completeness | Essential | Numeric values equal to zero. Flags columns where zero may be a placeholder for missing data. |
| Distinct | Uniqueness | Uniqueness | Essential | Count of unique values. Low distinct counts on a high-cardinality column can indicate data collapse. |
| Repeating | Uniqueness | Uniqueness | Advanced | Count of values appearing more than once. High repeating counts on columns expected to be unique (e.g. IDs) signal duplicates. |
| Leading space | Space | Validity | Advanced | Values with spaces at the start. Can cause join failures and mismatches in downstream queries. |
| Trailing space | Space | Validity | Advanced | Values with spaces at the end. A frequent cause of silent quality issues in string comparisons. |
| Inner space | Space | Validity | Advanced | Values containing consecutive internal spaces (e.g. "John Doe"). Indicates formatting artefacts. |
| Outer space | Space | Validity | Advanced | Values with spaces at either end — combined view of leading and trailing space issues. |
| Whitespace | Space | Validity | Advanced | Values consisting entirely of whitespace characters (spaces, tabs, newlines). |
| Character | Character | Validity | Advanced | Distribution of character types — letters, digits, special characters, and mixed. |
| Alpha numeric | Character | Validity | Advanced | Values containing only alphabetic and numeric characters. Validates ID and code columns. |
| Digit | Character | Validity | Advanced | Values composed entirely of numeric digits. Useful for columns stored as strings (e.g. zip codes, phone numbers). |
| Special character | Character | Validity | Advanced | Values containing at least one special character (e.g. @, #, !). |
| Alphabet | Character | Validity | Advanced | Values containing only alphabetic characters — no digits or special characters. |
| Positive | Numeric | Validity | Advanced | Count of numeric values greater than zero. |
| Negative | Numeric | Validity | Advanced | Count of numeric values less than zero. Unexpected negatives in columns like age or price typically indicate bad data. |
| Metric | Subcategory | Dimension | Profile Type | Description |
|---|---|---|---|---|
| Enum | Value | Validity | Advanced | Distinct values and their frequencies — a value histogram. Reveals category distribution and flags values outside an expected allowed list. |
| Min value | Value | Validity | Essential | Smallest value in the column. Establishes the lower bound and flags records below expected thresholds. |
| Max value | Value | Validity | Essential | Largest value in the column. Detects outliers or values exceeding business-defined limits. |
| Value range | Range | Validity | Advanced | Spread between minimum and maximum values. Sudden changes signal data drift or pipeline contamination. |
| Min length | Length | Validity | Essential | Shortest string length found. Catches truncated values — e.g. a phone number column where some values are unexpectedly short. |
| Max length | Length | Validity | Essential | Longest string length found. Detects values exceeding expected character limits that can cause issues in downstream systems. |
| Length | Length | Validity | Advanced | Full distribution of string lengths. Reveals whether values cluster around a consistent length or vary widely. |
| Length range | Range | Validity | Advanced | Spread between minimum and maximum string lengths. A wide range on a uniform column (e.g. country codes) signals a format problem. |
| Metric | Subcategory | Dimension | Profile Type | Description |
|---|---|---|---|---|
| Short pattern | Pattern | Validity | Advanced | Condensed format — collapses consecutive identical character types into a single token (e.g. "John" → A). Useful for spotting high-level format anomalies quickly. |
| Long pattern | Pattern | Validity | Advanced | Character-by-character format — preserves full length and composition (e.g. "John" → AAAA). Reveals length drift and mixed-format issues. |
| Regular expressions | Pattern | Validity | Essential | User-defined regex patterns to validate that column values conform to a known format — e.g. email addresses, phone numbers, or custom ID formats. |
Use Short pattern for a quick high-level view of format variety in a column, and Long pattern when you need to detect subtle length or composition changes — for example, a country code column that sometimes contains 3 characters instead of 2.
| Metric | Subcategory | Dimension | Profile Type | Description |
|---|---|---|---|---|
| Mean | Descriptive | Accuracy | Advanced | Arithmetic average of all non-null values. A shift in the mean is one of the earliest signals of data drift or a change in the upstream source. |
| Median | Descriptive | Accuracy | Advanced | Middle value when sorted. Less sensitive to outliers than the mean — useful for skewed distributions like revenue or transaction amounts. |
| Mode | Descriptive | Accuracy | Advanced | Most frequently occurring value. A sudden change can indicate a default value being injected or a new dominant category appearing. |
| Standard deviation | Descriptive | Accuracy | Advanced | Spread of values around the mean. A rising stddev indicates increasing variability — often a sign of data quality degradation. |
| Sum | Descriptive | Accuracy | Advanced | Total of all non-null numeric values. Useful for financial columns where totals should remain consistent across pipeline stages. |
| Q1 | Quantile | Accuracy | Advanced | 25th percentile — 25% of values fall below this point. Used with Q3 to define the interquartile range. |
| Q3 | Quantile | Accuracy | Advanced | 75th percentile — 75% of values fall below this point. Together with Q1 defines the middle 50% of the data distribution. |
| Range | Shape | Accuracy | Advanced | Difference between max and minimum values. Quick measure of the total spread of the data in a column. |
| Skewness | Shape | Accuracy | Advanced | Asymmetry of the distribution. Positive skew = long right tail; negative skew = long left tail. Useful for detecting non-normal distributions. |
| Kurtosis | Shape | Accuracy | Advanced | Tail weight of the distribution. High kurtosis indicates more extreme outliers than a normal distribution. |
| Variance | Shape | Accuracy | Advanced | Average of squared differences from the mean. Used in statistical tests and anomaly detection models. |
| Margin of error | Shape | Accuracy | Advanced | Uncertainty estimate in the sample statistics. Indicates reliability when profiling on a sample rather than the full dataset. |
Business metrics are user-defined measurements aligned to domain rules, KPIs, and data contracts. Unlike structural and operational metrics which are auto-generated, business metrics are created by analysts, engineers, and domain SMEs to capture logic the platform cannot infer automatically. This is the second-largest context with 8.8K+ metrics.
When to use business metrics:
| Type | Description |
|---|---|
| Custom | User-defined metric applied to a specific asset using filters, expressions, or threshold rules. Covers the majority of domain-specific quality checks — e.g. “percentage of orders with a valid product ID” or “revenue column must not be negative.” |
| Conditional | Metric applying different logic or thresholds depending on another column’s value — e.g. if region = 'US', check for a 10-digit phone format; otherwise apply a different rule. Useful for mixed-population assets. |
| Query | Metric defined by a SQL query returning a scalar value. Enables joins, aggregations, and CTEs — ideal for rules requiring data from multiple sources. |
| Standalone | Metric not bound to a specific asset. Used for org-wide KPIs, cross-domain aggregations, or summary metrics drawing from multiple datasets. |
| Behavioral | Metric evaluating patterns over time rather than a single point. Detects trend changes, seasonal deviations, or gradual drift — e.g. daily sales volume trending 20% below its 30-day moving average. |
Custom, Conditional, and Standalone business metrics do contribute to trust scoring (
Score: Yes). Behavioral metrics do not.| Scenario | Recommended type | Example |
|---|---|---|
| Domain rule the platform cannot infer | Custom | ”The discount_pct column must never exceed 100” |
| Same column, different valid formats by segment | Conditional | ”If country = 'US', zip_code must be 5 digits; otherwise 6–8 characters” |
| Rule spans multiple tables or requires a JOIN | Query | ”Count of orders with no matching customer record in the CRM table” |
| KPI that rolls up across many assets | Standalone | ”% of customer records with a valid email across all regional tables” |
| Catch gradual drift by dimension or segment | Behavioral | ”Alert if daily transaction volume for Product A drops >15% below the 30-day rolling average” |
| Auto profiling scores don’t reflect real business impact | Custom | ”Flag any claim where claim_status = 'closed' but settlement_amount is null” |
| Enforce a data contract with a downstream consumer | Custom or Query | ”Revenue in the mart must match the source system to within 0.1%“ |
| Column passes structural checks but violates a business constraint | Custom | ”age is non-null and numeric, but values below 18 are not valid for this product” |
Reconciliation metrics compare data across sources, snapshots, or reference tables to verify consistency and correctness. The smallest context by metric count, but critical for cross-system data contracts, ETL validation, and regulatory compliance. Reconciliation metrics support scheduling, alerting, and manual run triggers.
When to use reconciliation:
Exact metric counts vary by tenant and change over time — check the live count on your Metric page rather than a number documented here.
| Type | Description |
|---|---|
| Comparison | Compares a column or aggregate value between two registered assets — e.g. the row count in a source table versus the same table after an ETL load, or a revenue total in the warehouse versus the source system. Returns a match score and flags discrepancies exceeding a defined tolerance. Typical use cases: source vs target row count match, delta scoring, cross-system segment comparison. |
| Lookup | Validates that every value in a column exists in a reference dataset or allowed set. Used for referential integrity — e.g. every product_id in a fact table must exist in the product dimension. Typical use cases: FK validation, reference table match, allowed-value conformity checks. |
- Two sources should contain the same data — e.g. a data warehouse and an operational database after an overnight sync.
- Before and after a load — verifying row counts and key values match between source and destination.
- Reference table validation — ensuring foreign key values in a fact table exist in the corresponding dimension.
- Regulatory reporting — finance and compliance use cases where exact value matching is required.
- ETL pipeline validation — confirming upstream row counts are preserved through each transformation step.
Comparison metrics require both source and target assets to be registered and accessible in Prizm. If a source asset is unreachable, the metric shows
Score: NA and raises an availability alert rather than a comparison result.Semantic metrics are derived at a logical or business grouping level — domain, product, application, tag, or connection — rather than at the individual asset level. They aggregate quality signals across ownership boundaries to enable domain- and product-level KPIs and SLOs.
Semantic metrics align Prizm quality signals with your organisation’s ownership model — making it possible for domain owners, product managers, and data stewards to track the health of their data without navigating individual asset pages.
| Grouping | Example Metric | Description |
|---|---|---|
| Domain | DQ score for Customer360 domain | Weighted quality score rolled up across all assets assigned to a domain |
| Product | Data product health score | Aggregate quality and freshness across all assets in a defined data product |
| Application | App-level SLO compliance | % of assets in an application meeting their quality and freshness SLOs |
| Tag | PII tag coverage | % of assets with PII columns that have sensitivity tags applied and approved |
| Connection | Source reliability score | Aggregate quality score across all assets from a given Databricks connection |
Semantic metrics are the right lens for SLO reporting and executive dashboards — they answer “how healthy is the Customer360 domain?” rather than “how healthy is this one table?”
Pipeline metrics apply specifically to Databricks Job and Pipeline objects — Delta Live Tables, job runs, and orchestrated pipeline executions. They track operational health at the pipeline level rather than the individual asset level, and are the primary signals for pipeline SLA monitoring and incident detection.
| Metric | Description |
|---|---|
| Execution Status | Whether the most recent pipeline run succeeded, failed, or was cancelled. The primary signal for pipeline health alerting — any non-success state triggers an immediate alert. |
| Freshness | Time elapsed since the last successful pipeline completion. Fires when a pipeline has not completed a successful run within its expected window, signalling a stalled or skipped execution. |
| Rows Affected | Number of rows written, updated, or deleted by the pipeline run. Sudden drops indicate partial loads or upstream data gaps; spikes may signal duplicate processing. |
| Execution Time | Total elapsed wall-clock time for the pipeline run from start to finish. Tracks SLA compliance and detects regressions — a rising trend often precedes outright failures. |
| Failure Rate | Percentage of pipeline runs that have failed over a rolling window. A rising failure rate surfaces flaky pipelines before they cause persistent data outages. |
| Dropped Records | Count of records silently discarded during pipeline execution — typically due to schema mismatches, filter conditions, or quarantine rules. Unexpected spikes indicate upstream format changes or corrupted input data. |
| Failed Records | Count of records that failed validation or processing rules and were not written to the target. Distinct from dropped records — these are records the pipeline attempted to process but could not. |
| Platform Outage | Indicates whether a pipeline failure is attributable to a Databricks platform-level incident rather than a data or logic error. Prizm correlates run failures with Databricks status signals to suppress false-positive alerts during outage windows. |
Pipeline metrics operate at the Job / Pipeline object level. They are surfaced on the Pipeline asset page in Prizm and feed into the Pipeline observability dashboard. Alerting thresholds for Execution Status and Freshness are configured per pipeline.
Supported Databricks Objects
| Object | Catalog | Quality | Observability | Lineage |
|---|---|---|---|---|
| Table (Managed) | ✅ | ✅ | ✅ | ✅ |
| Table (External) | ✅ | ✅ | ✅ | ✅ |
| View | ✅ | ✅ | Schema only | ✅ |
| Delta Live Table | ✅ | ✅ | ✅ | ✅ |
| Job / Pipeline | ✅ | — | ✅ | — |
| Stored Procedure | ✅ | — | — | — |
The Databricks connector targets Unity Catalog workspaces on AWS, Azure, and GCP. Hive Metastore (legacy) workspaces without Unity Catalog enabled are not supported.
Next Steps
Setup
Connect Databricks to Prizm — prerequisites, authentication, and configuration.
What We Collect
Full field-level breakdown of every metadata object Prizm extracts from Databricks.
FAQ
Common questions about the Databricks connector.
Snowflake Connector
Connect your Snowflake data warehouse.
Glossary
Definitions for key terms used across Prizm.