> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Metric Overview
> Metric types, execution engine, scheduling, and quality scoring pipeline.
## Overview
Every metric in Prizm belongs to a **context** — the stakeholder lens that defines who the metric serves and why it matters. Contexts appear as the six summary cards at the top of the Metric page, each containing its own categories and individual metrics.
Measure Day-to-day health of data assets
How efficiently the platform itself is running
Shape of data at the column level
User-defined metric aligned to domain
Compare data across sources and libraries
Health rolled up at the domain, product, application, tag, or connection level
***
## Operational
Operational metrics monitor the **day-to-day health of data assets** — whether data is arriving on time, in the expected volume, and with the correct structure. They run at the **Asset** level and are the primary driver of alerting and anomaly detection in Prizm. All operational metrics have `Monitor: Yes`.
All Operational metrics feed directly into the Alerts dashboard. They are the most actively monitored metrics in a typical Prizm deployment, with Execution Status and Freshness generating the highest alert volumes in production.
Tracks how much data exists in an asset — row count, data size, and absolute volume over time.
Measures how recently data was loaded or updated. Detects stale datasets and pipeline delays before they impact consumers.
Monitors the structure of a dataset — column presence, data types, and schema drift. Catches breaking changes at the source.
### Availability metrics
| Metric | Subcategory | Dimension | Level | Monitor | Description |
| ------------ | ----------- | ------------ | ----- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Row | Volume | Completeness | Asset | Yes | The total number of rows in the asset at the time of the last load or scan. Tracks volume over time and triggers alerts when row counts drop or spike unexpectedly. |
| Volume | Volume | Completeness | Asset | Yes | A higher-level measure of data volume combining row count and data size signals. Used to detect significant changes in the amount of data arriving in an asset. |
| Data size | Volume | Completeness | Asset | Yes | The physical size of the asset in bytes (or KB/MB). Complements row count — a large size increase with no row increase may indicate wide or bloated records. |
| Freshness | Freshness | Availability | Asset | Yes | The time elapsed since the asset was last updated. Fires an alert when data has not been refreshed within the expected window — the most common operational alert in production. |
| Last updated | Freshness | Timeliness | Asset | Yes | The timestamp of the most recent data load or write to the asset. Used to verify that pipelines are running on schedule and data consumers are not working with stale records. |
| Column | Schema | Validity | Asset | Yes | The current number of columns in the asset. Detects column additions or removals that may indicate a schema change at the source without a formal migration. |
| Schema | Schema | Validity | Asset | Yes | A snapshot of the full column schema — names, data types, and order. Alerts when any column name or type changes, which can silently break downstream transformations. |
| Schema name | Schema | Validity | Asset | Yes | The name of the schema or namespace the asset belongs to in the source system. Tracks unexpected schema renames that would break hard-coded references. |
| Duplicates | Duplicate | Uniqueness | — | Yes | The count of duplicate rows in the asset based on a defined key or all columns. Unexpected duplicates often indicate a pipeline re-run without deduplication or a broken merge logic. |
**Provisioning note.** On some tenants, only a single **Schema** metric is currently provisioned per asset, and its displayed value is the column count (the behavior documented above for **Column**) rather than a full name/type/order snapshot. If the Column and Schema name metrics aren't appearing separately on your asset's Metric tab, this is expected on those tenants — treat the single Schema metric's value as the column count until Column and Schema name are separately provisioned for your tenant. Confirm with your Prizm administrator which of these three metrics are actually enabled for your environment.
Operational metrics do **not** contribute to trust scoring (`Score: No`). They are observability signals — designed to trigger alerts, not to roll up into a quality score.
***
## Performance
Performance metrics measure **how efficiently the data platform itself is running** — query execution time, compute utilization, job duration, and credit consumption. Unlike other contexts, performance metrics operate at the **Source and System level** (Warehouse, Database, Account) rather than the individual asset level.
Tracks who is querying data assets, how frequently, and the success rate of those queries. Useful for adoption tracking and access governance.
Measures execution time, job duration, blocked queries, and throughput. Identifies slow pipelines and compute bottlenecks.
Tracks compute credit consumption at the warehouse and database level. Essential for cloud cost management and chargeback reporting.
### Performance metrics
| Metric | Level | Profile Type | Description |
| --------------------- | ------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Total queries per day | Warehouse, Database | Performance | The total number of queries executed against a warehouse or database in a given day. Tracks usage trends and surfaces unexpected spikes that may indicate runaway jobs or inefficient query patterns. |
| Query success rate | Warehouse, Database | Performance | The percentage of queries that completed successfully without error. A falling success rate signals pipeline instability, permission issues, or query timeouts. |
| Execution time | Warehouse, Database | Performance | The average time taken to execute a query from submission to completion. Rising execution times indicate compute pressure, missing indexes, or poorly optimised SQL. |
| Job duration | Warehouse, Database | Performance | The total elapsed time for a data job or pipeline run. Monitors whether batch jobs are completing within their SLA windows — a common metric for nightly ETL pipelines. |
| Test execution time | Warehouse, Database | Performance | The time taken to run data quality test suites against the warehouse. Tracks whether test runtime is growing as asset counts increase, which can delay pipeline signoff. |
| Avg blocked | Warehouse, Database | Performance | The average number of queries in a blocked state at any given time — waiting on locks or resource conflicts. High blocked counts indicate concurrency issues or long-running transactions holding locks. |
| Avg queue load | Warehouse, Database | Performance | The average number of queries waiting in the execution queue. A persistently high queue load means the warehouse is under-provisioned for the current workload. |
| Avg running | Warehouse, Database | Performance | The average number of queries actively running at any given time. Combined with queue load, gives a full picture of warehouse concurrency and throughput. |
| Database storage | Database | Performance | The total storage consumed by the database, including tables, indexes, and staging data. Tracks growth trends and flags databases approaching capacity thresholds. |
| Total credits per day | Warehouse, Database | Cost | The total compute credits consumed per day across the warehouse or database. The primary metric for cloud cost management — used to set budgets and trigger cost alerts. |
| Credits used | Warehouse, Database | Cost | The cumulative credits consumed over a defined period. Used for chargeback reporting, team-level cost allocation, and identifying high-spend workloads. |
Performance metrics are most valuable to **platform and infrastructure teams** tracking SLAs and compute spend. They are surfaced separately from data quality metrics so they don't dilute profiling results.
Performance metrics do **not** contribute to trust scoring (`Score: No`). They are platform and infrastructure observability signals — designed to trigger alerts, not to roll up into a quality score.
***
## Structural
Structural metrics profile the **internal shape of data at the column level**. They answer: *what does this data look like — how are values distributed, how often do patterns repeat, what do the numbers say, and what structural patterns exist?* This is the largest context in Prizm with 33K+ metrics. It powers data profiling and feeds trust scoring at the attribute level.
Structural metrics are all **auto-generated** — they run automatically when an asset is profiled and require no user configuration.
How values are spread across a column — completeness, character makeup, space characters, and numeric shape.
How often specific values, lengths, and value ranges repeat. Captures enums, min/max boundaries, and value range conformity.
Auto-discovered and user-defined structural templates that describe the format of values in a column — short and long patterns, regex expressions.
Numeric summaries of data characteristics — central tendency, spread, quantiles, and distribution shape.
### Distribution
Distribution metrics capture how values inside a column are composed and spread. They answer questions like: how many nulls are there, how many are distinct, are there leading spaces, are there unexpected special characters?
| Metric | Subcategory | Dimension | Profile Type | Description |
| ----------------- | ------------ | ------------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Null count | Completeness | Completeness | Essential | The total number of null (missing) values in a column. A high null count signals missing data that may break downstream logic or skew analysis. |
| Empty | Completeness | Completeness | Essential | The count of values that are empty strings (`""`). Distinct from null — a value may be present but contain no meaningful content. |
| Non-empty | Completeness | Completeness | Advanced | The count of values that are neither null nor empty. Useful as a baseline for understanding how much of the column actually has content. |
| Space | Completeness | Completeness | Essential | The count of values that contain only whitespace characters. These appear non-empty but carry no data — a common data entry artefact. |
| Zero value | Numeric | Completeness | Essential | The count of numeric values equal to zero. Flags columns where zero may be a placeholder for missing data rather than a meaningful measurement. |
| Distinct | Uniqueness | Uniqueness | Essential | The count of unique values in the column. Low distinct counts on a high-cardinality column can indicate data collapse or pipeline errors. |
| Repeating | Uniqueness | Uniqueness | Advanced | The count of values that appear more than once. High repeating counts on a column expected to be unique (e.g. IDs) signal duplicates. |
| Leading space | Space | Validity | Advanced | The count of values with one or more space characters at the start. Can cause join failures and unexpected mismatches in downstream queries. |
| Trailing space | Space | Validity | Advanced | The count of values with one or more space characters at the end. A frequent cause of silent data quality issues in string comparisons. |
| Inner space | Space | Validity | Advanced | The count of values containing consecutive spaces within the string (e.g. `"John Doe"`). Indicates inconsistent formatting or copy-paste artefacts. |
| Outer space | Space | Validity | Advanced | The count of values with spaces at either the start or end (leading or trailing). A combined view of leading and trailing space issues. |
| Whitespace | Space | Validity | Advanced | The count of values that consist entirely of whitespace characters (spaces, tabs, newlines). Similar to Space but captures a broader set of invisible characters. |
| Character | Character | Validity | Advanced | The distribution of character types present in the column — letters, digits, special characters, and mixed. Gives a fingerprint of the column's character composition. |
| Alpha numeric | Character | Validity | Advanced | The count of values containing only alphabetic and numeric characters, with no special characters or spaces. Useful for validating ID and code columns. |
| Digit | Character | Validity | Advanced | The count of values composed entirely of numeric digits. Useful for columns that should be numeric but are stored as strings (e.g. zip codes, phone numbers). |
| Special character | Character | Validity | Advanced | The count of values containing at least one special character (e.g. `@`, `#`, `!`). Unexpected special characters often indicate data entry errors or encoding issues. |
| Alphabet | Character | Validity | Advanced | The count of values containing only alphabetic characters, no digits or special characters. Validates columns that should be pure text (e.g. country names, category labels). |
| Positive | Numeric | Validity | Advanced | The count of numeric values greater than zero. Used alongside Negative and Zero value to understand the sign distribution of a numeric column. |
| Negative | Numeric | Validity | Advanced | The count of numeric values less than zero. Unexpected negative values in columns like age, price, or quantity typically indicate bad data. |
### Frequency
Frequency metrics count how often specific values, formats, or lengths appear. They surface the most common patterns in a column — useful for detecting unexpected enums, outlier lengths, or format drift.
| Metric | Subcategory | Dimension | Profile Type | Description |
| ------------ | ----------- | --------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Enum | Value | Validity | Advanced | The set of distinct values and their frequencies in the column — effectively a value histogram. Reveals the actual distribution of categories and flags values that fall outside an expected allowed list. |
| Min value | Value | Validity | Essential | The smallest value in the column (numeric or date). Establishes the lower bound of the data and flags records that fall below expected thresholds. |
| Max value | Value | Validity | Essential | The largest value in the column (numeric or date). Establishes the upper bound and detects outliers or values that exceed business-defined limits. |
| Value range | Range | Validity | Advanced | The spread between the minimum and maximum values. A sudden change in value range over time is a strong signal of data drift or pipeline contamination. |
| Min length | Length | Validity | Essential | The shortest string length found in the column. Helps catch truncated values — for example, a phone number column where some values are unexpectedly short. |
| Max length | Length | Validity | Essential | The longest string length found in the column. Useful for detecting values that exceed expected character limits, which can cause issues in downstream systems. |
| Length | Length | Validity | Advanced | The full distribution of string lengths across all values in the column. Reveals whether values cluster around a consistent length or vary widely. |
| Length range | Range | Validity | Advanced | The spread between the minimum and maximum string lengths. A wide length range on a column expected to have uniform values (e.g. country codes) signals a format problem. |
### Pattern
Pattern metrics describe the **structural template** of values in a column — the repeating format that emerges from the data itself. Prizm auto-discovers patterns by analysing the character composition of each value (letters, digits, special characters) and grouping them into short and long pattern representations. You can also define your own patterns using regular expressions.
Pattern metrics are particularly useful for columns like names, addresses, IDs, and codes where you expect a consistent format. As shown in the screenshot above, common examples include `LONG PATTERN` on columns like `ADDRESS_LINE2`, `LAST_NAME`, and `COUNTRY`, and `SHORT PATTERN` on columns like `STATE`, `RISK_SCORE`, and `FIRST_NAME`.
| Metric | Subcategory | Dimension | Profile Type | Description |
| ------------------- | ----------- | --------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Short pattern | Pattern | Validity | Advanced | A condensed representation of the value format — collapses consecutive identical character types into a single token. For example, `"John"` becomes `A` (alpha). Useful for spotting high-level format anomalies quickly. |
| Long pattern | Pattern | Validity | Advanced | A character-by-character representation of the value format — preserves the full length and composition. For example, `"John"` becomes `AAAA`. More precise than short pattern; reveals length drift and mixed-format issues. |
| Regular expressions | Pattern | Validity | Essential | User-defined regex patterns applied to validate that column values conform to a known format — for example, email addresses, phone numbers, or custom ID formats. |
Short pattern and Long pattern are complementary. Use **Short pattern** for a quick high-level view of format variety in a column, and **Long pattern** when you need to detect subtle length or composition changes — for example, a country code column that sometimes contains a 3-character value instead of 2.
***
### Statistics
Statistics metrics produce numeric summaries of a column's values. They are applied to numeric and date columns and are essential for detecting drift, anomalies, and data quality degradation over time.
| Metric | Subcategory | Dimension | Profile Type | Description |
| ------------------- | ----------- | --------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Mean | Descriptive | Accuracy | Advanced | The arithmetic average of all non-null values. A shift in the mean over time is one of the earliest signals of data drift or a change in the upstream source. |
| Median | Descriptive | Accuracy | Advanced | The middle value when all non-null values are sorted. Less sensitive to outliers than the mean — useful for skewed distributions like revenue or transaction amounts. |
| Mode | Descriptive | Accuracy | Advanced | The most frequently occurring value in the column. A sudden change in mode can indicate a default value being injected or a dominant new category appearing in the data. |
| Standard deviation | Descriptive | Accuracy | Advanced | A measure of how spread out values are around the mean. A rising standard deviation indicates increasing variability — often a sign of data quality degradation. |
| Sum | Descriptive | Accuracy | Advanced | The total of all non-null numeric values. Useful for financial columns where the total amount should remain consistent across pipeline stages. |
| Q1 | Quantile | Accuracy | Advanced | The 25th percentile — 25% of values fall below this point. Used with Q3 to define the interquartile range and identify the typical spread of data. |
| Q3 | Quantile | Accuracy | Advanced | The 75th percentile — 75% of values fall below this point. Together with Q1, defines the middle 50% of the data distribution. |
| Range | Shape | Accuracy | Advanced | The difference between the maximum and minimum values (`max - min`). A quick measure of the total spread of the data in a column. |
| Skewness | Shape | Accuracy | Advanced | A measure of the asymmetry of the value distribution. Positive skew means a long tail on the right; negative skew means a long tail on the left. Useful for detecting non-normal distributions. |
| Kurtosis | Shape | Accuracy | Advanced | A measure of the "peakedness" or tail weight of the distribution. High kurtosis indicates more extreme outliers than a normal distribution; low kurtosis indicates a flatter distribution. |
| Variance | Shape | Accuracy | Advanced | The average of the squared differences from the mean. Like standard deviation but in squared units — used in statistical tests and anomaly detection models. |
| The margin of error | Shape | Accuracy | Advanced | An estimate of the uncertainty in the sample statistics. Indicates how reliably the computed metrics represent the true population, especially when profiling on a sample. |
***
## Business
Business metrics are **user-defined measurements aligned to domain rules, KPIs, and data contracts**. Unlike structural and operational metrics which are auto-generated, business metrics are created by data analysts, engineers, and domain SMEs to capture logic the platform cannot infer automatically. They are the second-largest context at 8.8K metrics.
User-defined metrics using business rules, filters, or expressions tailored to a specific asset or domain. The most common type — accounts for the vast majority of business metrics.
Metrics that evaluate differently based on conditions — for example, applying different thresholds depending on a segment, region, or category value.
Metrics defined by a SQL query, enabling arbitrary logic and multi-table expressions that go beyond standard profiling.
Independent metrics not tied to a specific asset. Used for global KPIs or cross-source aggregations that exist at the domain or org level.
Metrics that track patterns over time — detecting drift, anomalies, or trend deviations in business data rather than point-in-time values.
| Type | Description |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Custom | A user-defined metric applied to a specific asset using filters, expressions, or threshold rules. The most flexible type — covers the majority of domain-specific quality checks such as "percentage of orders with a valid product ID" or "revenue column must not be negative." |
| Conditional | A metric that applies different logic or thresholds depending on the value of another column or condition. For example: if `region = 'US'`, check that `phone_number` matches a 10-digit format; otherwise apply a different rule. Useful for assets that contain mixed populations. |
| Query | A metric defined entirely by a SQL query that returns a scalar value. Enables arbitrarily complex logic — joins across tables, aggregations, CTEs — beyond what expression-based metrics support. Ideal for business rules that require data from multiple sources. |
| Standalone | A metric that is not bound to a specific asset. Used for organisation-wide KPIs, cross-domain aggregations, or summary metrics that draw from multiple datasets. Scores and alerts are raised at the domain or global level. |
| Behavioral | A metric that evaluates patterns in data over time rather than at a single point. Detects trend changes, seasonal deviations, or gradual drift — for example, a daily sales volume that is trending 20% below its 30-day moving average. |
Business metrics support scheduling, alerting, and scoring just like auto metrics. They appear alongside structural and operational metrics in the Metric list view — use the type sub-tabs to filter by Custom, Conditional, Query, Standalone, or Behavioral.
Business metric categories such as Custom, Conditional, Standalone **do** contribute to trust scoring (`Score: Yes`). They are business quality metric — designed to trigger alerts, and roll up into a quality score.
Behavioral and Comparison does not contribute to trust scoring.
### When to use business metrics
Auto-generated structural and operational metrics give you broad coverage of data shape and pipeline health, but they cannot know your business rules. Reach for business metrics when the quality check you need requires domain knowledge — something only a human who understands the data can define.
| Scenario | Recommended type | Example |
| ---------------------------------------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------ |
| You have a domain rule that the platform cannot infer | Custom | "The `discount_pct` column must never exceed 100" |
| The same column has different valid formats depending on another column | Conditional | "If `country = 'US'`, `zip_code` must be 5 digits; otherwise 6–8 characters" |
| Your rule spans multiple tables or requires a JOIN | Query | "Count of orders with no matching customer record in the CRM table" |
| You need a KPI that rolls up across many assets | Standalone | "Overall percentage of customer records with a valid email across all regional tables" |
| You need to catch gradual drift by dimension or segement rather than a point-in-time violation | Behavioral | "Alert if daily transaction volume drops for Product A more than 15% below the 30-day rolling average" |
| Auto profiling scores are not reflecting real business impact | Custom | "Flag any claim record where `claim_status = 'closed'` but `settlement_amount` is null" |
| You need to enforce a data contract with a downstream consumer | Custom or Query | "Revenue in the mart must match revenue in the source system to within 0.1%" |
| A column passes structural checks but violates a business constraint | Custom | "`age` is non-null and numeric, but values below 18 are not valid for this product" |
***
## Reconciliation
Reconciliation metrics **compare data across sources, snapshots, or reference tables** to verify consistency and correctness. They are the smallest context by metric count but critical for cross-system data contracts, regulatory compliance, and ETL validation. Reconciliation metrics support scheduling, alerting, and manual run triggers.
Exact counts for each context (metrics, assets, etc.) vary by tenant and change as the platform evolves — check the live count shown on your Metric page rather than relying on a number documented here.
Compares values, row counts, or segments across two datasets, systems, or time snapshots to detect discrepancies. Used to validate that a source and target agree after a load or transformation.
Validates column values against a reference table or allowed set. Checks referential integrity — for example, that every foreign key in a fact table exists in the corresponding dimension.
| Type | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Comparison | Compares a column or aggregate value between two registered assets — for example, the row count in a source table versus the same table after an ETL load, or a revenue total in the data warehouse versus the source system. Returns a match score and flags discrepancies that exceed a defined tolerance. Typical use cases revolves around Source vs target row count match, delta scoring, cross-system segment comparison |
| Lookup | Validates that every value in a column exists in a reference dataset or allowed set. Used for referential integrity checks — for example, confirming that every `product_id` in a fact table exists in the product dimension, or that every country code is in the ISO standard list. Typical use cases revolves around FK validation, reference table match, allowed-value conformity checks |
### When to use reconciliation
Reconciliation metrics are most valuable when:
* **Two sources should contain the same data** — for example, a data warehouse and an operational database after an overnight sync.
* **Before and after a load** — verifying row counts and key values match between source and destination.
* **Reference table validation** — ensuring foreign key values in a fact table exist in the corresponding dimension.
* **Regulatory reporting** — finance and compliance use cases where exact value matching is required.
* **ETL pipeline validation** — confirming upstream row counts are preserved through each transformation step.
Comparison metrics require both source and target assets to be registered and accessible in Prizm. If a source asset is unreachable, the metric will show `Score: NA` and generate an availability alert rather than a comparison result.
***
## Semantic
Semantic metrics are derived at a **logical or business grouping level** — domain, product, application, tag, or connection — rather than at the individual asset level. They aggregate quality signals across ownership boundaries to enable domain- and product-level KPIs and SLOs.
| Grouping | Example Metric | Description |
| --------------- | ------------------------------- | ---------------------------------------------------------------------------- |
| **Domain** | DQ score for Customer360 domain | Weighted quality score rolled up across all assets assigned to a domain |
| **Product** | Data product health score | Aggregate quality and freshness across all assets in a defined data product |
| **Application** | App-level SLO compliance | % of assets in an application meeting their quality and freshness SLOs |
| **Tag** | PII tag coverage | % of assets with PII columns that have sensitivity tags applied and approved |
| **Connection** | Source reliability score | Aggregate quality score across all assets from a given connection |
Semantic metrics align Prizm quality signals with your organization's ownership model — making it possible for domain owners, product managers, and data stewards to track the health of their data without navigating individual asset pages.
Semantic metrics are the right lens for **SLO reporting and executive dashboards** — they answer "how healthy is the Customer360 domain?" rather than "how healthy is this one table?"
Semantic metrics do **not** drive alerting (`Monitor: —`). They are rolled up from the underlying asset-level scores, so their `Score` reflects the health of the assets beneath them rather than an independently computed result.