- Distribution
- Frequency
- Pattern
- Statistical
| Profile Insight | Description | Quality Dimension | Profile Type |
|---|---|---|---|
| Null Count | Total number of records where the field contains a NULL value, indicating completely missing data. High null counts may signal data collection failures, unpopulated optional fields, or ETL pipeline gaps. | Completeness | Essential |
| Empty Count | Count of records containing an empty string ('') rather than NULL. Empty strings are non-null but carry no meaningful content — common in unvalidated form inputs and text fields. | Completeness | Essential |
| Non-Empty Count | Count of records containing a meaningful, non-null, non-empty value. The primary completeness indicator. Calculated as Total Records - Null Count - Empty Count. | Completeness | Essential |
| Space Count | Count of records where the field contains only whitespace characters (spaces, tabs, newlines). Space-only values are neither null nor empty yet contain no useful information — a hidden completeness gap often missed by basic null checks. | Completeness | Essential |
| Distinct Count | The number of unique values present in a field, excluding nulls. For a primary key field, distinct count must equal total record count. Helps identify cardinality and validate field uniqueness expectations. | Uniqueness / Integrity | Essential |
| Repeating Count | The count of values that appear more than once. Calculated as Total Count - Distinct Count. High repeating counts in key fields (e.g., customer IDs, order numbers) indicate duplication and referential integrity violations. | Uniqueness / Integrity | Essential |
| Digits Count | Count of records containing one or more numeric digit characters (0–9). Used to validate that identifiers, phone numbers, or postal codes contain the expected digit composition. | Validity / Format | Essential |
| Special Character Count | Count of records containing non-alphanumeric characters such as @, #, $, %, &, *. Elevated counts in fields expected to contain only letters or numbers indicate data entry errors, encoding issues, or injection risks. | Validity / Format | Essential |
| Alphabet Count | Count of records containing purely alphabetic characters (A–Z, a–z). Validates that text-only fields such as first and last names do not contain unexpected numeric or special characters. | Validity / Format | Essential |
| Alpha Numeric Count | Count of records containing both alphabetic and numeric characters. Essential for validating mixed-format fields like product codes, license plates, and reference numbers. | Validity / Format | Essential |
| Leading Space | Count of records where the field value begins with one or more space characters. Leading spaces are invisible in most UIs but silently break exact-match comparisons and join operations. Example: ' John' ≠ 'John'. | Consistency / Format | Essential |
| Trailing Space | Count of records where the field value ends with one or more space characters. Trailing spaces are a common artifact in fixed-length fields exported from legacy systems and cause lookup failures. | Consistency / Format | Essential |
| Outer Space | Combined count of records with either leading or trailing spaces. An aggregated signal for edge whitespace issues across the entire field, covering both leading and trailing space cases in a single metric. | Consistency / Format | Essential |
| Inner Space | Count of records containing multiple consecutive internal space characters. Example: 'John Smith' (two spaces) vs 'John Smith' (one space). Typically indicates data merging errors or unvalidated free-text input. | Consistency / Format | Advanced |
| Whitespace Count | Total count of all whitespace characters (spaces, tabs, carriage returns, line feeds) across all field values. A high ratio relative to total character count may indicate padded, poorly formatted, or embedded-formatting values from source systems. | Consistency / Format | Advanced |
| Zero Count | Count of records where a numeric field contains exactly 0. Distinguishing zeros from nulls is business-critical — a zero account balance differs fundamentally from an unknown balance. Elevated zero counts may indicate default value substitution or rounding errors. | Accuracy / Validity | Essential |
| Positive Count | Count of records where a numeric field contains a value greater than zero. Used alongside Negative Count and Zero Count to validate sign distribution expectations for financial, operational, or scientific data. | Accuracy / Validity | Essential |
| Negative Count | Count of records where a numeric field contains a value less than zero. Unexpected negatives in fields like age, quantity, or price indicate data entry errors or sign convention mismatches between source systems. | Accuracy / Validity | Essential |
| Metric | Description | Quality Dimension | Profile Type |
|---|---|---|---|
| Min Value | The smallest value in the field for its data type — arithmetic minimum for numerics, lexicographic minimum for strings, earliest date for temporals. Identifies lower-bound outliers and validates minimum value constraints. | Accuracy / Range | Essential |
| Max Value | The largest value in the field. Identifies upper-bound outliers and validates ceiling constraints. Example: a Max Value of 150 in an Age field is a clear data error signal. | Accuracy / Range | Essential |
| Enum (Value Occurrence Counts) | A frequency distribution table of all distinct values and how many records contain each. Invaluable for discovering unexpected or invalid values in categorical fields, identifying dominant values, and detecting rare values that may indicate errors. | Accuracy / Validity | Essential |
| Min Length | The minimum character length observed across all non-null values. A value below the expected minimum (e.g., a 2-character value in a 5-character ZIP code field) indicates truncation, premature data entry termination, or schema mismatches. | Consistency / Format | Essential |
| Max Length | The maximum character length observed across all non-null values. Values exceeding defined field length constraints indicate data truncation risk at ingestion or schema mismatches between source and target systems. | Consistency / Format | Essential |
| Length Distribution | A frequency distribution showing how many records fall at each distinct character length. A uniform distribution confirms fixed-format codes; high variance in an expected fixed-length field reveals format inconsistencies across records or source systems. | Consistency / Format | Advanced |
| Regular Expressions | Count and percentage of values that match predefined or auto-discovered regular expression patterns. Non-matching records are flagged as format violations. Supports multiple patterns simultaneously for fields with more than one valid format. | Validity / Format | Advanced |
| Long Pattern | Frequency distribution of structural patterns derived by replacing each character with its type code: A for alphabetic, N for numeric, with punctuation and delimiters preserved. Example: 'PRD-001' → 'AAA-NNN'. Reveals the full landscape of format variants in a field. | Consistency / Format | Advanced |
| Short Pattern | A compressed variant of Long Pattern where consecutive identical character type codes are collapsed into single tokens. Example: 'AAANNN' → 'AN'. Quickly exposes fundamentally different structural categories within a field without the granularity of Long Pattern. | Consistency / Format | Advanced |
| Value Range | The span between the minimum and maximum values (Max - Min). A range far larger than expected for the domain suggests the presence of outliers or errors at the extremes of the distribution. | Accuracy / Range | Essential |
| Length Range | The span between the minimum and maximum observed character lengths (Max Length - Min Length). A range of 0 confirms a fixed-format field. A large range in an expected fixed-length field immediately signals format inconsistencies. | Consistency / Format | Essential |
| Metric | Description | Quality Dimension | Profile Type |
|---|---|---|---|
| Regular Expression Matching | Validates field values against precisely defined structural rules using regex patterns. Supports both predefined patterns for common types (email, phone, ZIP, date, currency) and auto-discovered patterns derived from observed data. Reports match vs. non-match counts per pattern rule. Essential for fields with strict format requirements. | Validity / Format | Advanced |
| Long Pattern Analysis | Auto-discovers all structural patterns present in a field by encoding each character as its type — A for alphabetic, N for numeric, with literal delimiters preserved. Collects unique patterns and their frequency, providing a complete structural fingerprint. Best used for initial discovery and cross-source format comparison. | Consistency / Format | Advanced |
| Short Pattern Analysis | Provides a compressed structural overview by collapsing consecutive identical character type codes into single tokens. Complements Long Pattern by categorizing values at a higher level of abstraction. Especially useful for detecting repeating character sequences and identifying high-level structural divergence. | Consistency / Format | Advanced |
| Metric | Description | Quality Dimension | Profile Type |
|---|---|---|---|
| Mean | The arithmetic average of all non-null values. Calculated as Sum / Non-Null Count. Sensitive to outliers — a single extreme value can significantly shift the mean away from the typical value. Use with caution in skewed or heavy-tailed distributions. | Accuracy / Distribution | Essential |
| Mode | The value (or values) that appears most frequently in the dataset. Applicable to both numeric and categorical fields. A field may be unimodal, multimodal, or have no mode. Resistant to outliers and useful for understanding the most commonly observed value. | Accuracy / Distribution | Essential |
| Median (Q2) | The middle value when all non-null values are sorted in ascending order. For even counts, the average of the two middle values. Robust to outliers and skewed distributions — more reliable than the mean as a central tendency indicator when extremes are present. | Accuracy / Distribution | Essential |
| Range | The difference between the maximum and minimum values (Max - Min). Provides a simple measure of total spread but is highly sensitive to outliers since it depends solely on the two most extreme values. | Accuracy / Distribution | Essential |
| Standard Deviation | The average distance of each value from the mean, expressed in the same units as the original data. Low SD indicates tight clustering around the mean; high SD indicates wide spread. Values more than ±3 SD from the mean are typically flagged as statistical outliers. SD = √(Σ(x − mean)² / N) | Accuracy / Distribution | Essential |
| Variance | The square of the standard deviation (SD²). Expressed in squared units of the original data, making it less directly interpretable than SD but mathematically fundamental for statistical modelling and cross-scale comparisons. | Accuracy / Distribution | Advanced |
| Kurtosis | Measures the tailedness of a distribution relative to a normal distribution. High kurtosis (leptokurtic) indicates heavy tails with more frequent extreme values. Low kurtosis (platykurtic) indicates light tails. Excess kurtosis = Kurtosis - 3, where 0 represents normal distribution behavior. | Accuracy / Distribution | Advanced |
| Skewness | Measures the asymmetry of the distribution around its mean. 0 = symmetric. Positive = right-skewed (long right tail, few high extremes). Negative = left-skewed (long left tail, few low extremes). Values beyond ±1 indicate meaningful asymmetry; beyond ±2 suggest outliers or errors. | Accuracy / Distribution | Advanced |
| Q0 (Minimum) | The 0th percentile — the smallest value in the distribution. Equivalent to Min Value. Represents the absolute lower bound and anchors the full quantile profile. | Accuracy / Range | Essential |
| Q1 (1st Quartile) | The 25th percentile. 25% of non-null values fall at or below this point. Combined with Q3, defines the Interquartile Range (IQR = Q3 − Q1) — the most robust measure of spread and the basis for Tukey Fence outlier detection. | Accuracy / Distribution | Essential |
| Q3 (3rd Quartile) | The 75th percentile. 75% of non-null values fall at or below this point. Values above Q3 + 1.5 × IQR are conventionally flagged as outliers. Use 3.0 × IQR for extreme outlier detection only. | Accuracy / Distribution | Essential |
| Q4 (Maximum) | The 100th percentile — the largest value in the distribution. Equivalent to Max Value. Represents the absolute upper bound and completes the five-number summary (Q0, Q1, Q2, Q3, Q4). | Accuracy / Range | Essential |
| Sum | The arithmetic total of all non-null numeric values. Most meaningful for additive measures such as revenue, transaction counts, or durations. Should be validated against control totals from source systems to detect data loss during ingestion or transformation. | Accuracy / Aggregation | Essential |
| Margin of Error | Quantifies the uncertainty in profiled estimates when working with a sample of a larger population rather than the full dataset. MOE = Z × (SD / √n) where Z = 1.96 for 95% confidence. Indicates the range within which the true population metric value falls. | Reliability / Confidence | Advanced |