Skip to main content

​
Connection & Authentication

Prizm supports three authentication methods for Databricks:
  • Token: Uses a Databricks Personal Access Token (PAT) — a long-lived credential generated in Databricks User Settings → Developer → Access Tokens. Use this only when OAuth is not available.
  • OAuth (M2M) (Recommended): Uses Databricks Service Principals with an OAuth Client ID and Client Secret. Tokens are short-lived and automatically refreshed by Prizm — no manual re-authentication required. Client credentials are stored encrypted in Vault.
  • OAuth (Microsoft Entra ID): Uses an Azure Active Directory / Microsoft Entra ID app registration with a Tenant ID, Client ID, and Client Secret. Recommended for Databricks workspaces hosted on Azure.
Use OAuth (Microsoft Entra ID) when your Databricks workspace is deployed on Azure and your organisation manages identities through Azure Active Directory (now Microsoft Entra ID). In this case, your service principals and app registrations live in Entra ID rather than in the Databricks account console. You will need:
  • A registered application in the Azure portal under Microsoft Entra ID → App Registrations
  • A Tenant ID (your Azure AD directory ID)
  • A Client ID (the Application ID of the registered app)
  • A Client Secret generated under the app registration → Certificates & Secrets
If your workspace is on AWS or GCP, use OAuth (M2M) instead.
Create a separate connector for each workspace. Each connector has its own credentials, SQL Warehouse HTTP path, scope configuration, and job schedules. You can name connectors to distinguish environments (e.g., databricks-prod, databricks-dev).
Yes. Account-level OAuth tokens (issued by accounts.cloud.databricks.com) provide access across all workspaces in the account. Workspace-level tokens are scoped to a single workspace. For multi-workspace monitoring, use account-level OAuth M2M with a Service Principal that has been assigned to the relevant workspaces.
Use Vault to manage your Databricks credentials when creating the connection. With Vault-backed connections, credential rotation happens entirely within your secrets manager — you update the secret in Vault, and Prizm automatically picks up the new credentials on the next job run without any changes to the connector configuration.
Prizm controls concurrency at the job orchestration layer. Invalid or inactive sources are excluded from scheduling.

​
Permissions

For catalog-only discovery (no observability, quality, lineage, or performance):
No access to system.* tables is required for metadata-only discovery.
System tables power the advanced capabilities beyond basic cataloging:
No. The Prizm Databricks connector targets Unity Catalog workspaces. Legacy Hive Metastore workspaces without Unity Catalog enabled are not supported. Databricks recommends migrating to Unity Catalog for governance and lineage capabilities.
Yes. Any service principal with the required permission grants will work. The setup instructions use a new service principal as a recommended practice for isolation, but an existing one is fine if it has the necessary grants.

​
Scoping & Object Inclusion

Yes. Use the Include / Exclude wildcard patterns in the Asset Scope step of the connector wizard. Exclude rules take precedence over include rules when both match the same object. For example:
  • Include: prod_*
  • Exclude: prod_temp_*, prod_dev_*
The information_schema and system schemas are excluded by default and cannot be included.
Check that:
  1. The table’s catalog and schema match your include patterns and are not excluded.
  2. The service principal has USE CATALOG, USE SCHEMA, and SELECT on information_schema.tables.
  3. The Catalog job has completed at least one full run since the table was created. Check Settings → Connectors → Logs for the last successful Catalog job timestamp.
Yes. Iceberg tables are detected via DESCRIBE TABLE EXTENDED (looking for Provider = iceberg). They are cataloged and monitored for freshness (using DESCRIBE DETAIL → lastModified) and volume. Column-level profiling depends on the Iceberg table being registered in Unity Catalog with accessible metadata.
Yes. External tables are detected by table_type = 'EXTERNAL' in information_schema.tables. Freshness is derived from DESCRIBE DETAIL → lastModified. Profiling and observability are supported; write-back is not available for external tables.

​
Observability & Freshness

Prizm uses DESCRIBE HISTORY <table> LIMIT 1 to get the latest Delta commit timestamp. This reads only the Delta transaction log (JSON files in _delta_log/) — it does not scan Parquet data files. Cost is constant regardless of table size (~50–200ms).This is preferred over LAST_ALTERED from information_schema.tables, which captures DDL changes only and may not reflect data writes.
An alert is an automated signal that a monitored metric or quality check has breached its threshold. Alerts are generated by the observability engine without any human action.An issue is a validated, actionable problem created from one or more related alerts. Issues are tracked work items that can be assigned, commented on, and resolved. The progression is:
Prizm uses time-series forecasting models to compute dynamic lower and upper thresholds for each metric automatically. The system:
  • Uses up to 100 recent historical run values as the lookback window per metric
  • Applies a Hampel filter (MAD-based, window 10, 3-sigma) to remove outliers before modeling
  • Selects the forecasting model automatically:
  • A new metric spends its first more than 5 historical collection runs in Learning mode, accumulating a baseline, before alerts begin firing. This threshold counts collection runs, not elapsed time.
  • Alert priority (LOW, MEDIUM, HIGH, CRITICAL) is determined by how many standard deviations the current value falls outside the threshold band
  • Thresholds are recalculated after every run
Schema changes are detected by comparing last_altered timestamps on tables between Observability job runs. When a change is detected, Prizm re-extracts the current column list from information_schema.columns and diffs it against the stored snapshot. Added, removed, renamed, and type-changed columns are each reported as separate events.

​
Write-Back & Data Safety

Prizm is read-only by default. The integration is metadata-only — Prizm agents access statistics, schema objects, and catalog properties but never copy or store raw customer data rows.Optional write-back capabilities include: publishing quality scores and flags to Unity Catalog table properties (using the prizm.* namespace), and bi-directional tag sync (requires ALTER TABLE SET TAGS privilege on the service principal).
No. Prizm reads metadata and statistical aggregates (counts, min/max, means, null rates). Profile jobs run aggregate SQL queries and store only the computed statistics — never individual data rows.
When write-back is enabled, Prizm publishes quality results to Unity Catalog table properties using the prizm.* key prefix — for example, prizm.quality_score, prizm.flag.schema_change, prizm.col.<column>.null_rate. These properties are queryable from the Databricks catalog browser and directly from information_schema.

​
Tag Sync

Prizm pulls tags from Databricks during the Catalog job (daily). Tags are read from:
  • system.information_schema.table_tags — table-level tag key/value pairs
  • system.information_schema.column_tags — column-level tag key/value pairs
Tags are stored in the Prizm catalog and are visible in the AI Data Catalog alongside AI-generated classifications. No ALTER TABLE privileges are required for pull-only tag sync.
Bi-directional tag sync (Prizm → Databricks) is supported as an optional feature. Enabling it requires the ALTER TABLE SET TAGS privilege on the connector service principal and must be explicitly enabled in the connector feature settings.

​
Pipeline Observability

In the connector setup wizard (or on the connector edit page), scroll to the Advanced Configuration section and toggle on Enable pipelines and jobs. Once enabled, Prizm will discover all in-scope Databricks Jobs, Workflows, and Delta Live Table pipelines and begin monitoring their run history, execution status, freshness, and DLT data quality expectations.You can also configure the No. of Runs (Days) setting (default: 7) to control how many days of historical run data Prizm backfills on the initial sync.
This setting controls the lookback window for the initial pipeline run history ingestion. When you first enable pipeline observability, Prizm fetches all pipeline runs from this many days in the past to seed the observability baseline. The default is 7 days.Increase this value if you need deeper historical context — for example, if a pipeline runs weekly and you want to see at least a few historical runs on day one. After the initial sync, all subsequent ingestion is incremental.
When pipeline observability is enabled, Prizm discovers and monitors:
  • Databricks Jobs — individual jobs and their task runs
  • Workflows — multi-task orchestrated workflows
  • Delta Live Tables (DLT) — streaming tables and materialized views, including per-flow metrics and EXPECT constraint results
  • Lakeflow Pipelines — Databricks Lakeflow declarative pipeline objects
  • Stored Procedures and Functions — cataloged with metadata; run history where available
Prizm polls for new run events every 15 minutes. If any pipeline has a freshness SLA shorter than 15 minutes, polling frequency adapts automatically.
Prizm computes the following metrics for each monitored pipeline:
Enable dashboard turns on the Pipeline Analytics dashboard for this Databricks source. The dashboard provides aggregate views across all monitored pipelines — health overview, execution trend charts, failure rate trends, and domain/product/owner-level groupings. It is separate from the per-pipeline asset pages and is designed for operational reviews and SLO reporting.The dashboard can be enabled independently of Enable pipelines and jobs, but pipeline observability data must be collected before the dashboard has anything to show.
When a pipeline run fails, Prizm correlates the failure timestamp against Databricks platform status signals. If the failure window aligns with a known Databricks service incident, the Platform Outage metric is flagged on the pipeline asset. This suppresses false-positive alerts that would otherwise fire during platform downtime — so your team isn’t paged for infrastructure issues outside your control.

​
Storage Configuration

By default, Prizm uses the storage provider configured at the organisation level (Settings → Configuration → External Storage) to store exception records, DLT pipeline output data, and profiling artefacts for all sources.The Additional Configuration section on each Databricks connector allows you to override this org-level storage setting for that specific source only — redirecting its outputs to a different storage location. This is useful when, for example, your Databricks workspace is on Azure but your org-level storage is on AWS, and you want exception records written to an Azure-hosted container instead.
Enable Overwrite storage when:
  • The Databricks workspace is in a different cloud region or provider than your org-level storage (e.g., Azure workspace, AWS org-level storage)
  • Data residency or compliance requirements mandate that exception records stay within a specific cloud environment
  • You want to isolate pipeline output artefacts for a particular source into a dedicated storage bucket or container
If none of these apply, leave the toggle off — the source will inherit the org-level storage configuration automatically.
When Overwrite storage is toggled on, two fields become required:The Storage type field (e.g., AWS, Azure) is set automatically based on your organisation’s available storage providers.

​
Connector Lifecycle

Deleting a source triggers an approval workflow based on your organization’s settings. Once approved, the source and all associated metadata are permanently deleted. The deletion cannot be undone after approval.
Existing metadata, quality scores, and alerts are retained while the connector is inactive. No new data is collected until the connector is re-enabled. If the connector is permanently deleted (after approval), all associated data is removed.
The default data retention window for time-series observability data, quality score history, and query usage history is 90 days.
The platform supports both manual and scheduled refresh/re-sync operations. Triggering a manual scan runs the job immediately without affecting the configured schedule. Any schedule update overrides the existing schedule and applies to all future executions.
No. The Performance job (which collects query history, warehouse utilization, and billing data from system.* tables) is disabled by default and must be explicitly enabled per connector instance. This is because it requires additional system table permissions and may generate noticeable query load on large workspaces.

Troubleshooting

For connection errors, permission issues, and performance problems.

Glossary

For definitions of terms used in this FAQ.