> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# FAQ

> Common questions about connecting dbt to Prizm, artifact ingestion, lineage, test results, and troubleshooting.

<script type="application/ld+json">
  {`{
            "@context": "https://schema.org",
            "@type": "TechArticle",
            "headline": "Databricks FAQ",
            "description": "Common questions about connecting dbt to Prizm, artifact ingestion, lineage, test results, and troubleshooting.",
            "url": "https://docs.dqlabs.ai/sources/databricks/faq",
            "publisher": {
              "@type": "Organization",
              "name": "DQLabs Inc",
              "logo": "https://media.brand.dev/332adc35-5bc4-4d2b-bf78-256aa4a5e414.svg"
            }
            }`}
</script>

## Connection & Authentication

<AccordionGroup>
  <Accordion title="Which authentication methods does Prizm support for Databricks?">
    Prizm supports three authentication methods for Databricks:

    * **Token:** Uses a Databricks Personal Access Token (PAT) — a long-lived credential generated in Databricks User Settings → Developer → Access Tokens. Use this only when OAuth is not available.
    * **OAuth (M2M) (Recommended):** Uses Databricks Service Principals with an OAuth Client ID and Client Secret. Tokens are short-lived and automatically refreshed by Prizm — no manual re-authentication required. Client credentials are stored encrypted in Vault.
    * **OAuth (Microsoft Entra ID):** Uses an Azure Active Directory / Microsoft Entra ID app registration with a Tenant ID, Client ID, and Client Secret. Recommended for Databricks workspaces hosted on Azure.
  </Accordion>

  <Accordion title="Why is OAuth (M2M) recommended over Token?">
    Token authentication uses a Databricks Personal Access Token (PAT), which is a long-lived credential. If a PAT is compromised, it remains valid until manually rotated or revoked. OAuth (M2M) tokens are short-lived and auto-refreshed by Prizm — eliminating the risk of long-lived credential exposure. For production environments, OAuth (M2M) provides better security posture and is aligned with Databricks' own guidance.
  </Accordion>

  <Accordion title="When should I use OAuth (Microsoft Entra ID) instead of OAuth (M2M)?">
    Use **OAuth (Microsoft Entra ID)** when your Databricks workspace is deployed on Azure and your organisation manages identities through Azure Active Directory (now Microsoft Entra ID). In this case, your service principals and app registrations live in Entra ID rather than in the Databricks account console. You will need:

    * A registered application in the Azure portal under **Microsoft Entra ID → App Registrations**
    * A **Tenant ID** (your Azure AD directory ID)
    * A **Client ID** (the Application ID of the registered app)
    * A **Client Secret** generated under the app registration → Certificates & Secrets

    If your workspace is on AWS or GCP, use **OAuth (M2M)** instead.
  </Accordion>

  <Accordion title="How do I connect multiple Databricks workspaces?">
    Create a separate connector for each workspace. Each connector has its own credentials, SQL Warehouse HTTP path, scope configuration, and job schedules. You can name connectors to distinguish environments (e.g., `databricks-prod`, `databricks-dev`).
  </Accordion>

  <Accordion title="Can I use an account-level token instead of a workspace-level token?">
    Yes. Account-level OAuth tokens (issued by `accounts.cloud.databricks.com`) provide access across all workspaces in the account. Workspace-level tokens are scoped to a single workspace. For multi-workspace monitoring, use account-level OAuth M2M with a Service Principal that has been assigned to the relevant workspaces.
  </Accordion>

  <Accordion title="How do I rotate credentials without downtime?">
    Use Vault to manage your Databricks credentials when creating the connection. With Vault-backed connections, credential rotation happens entirely within your secrets manager — you update the secret in Vault, and Prizm automatically picks up the new credentials on the next job run without any changes to the connector configuration.
  </Accordion>

  <Accordion title="How is query concurrency managed?">
    Prizm controls concurrency at the job orchestration layer. Invalid or inactive sources are excluded from scheduling.

    | **Setting**                 | **Default**                         | **Impact**                                             |
    | --------------------------- | ----------------------------------- | ------------------------------------------------------ |
    | Global concurrent jobs      | 20 (overridable via `MAX_JOBS` env) | Caps how many Prizm jobs run in parallel platform-wide |
    | Catalog metadata extraction | Batch concurrency 8                 | Parallel `information_schema` queries                  |
    | Lineage extraction          | Per DDL change trigger              | Only runs when `LAST_ALTERED` changes on a table       |
  </Accordion>
</AccordionGroup>

## Permissions

<AccordionGroup>
  <Accordion title="What is the minimum permission set for catalog discovery only?">
    For catalog-only discovery (no observability, quality, lineage, or performance):

    ```sql theme={null}
    GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_name>`;
    GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_name>`;
    GRANT SELECT ON TABLE <catalog_name>.information_schema.tables TO `<service_principal_name>`;
    GRANT SELECT ON TABLE <catalog_name>.information_schema.columns TO `<service_principal_name>`;
    GRANT SELECT ON TABLE <catalog_name>.information_schema.schemata TO `<service_principal_name>`;
    ```

    No access to `system.*` tables is required for metadata-only discovery.
  </Accordion>

  <Accordion title="Why does Prizm need access to system tables like system.access.table_lineage?">
    System tables power the advanced capabilities beyond basic cataloging:

    | System Table                   | Used For                                   |
    | :----------------------------- | :----------------------------------------- |
    | `system.access.table_lineage`  | Table-level data lineage                   |
    | `system.access.column_lineage` | Column-level lineage                       |
    | `system.lakeflow.job_runs`     | Pipeline/job run history and observability |
    | `system.query.history`         | Query performance metrics                  |
    | `system.billing.usage`         | Compute and storage cost metrics           |
  </Accordion>

  <Accordion title="Does Prizm support Hive Metastore (non-Unity Catalog) workspaces?">
    No. The Prizm Databricks connector targets Unity Catalog workspaces. Legacy Hive Metastore workspaces without Unity Catalog enabled are not supported. Databricks recommends migrating to Unity Catalog for governance and lineage capabilities.
  </Accordion>

  <Accordion title="Can I use a pre-existing service principal instead of creating a new one?">
    Yes. Any service principal with the required permission grants will work. The setup instructions use a new service principal as a recommended practice for isolation, but an existing one is fine if it has the necessary grants.
  </Accordion>
</AccordionGroup>

## Scoping & Object Inclusion

<AccordionGroup>
  <Accordion title="Can I exclude specific catalogs or schemas?">
    Yes. Use the **Include / Exclude** wildcard patterns in the Asset Scope step of the connector wizard. Exclude rules take precedence over include rules when both match the same object. For example:

    * Include: `prod_*`
    * Exclude: `prod_temp_*`, `prod_dev_*`

    The `information_schema` and `system` schemas are excluded by default and cannot be included.
  </Accordion>

  <Accordion title="Why is a table I expect not appearing in Prizm?">
    Check that:

    1. The table's catalog and schema match your include patterns and are not excluded.
    2. The service principal has `USE CATALOG`, `USE SCHEMA`, and `SELECT` on `information_schema.tables`.
    3. The Catalog job has completed at least one full run since the table was created. Check **Settings → Connectors → Logs** for the last successful Catalog job timestamp.
  </Accordion>

  <Accordion title="Does Prizm support Iceberg tables on Databricks?">
    Yes. Iceberg tables are detected via `DESCRIBE TABLE EXTENDED` (looking for `Provider = iceberg`). They are cataloged and monitored for freshness (using `DESCRIBE DETAIL` → `lastModified`) and volume. Column-level profiling depends on the Iceberg table being registered in Unity Catalog with accessible metadata.
  </Accordion>

  <Accordion title="Are external tables (e.g., tables over S3 or ADLS) supported?">
    Yes. External tables are detected by `table_type = 'EXTERNAL'` in `information_schema.tables`. Freshness is derived from `DESCRIBE DETAIL` → `lastModified`. Profiling and observability are supported; write-back is not available for external tables.
  </Accordion>
</AccordionGroup>

## Observability & Freshness

<AccordionGroup>
  <Accordion title="How does Prizm compute freshness for Delta tables?">
    Prizm uses `DESCRIBE HISTORY <table> LIMIT 1` to get the latest Delta commit timestamp. This reads only the Delta transaction log (JSON files in `_delta_log/`) — it does not scan Parquet data files. Cost is constant regardless of table size (\~50–200ms).

    This is preferred over `LAST_ALTERED` from `information_schema.tables`, which captures DDL changes only and may not reflect data writes.
  </Accordion>

  <Accordion title="What is the difference between an alert and an issue?">
    An **alert** is an automated signal that a monitored metric or quality check has breached its threshold. Alerts are generated by the observability engine without any human action.

    An **issue** is a validated, actionable problem created from one or more related alerts. Issues are tracked work items that can be assigned, commented on, and resolved. The progression is:

    | Stage           | What it means                                                        |
    | :-------------- | :------------------------------------------------------------------- |
    | **Alert**       | A metric or test breached its threshold                              |
    | **Alert Group** | Related alerts on the same asset are clustered together              |
    | **Issue**       | The group meets priority or correlation criteria and requires action |
  </Accordion>

  <Accordion title="How are anomaly detection thresholds calculated?">
    Prizm uses time-series forecasting models to compute dynamic lower and upper thresholds for each metric automatically. The system:

    * Uses up to **100 recent historical run values** as the lookback window per metric
    * Applies a **Hampel filter** (MAD-based, window 10, 3-sigma) to remove outliers before modeling
    * Selects the forecasting model automatically:

    | Model       | When used                                                           |
    | :---------- | :------------------------------------------------------------------ |
    | **EWMA**    | Low variance (stddev \< 2); narrow, stable band around the mean     |
    | **AutoReg** | Very short series (2–3 points); AR(1) forecast with residual bounds |
    | **Prophet** | Default for higher-variance metrics without seasonal patterns       |
    | **SARIMAX** | 60+ data points with a detected seasonal pattern                    |

    * A new metric spends its first **more than 5 historical collection runs** in **Learning mode**, accumulating a baseline, before alerts begin firing. This threshold counts collection **runs**, not elapsed time.
    * **Alert priority** (LOW, MEDIUM, HIGH, CRITICAL) is determined by how many standard deviations the current value falls outside the threshold band
    * Thresholds are **recalculated after every run**
  </Accordion>

  <Accordion title="How does Prizm detect schema changes in Databricks?">
    Schema changes are detected by comparing `last_altered` timestamps on tables between Observability job runs. When a change is detected, Prizm re-extracts the current column list from `information_schema.columns` and diffs it against the stored snapshot. Added, removed, renamed, and type-changed columns are each reported as separate events.
  </Accordion>
</AccordionGroup>

## Write-Back & Data Safety

<AccordionGroup>
  <Accordion title="Does Prizm write data back to Databricks?">
    Prizm is read-only by default. The integration is metadata-only — Prizm agents access statistics, schema objects, and catalog properties but never copy or store raw customer data rows.

    Optional write-back capabilities include: publishing quality scores and flags to Unity Catalog table properties (using the `prizm.*` namespace), and bi-directional tag sync (requires `ALTER TABLE SET TAGS` privilege on the service principal).
  </Accordion>

  <Accordion title="Does Prizm store or copy my Databricks data?">
    No. Prizm reads metadata and statistical aggregates (counts, min/max, means, null rates). Profile jobs run aggregate SQL queries and store only the computed statistics — never individual data rows.
  </Accordion>

  <Accordion title="What quality intelligence can Prizm write back to Unity Catalog?">
    When write-back is enabled, Prizm publishes quality results to Unity Catalog table properties using the `prizm.*` key prefix — for example, `prizm.quality_score`, `prizm.flag.schema_change`, `prizm.col.<column>.null_rate`. These properties are queryable from the Databricks catalog browser and directly from `information_schema`.
  </Accordion>
</AccordionGroup>

## Tag Sync

<AccordionGroup>
  <Accordion title="How does tag sync work for Databricks?">
    Prizm pulls tags from Databricks during the Catalog job (daily). Tags are read from:

    * `system.information_schema.table_tags` — table-level tag key/value pairs
    * `system.information_schema.column_tags` — column-level tag key/value pairs

    Tags are stored in the Prizm catalog and are visible in the AI Data Catalog alongside AI-generated classifications. No `ALTER TABLE` privileges are required for pull-only tag sync.
  </Accordion>

  <Accordion title="Can Prizm sync tags back to Databricks?">
    Bi-directional tag sync (Prizm → Databricks) is supported as an optional feature. Enabling it requires the `ALTER TABLE SET TAGS` privilege on the connector service principal and must be explicitly enabled in the connector feature settings.
  </Accordion>
</AccordionGroup>

## Pipeline Observability

<AccordionGroup>
  <Accordion title="How do I enable pipeline and job monitoring for a Databricks source?">
    In the connector setup wizard (or on the connector edit page), scroll to the **Advanced Configuration** section and toggle on **Enable pipelines and jobs**. Once enabled, Prizm will discover all in-scope Databricks Jobs, Workflows, and Delta Live Table pipelines and begin monitoring their run history, execution status, freshness, and DLT data quality expectations.

    You can also configure the **No. of Runs (Days)** setting (default: 7) to control how many days of historical run data Prizm backfills on the initial sync.
  </Accordion>

  <Accordion title="What does the No. of Runs (Days) setting control?">
    This setting controls the **lookback window** for the initial pipeline run history ingestion. When you first enable pipeline observability, Prizm fetches all pipeline runs from this many days in the past to seed the observability baseline. The default is **7 days**.

    Increase this value if you need deeper historical context — for example, if a pipeline runs weekly and you want to see at least a few historical runs on day one. After the initial sync, all subsequent ingestion is incremental.
  </Accordion>

  <Accordion title="What pipeline types does Prizm monitor?">
    When pipeline observability is enabled, Prizm discovers and monitors:

    * **Databricks Jobs** — individual jobs and their task runs
    * **Workflows** — multi-task orchestrated workflows
    * **Delta Live Tables (DLT)** — streaming tables and materialized views, including per-flow metrics and `EXPECT` constraint results
    * **Lakeflow Pipelines** — Databricks Lakeflow declarative pipeline objects
    * **Stored Procedures** and **Functions** — cataloged with metadata; run history where available

    Prizm polls for new run events every **15 minutes**. If any pipeline has a freshness SLA shorter than 15 minutes, polling frequency adapts automatically.
  </Accordion>

  <Accordion title="What pipeline metrics does Prizm compute?">
    Prizm computes the following metrics for each monitored pipeline:

    | Metric               | Description                                                                      |
    | :------------------- | :------------------------------------------------------------------------------- |
    | **Execution Status** | Latest run result — SUCCESS, FAILED, CANCELLED, RUNNING                          |
    | **Freshness**        | Time since the last successful run completion                                    |
    | **Rows Affected**    | Output rows written, upserted, and deleted per run                               |
    | **Execution Time**   | Wall-clock duration per run — tracked over time for SLA regression detection     |
    | **Failure Rate**     | Rolling % of runs that failed over the lookback window                           |
    | **Dropped Records**  | Records silently discarded due to DLT `EXPECT DROP` constraints                  |
    | **Failed Records**   | Records that violated DLT expectations but were not dropped                      |
    | **Platform Outage**  | Flag indicating whether a failure correlates with a Databricks platform incident |
  </Accordion>

  <Accordion title="What is the Enable dashboard setting?">
    **Enable dashboard** turns on the **Pipeline Analytics dashboard** for this Databricks source. The dashboard provides aggregate views across all monitored pipelines — health overview, execution trend charts, failure rate trends, and domain/product/owner-level groupings. It is separate from the per-pipeline asset pages and is designed for operational reviews and SLO reporting.

    The dashboard can be enabled independently of **Enable pipelines and jobs**, but pipeline observability data must be collected before the dashboard has anything to show.
  </Accordion>

  <Accordion title="How does Prizm distinguish a platform outage from a data issue?">
    When a pipeline run fails, Prizm correlates the failure timestamp against Databricks platform status signals. If the failure window aligns with a known Databricks service incident, the **Platform Outage** metric is flagged on the pipeline asset. This suppresses false-positive alerts that would otherwise fire during platform downtime — so your team isn't paged for infrastructure issues outside your control.
  </Accordion>
</AccordionGroup>

## Storage Configuration

<AccordionGroup>
  <Accordion title="What is the Additional Configuration / storage override for?">
    By default, Prizm uses the storage provider configured at the **organisation level** (Settings → Configuration → External Storage) to store exception records, DLT pipeline output data, and profiling artefacts for all sources.

    The **Additional Configuration** section on each Databricks connector allows you to override this org-level storage setting for that specific source only — redirecting its outputs to a different storage location. This is useful when, for example, your Databricks workspace is on Azure but your org-level storage is on AWS, and you want exception records written to an Azure-hosted container instead.
  </Accordion>

  <Accordion title="When should I enable storage override for a Databricks source?">
    Enable **Overwrite storage** when:

    * The Databricks workspace is in a different cloud region or provider than your org-level storage (e.g., Azure workspace, AWS org-level storage)
    * Data residency or compliance requirements mandate that exception records stay within a specific cloud environment
    * You want to isolate pipeline output artefacts for a particular source into a dedicated storage bucket or container

    If none of these apply, leave the toggle off — the source will inherit the org-level storage configuration automatically.
  </Accordion>

  <Accordion title="What fields are required when enabling storage override?">
    When **Overwrite storage** is toggled on, two fields become required:

    | Field         | Description                                                                 |
    | :------------ | :-------------------------------------------------------------------------- |
    | **Schema**    | The target schema or container name within the override storage destination |
    | **Directory** | The path or folder within the container where outputs will be written       |

    The **Storage type** field (e.g., AWS, Azure) is set automatically based on your organisation's available storage providers.
  </Accordion>
</AccordionGroup>

## Connector Lifecycle

<AccordionGroup>
  <Accordion title="What happens if I delete a connector?">
    Deleting a source triggers an approval workflow based on your organization's settings. Once approved, the source and all associated metadata are permanently deleted. The deletion cannot be undone after approval.
  </Accordion>

  <Accordion title="What happens to existing data and alerts if a connector is paused or disconnected?">
    Existing metadata, quality scores, and alerts are retained while the connector is inactive. No new data is collected until the connector is re-enabled. If the connector is permanently deleted (after approval), all associated data is removed.
  </Accordion>

  <Accordion title="How long does Prizm retain historical data?">
    The default data retention window for time-series observability data, quality score history, and query usage history is **90 days**.
  </Accordion>

  <Accordion title="How does manual re-sync work alongside scheduled jobs?">
    The platform supports both manual and scheduled refresh/re-sync operations. Triggering a manual scan runs the job immediately without affecting the configured schedule. Any schedule update overrides the existing schedule and applies to all future executions.
  </Accordion>

  <Accordion title="Is the Performance job enabled by default?">
    No. The Performance job (which collects query history, warehouse utilization, and billing data from `system.*` tables) is disabled by default and must be explicitly enabled per connector instance. This is because it requires additional system table permissions and may generate noticeable query load on large workspaces.
  </Accordion>
</AccordionGroup>

<CardGroup cols={2}>
  <Card title="Troubleshooting" icon="plug" href="/help/troubleshooting">
    For connection errors, permission issues, and performance problems.
  </Card>

  <Card title="Glossary" icon="circle-question" href="/help/glossary">
    For definitions of terms used in this FAQ.
  </Card>
</CardGroup>