> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ADLS Overview

> What the Azure Data Lake Storage connector covers, the two ways Prizm can process your files, and what Unity Catalog Iceberg materialization adds.

<script type="application/ld+json">
  {`{
            "@context": "https://schema.org",
            "@type": "TechArticle",
            "headline": "ADLS Overview",
            "description": "What the Azure Data Lake Storage connector covers, the two ways Prizm can process your files, and what Unity Catalog Iceberg materialization adds.",
            "url": "https://docs.dqlabs.ai/sources/adls/overview",
            "publisher": {
              "@type": "Organization",
              "name": "DQLabs Inc",
              "logo": "https://media.brand.dev/332adc35-5bc4-4d2b-bf78-256aa4a5e414.svg"
            }
            }`}
</script>

Prizm connects to Azure Data Lake Storage (ADLS) Gen2 to catalog, monitor, and run quality checks against structured, semi-structured, and raw files — without duplicating the underlying data. A lightweight Prizm agent runs inside your environment and connects outbound only; there are no inbound connections, no VPN, and no firewall changes beyond outbound port 443.

## Two ways to process your files

| Option                                | What happens                                                                        | Setup required                                                                |
| :------------------------------------ | :---------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- |
| **Prizm Spark** (default)             | Prizm processes your files on its own managed Spark engine                          | None — works out of the box                                                   |
| **Your own Databricks Spark cluster** | Prizm creates jobs in your Databricks workspace and processes files on your compute | Azure, Unity Catalog, and Databricks setup (see [Setup](/sources/adls/setup)) |

If you're happy using Prizm Spark and don't need Unity Catalog Iceberg materialization, no additional setup is required beyond configuring the source itself.

## Unity Catalog Iceberg materialization (optional)

When you connect your own Databricks cluster, Prizm can also **materialize selected ADLS assets as Unity Catalog–managed Iceberg-compatible tables** (via Delta UniForm), so they become persistent, governed, queryable objects — with a `catalog.schema.table` identity and Unity Catalog–enforced permissions — rather than existing only transiently inside a job.

|                                  | Without Iceberg materialization                       | With Iceberg materialization                                                                        |
| :------------------------------- | :---------------------------------------------------- | :-------------------------------------------------------------------------------------------------- |
| **Where data lives after a job** | Nowhere — held in-cluster only for the run's duration | A persistent Unity Catalog–managed Delta table with Iceberg compatibility                           |
| **Downstream access**            | None — each run is self-contained                     | Any Iceberg REST–compatible engine (Trino, external Spark, Snowflake, DuckDB) can query it directly |
| **Governance**                   | N/A                                                   | Unity Catalog–enforced permissions                                                                  |

Because materialized tables expose Delta's UniForm Iceberg metadata, engines like Trino can query them directly through the metastore's Iceberg REST endpoint — no export step, no second copy of the data.

## Architecture

When using your own Databricks cluster, the Prizm Agent's **Core**, **Connector**, and **Trino** components run inside your Azure environment (not in the DQLabs-managed platform):

* **Two independent Azure connections.** The Agent authenticates to Azure resources via Microsoft Entra, separate from its outbound-only channel to Prizm.
* **Connector drives the Databricks job** — it invokes the notebook running on your Databricks cluster.
* **Databricks reads and writes ADLS** — the cluster reads source data from your ADLS source storage account and writes materialized Iceberg files to a separate ADLS storage account/container you designate for Prizm.
* **Trino surfaces metrics and metadata** back into Prizm for quality jobs and materialization tracking.

## Supported Environments

| Area                                  | Supported Today                                                                                                              |
| :------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------- |
| Storage                               | Azure Data Lake Storage (ADLS Gen2)                                                                                          |
| External compute                      | Databricks only — Amazon EMR and Cloudera are not supported                                                                  |
| Compute engine hosting                | Both Azure- and AWS-hosted Databricks are supported                                                                          |
| Unity Catalog Iceberg materialization | Azure Databricks workspaces with Unity Catalog enabled (external-engine access is currently in Public Preview on Databricks) |

## Next Steps

<CardGroup cols={2}>
  <Card title="Setup" icon="plug" href="/sources/adls/setup">
    Connect ADLS to Prizm — authentication, Unity Catalog, and Databricks configuration.
  </Card>

  <Card title="What We Collect" icon="database" href="/sources/adls/what-we-collect">
    Job types, compute metric types, and Delta Lake table handling.
  </Card>

  <Card title="FAQ" icon="circle-question" href="/sources/adls/faq">
    Common questions and troubleshooting for the ADLS + Databricks connector.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/help/troubleshooting">
    Diagnose connection, permission, and performance errors.
  </Card>
</CardGroup>