> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Plugin Setup

> Install and configure the Prizm Airflow Plugin so DAG and task run metadata is pushed to Prizm automatically the moment each run completes.

<script type="application/ld+json">
  {`{
            "@context": "https://schema.org",
            "@type": "TechArticle",
            "headline": "Airflow Plugin Setup",
            "description": "Install and configure the Prizm Airflow Plugin so DAG and task run metadata is pushed to Prizm automatically the moment each run completes.",
            "url": "https://docs.dqlabs.ai/sources/airflow/plugin-setup",
            "publisher": {
              "@type": "Organization",
              "name": "DQLabs Inc",
              "logo": "https://media.brand.dev/332adc35-5bc4-4d2b-bf78-256aa4a5e414.svg"
            }
            }`}
</script>

The Prizm Airflow Plugin hooks into Airflow's run lifecycle. When a DAG run completes — success or failure — the plugin sends that run's metadata to Prizm immediately, with no polling interval and no Airflow credentials stored in Prizm.

<Note>
  The plugin never reads or transmits Airflow connection secrets, variable values outside the ones listed in this guide, or data from your warehouses. It reports only DAG, task, and run metadata — plus, optionally, source code and failed-task logs, both of which you can disable.
</Note>

## Before you start

* Admin access to your Airflow deployment — you need to add plugin files and restart the scheduler
* Admin or Editor access to your Prizm workspace
* **Airflow 2.5 or later**, with the `requests` package available in the scheduler's Python environment (present in most Airflow installs by default)
* Network access from the Airflow scheduler host to your Prizm workspace URL

***

## Installation

<Steps>
  <Step title="Create the Airflow source in Prizm">
    In Prizm, go to **Sources → Add Source → Airflow** and set the **Integration Type** to **Plugin**.

    Give the source a **Name** and write it down exactly as typed — you will reuse it verbatim as `prizm_connection_name` in Step 5. Prizm uses this name to match incoming plugin data to the right source.

    <Warning>
      The name must match **exactly**, including case and spacing. The plugin sends its configured connection name with every request. If it doesn't match the source name character for character, Prizm rejects the data.
    </Warning>
  </Step>

  <Step title="Generate a Prizm API access token">
    Go to **Settings → Organization → Access Tokens** and generate a new token.

    <Warning>
      Copy the token immediately — it is shown only once. Store it as a secret: an Airflow Variable marked secret, or a secrets backend. Never commit it to source control.
    </Warning>
  </Step>

  <Step title="Download the plugin file">
    Download `prizm_airflow_plugin.py` from the Prizm connector assets bucket:

    ```bash theme={null}
    curl -O https://prizm-connector-assets.s3.us-east-1.amazonaws.com/airflow/prizm_airflow_plugin.py
    ```

    If you received the file as a direct attachment, skip this step.
  </Step>

  <Step title="Place the file in your Airflow plugins folder">
    Copy the file into the `plugins/` directory under your Airflow home, keeping the filename as-is:

    ```bash theme={null}
    # $AIRFLOW_HOME defaults to ~/airflow if not set
    cp prizm_airflow_plugin.py "$AIRFLOW_HOME/plugins/prizm_airflow_plugin.py"
    ```

    <Note>
      **Docker, Astronomer, Composer, or MWAA?** Mount or bake the file into whatever location your platform treats as the Airflow `plugins/` folder. The install mechanism is identical — only the path convention differs by platform.
    </Note>
  </Step>

  <Step title="Configure the plugin">
    Set the following as **Airflow Variables** via **Admin → Variables** in the Airflow UI, or via the CLI:

    ```bash theme={null}
    airflow variables set prizm_url "https://your-workspace.prizmdata.ai"
    airflow variables set prizm_api_token "<token from Step 2>"
    airflow variables set prizm_connection_name "<source name from Step 1>"
    ```

    The plugin also reads `PRIZM_URL`, `PRIZM_API_TOKEN`, and `PRIZM_CONNECTION_NAME` from the process **environment** first, falling back to Airflow Variables. Use environment injection (Docker, Helm) if your scheduler is already configured that way.

    You can use an existing Airflow secrets backend—such as **AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, or HashiCorp Vault**—instead of storing values directly in Airflow Variables. Store `prizm_url`,  `prizm_api_token`, and `prizm_connection_name`using the backend’s Airflow Variable naming convention; the plugin resolves them automatically. Alternatively, inject `prizm_url`,  `prizm_api_token`, and `prizm_connection_name` as environment variables.

    Optional variables for filtering and log capture are listed in the **Variable reference** section below.
  </Step>

  <Step title="Restart the scheduler">
    The plugin loads only on scheduler startup. Restart the scheduler — and any workers running the same image — after placing the file and setting the variables.

    **systemd:**

    ```bash theme={null}
    sudo systemctl restart airflow-scheduler
    ```

    **Docker Compose:**

    ```bash theme={null}
    docker compose restart airflow-scheduler
    ```

    The exact command depends on your deployment. The requirement is that the scheduler process restarts and loads the plugin from the same `AIRFLOW_HOME` where you placed the file.
  </Step>

  <Step title="Verify">
    Trigger or wait for any DAG run to complete. In Prizm, open **Sources** and select your source — the run appears within a few minutes, along with the DAG's tasks and lineage.

    If nothing shows up, see [Troubleshooting](#troubleshooting) below.
  </Step>
</Steps>

***

## How the plugin works

The plugin registers two listeners on scheduler startup:

* **`on_dag_run_success`** — fires when a DAG run completes successfully
* **`on_dag_run_failed`** — fires when a DAG run ends in a failure state

On each trigger, the plugin builds a payload for that run and POSTs it to `{prizm_url}/api/v1/schedule/airflow/plugin` using the configured Bearer token.

**Limitations to be aware of:**

* **Failed pushes are dropped.** If Prizm is briefly unreachable when a run completes, that run's data is lost. Check the scheduler logs for entries from `prizm_airflow_plugin` to detect dropped pushes.
* **Only success and failure fire.** Runs that are killed, marked as zombie, or otherwise end without triggering either listener produce no push — leaving a gap in the catalog.
* **Plugin pushes never prune.** Each push contains data for the run that just finished, not a complete DAG inventory. Deactivating deleted or renamed DAGs requires a full sync from an API Pull or CLI source.

***

## Variable reference

| Variable                    | Required     | Default  | Description                                                                                                                                        |
| :-------------------------- | :----------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------- |
| `prizm_url`                 | **Required** | —        | Your Prizm workspace base URL. The plugin appends the ingest path itself; a trailing slash is fine.                                                |
| `prizm_api_token`           | **Required** | —        | Access token from Step 2, sent as a Bearer token with every push.                                                                                  |
| `prizm_connection_name`     | **Required** | —        | Must exactly match the Airflow source's Name in Prizm.                                                                                             |
| `prizm_allowed_dags`        | Optional     | All DAGs | Comma-separated glob patterns (e.g., `prod_*, etl_*`) — only matching DAGs are reported.                                                           |
| `prizm_excluded_dags`       | Optional     | None     | Comma-separated glob patterns to exclude. Takes priority over `prizm_allowed_dags`.                                                                |
| `prizm_extract_source_code` | Optional     | `true`   | Sends each DAG's Python source so Prizm can display it and derive cross-DAG lineage. Set to `false` to skip.                                       |
| `prizm_include_full_log`    | Optional     | `false`  | Sends the full task log for **failed** tasks only. Successful tasks never send log content. Useful for troubleshooting failures directly in Prizm. |

***

## Troubleshooting

| Message or symptom                                        | Cause and fix                                                                                                                                                                   |
| :-------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Source not found, inactive, or not Airflow type`         | No active Airflow source in Prizm matches `prizm_connection_name`. Re-check the name against Step 1 — it is case-sensitive — and confirm the source Status is active.           |
| `Source is not configured for Airflow plugin integration` | The source exists but its Integration Type isn't Plugin. Create a new source with Plugin selected.                                                                              |
| `connection_name does not match Prizm source name`        | `prizm_connection_name` resolved to a source, but the value doesn't exactly match that source's name string. Fix the Airflow Variable and restart the scheduler.                |
| `connection_name is required`                             | `prizm_connection_name` is unset or empty. Confirm it was set in Step 5 and that the scheduler was restarted afterward.                                                         |
| No data arrives, no error visible                         | Check scheduler logs for entries from `prizm_airflow_plugin`. Confirm `prizm_url` is reachable from the scheduler's network, and that `prizm_api_token` is set and not revoked. |
| Plugin installed but never fires                          | Scheduler was not restarted after file placement. Restart the scheduler process.                                                                                                |
| Plugin fires for some runs but not others                 | Failed pushes are dropped on transient network issues, or some runs ended without triggering a listener (killed/zombie). Check scheduler logs.                                  |

***

## Next Steps

<CardGroup cols={2}>
  <Card title="Setup" icon="plug" href="/sources/airflow/setup">
    Create an Airflow source and configure API Pull or CLI mode.
  </Card>

  <Card title="What We Collect" icon="database" href="/sources/airflow/what-we-collect">
    Full breakdown of every field Prizm catalogs from DAG, task, and run metadata.
  </Card>
</CardGroup>