> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dqlabs.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# What We Collect

> Field-level breakdown of every metadata object Prizm extracts from Apache Airflow — DAGs, tasks, run history, and operational metrics.

<script type="application/ld+json">
  {`{
            "@context": "https://schema.org",
            "@type": "TechArticle",
            "headline": "Airflow: What We Collect",
            "description": "Field-level breakdown of every metadata object Prizm extracts from Apache Airflow - DAGs, tasks, run history, and operational metrics.",
            "url": "https://docs.dqlabs.ai/sources/airflow/what-we-collect",
            "publisher": {
              "@type": "Organization",
              "name": "DQLabs Inc",
              "logo": "https://media.brand.dev/332adc35-5bc4-4d2b-bf78-256aa4a5e414.svg"
            }
            }`}
</script>

Prizm runs two types of sync for each Airflow source. **Technical sync** catalogs what exists — DAGs, tasks, and run records. **Operational sync** evaluates how things are performing — it works entirely from the data Technical sync already persisted, without ever calling Airflow again.

## Technical Sync

### DAGs

Each active DAG in Airflow becomes one Prizm asset. Prizm captures:

| Field                  | Description                                                                                                                              |
| :--------------------- | :--------------------------------------------------------------------------------------------------------------------------------------- |
| **DAG ID**             | Airflow's unique identifier for the DAG                                                                                                  |
| **Description**        | DAG description from Airflow                                                                                                             |
| **Schedule interval**  | The configured cadence (e.g., `@daily`, `0 6 * * *`). Prizm uses this to compute the expected freshness window for the Freshness metric. |
| **Is paused**          | Whether the DAG is currently paused in Airflow                                                                                           |
| **Is active**          | Whether the DAG is currently active                                                                                                      |
| **Tags**               | Airflow tags — surfaced in Prizm for filtering and search                                                                                |
| **Owners**             | Airflow owner list — mapped to governance ownership in Prizm where governance rules are configured                                       |
| **File location**      | Path to the DAG file on disk (`fileloc`) — used for cross-DAG lineage detection and PR impact analysis                                   |
| **Source code**        | DAG Python source (`compiled_code`) — required for cross-DAG lineage and PR impact analysis; can be disabled per-source                  |
| **Created / Modified** | DAG creation and last-modified timestamps                                                                                                |

### Tasks

Each task within a DAG becomes a child Prizm asset, parented to its DAG:

| Field                   | Description                                                                                   |
| :---------------------- | :-------------------------------------------------------------------------------------------- |
| **Task ID**             | Airflow's unique task identifier within the DAG                                               |
| **Operator name**       | The Airflow operator class used (e.g., `PythonOperator`, `BashOperator`, `SnowflakeOperator`) |
| **Owner**               | Task-level owner, if set separately from the DAG owner                                        |
| **Pool**                | Airflow pool the task runs in                                                                 |
| **Retries**             | Number of configured retry attempts                                                           |
| **Upstream task IDs**   | Tasks this task depends on — renders within-DAG task lineage                                  |
| **Downstream task IDs** | Tasks that depend on this task                                                                |

### DAG Runs

Run history is the input for all operational metrics. Prizm captures the following per run:

| Field              | Description                                                                            |
| :----------------- | :------------------------------------------------------------------------------------- |
| **Run ID**         | Unique identifier for the DAG run                                                      |
| **Run type**       | `scheduled`, `manual`, `backfill`, or `dataset_triggered`                              |
| **State**          | Final run state: `success`, `failed`, `running`, `queued`, `scheduled`, `up_for_retry` |
| **Execution date** | Logical execution date of the run                                                      |
| **Start date**     | When the run started executing                                                         |
| **End date**       | When the run completed                                                                 |
| **Duration**       | Computed from `end_date` minus `start_date`, in milliseconds                           |

### Task Instances

Per-task run records, used for task-level metrics and failure investigation:

| Field                | Description                                                                                |
| :------------------- | :----------------------------------------------------------------------------------------- |
| **Task ID**          | Which task within the DAG                                                                  |
| **Run ID**           | Which DAG run this instance belongs to                                                     |
| **State**            | Task instance state: `success`, `failed`, `upstream_failed`, `skipped`, `retry`, `running` |
| **Try number**       | Which attempt this was (1 = first try, 2 = first retry, etc.)                              |
| **Start / End date** | Execution window for this task instance                                                    |
| **Duration**         | Task instance execution time in milliseconds                                               |
| **Error message**    | Failure reason, if the task instance failed                                                |
| **Log reference**    | Link to the task log — viewable from the Prizm asset without navigating to Airflow         |

### Deletion handling

On each full sync (API Pull or CLI), Prizm compares the DAG IDs it just saw against what it previously cataloged. Any DAG previously active and now missing is **deactivated**, and its child tasks cascade to deactivated. Plugin pushes never trigger pruning — only full syncs from API Pull or CLI can deactivate DAGs.

<Warning>
  There is no rename detection. Renaming a DAG in Airflow reads as one DAG disappearing and a new one appearing — not a rename. Lineage, governance metadata, and alert history attached to the old asset do not carry over. Plan renames accordingly.
</Warning>

***

## Operational Sync & Metrics

Operational sync evaluates run health from the data Technical sync already persisted. It never calls Airflow.

### Metrics

| Metric               | Level | Default | How it's calculated                                                                                                                                                                                                                            |
| :------------------- | :---- | :------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Freshness**        | DAG   | On      | Time elapsed between consecutive DAG run watermarks. Auto-thresholded — the normal interval is learned from up to 100 prior historical values for this specific DAG. An hourly DAG and a weekly DAG maintain separate, appropriate thresholds. |
| **Execution Status** | DAG   | Off     | `1.0` if the run's final state is `success` or `completed`; `0.0` for `failed` or `error`. Fixed threshold — any value below `1.0` is a breach.                                                                                                |
| **Job Duration**     | DAG   | Off     | Total DAG run duration in milliseconds. Auto-thresholded against historical run durations for this specific DAG.                                                                                                                               |
| **Execution Time**   | Task  | On      | Per-task instance duration in milliseconds. Auto-thresholded — isolates which task within a DAG is running slower than its historical norm.                                                                                                    |

**Auto-thresholded metrics** derive a normal band from up to 100 prior historical values. New DAGs or tasks with insufficient run history will show metric values without an alert threshold until enough history accumulates.

### Failure alerts

Two alert options fire on run state directly — independent of whether any metrics are enabled:

| Alert                  | Trigger                                                                                |
| :--------------------- | :------------------------------------------------------------------------------------- |
| **DAG Alert Failure**  | A DAG run ending in `failed` or `error` state                                          |
| **Task Alert Failure** | A task instance ending in `failed`, `error`, or `upstream_failed` state within any run |

Configure both in the source settings under [Sources → your Airflow source → Edit](/connectors/airflow/setup#alert-fields).

***

## Lineage

Prizm derives two types of lineage from Airflow metadata — warehouse lineage (which tables tasks read and write) and cross-DAG lineage (which DAGs trigger or wait on each other).

### Warehouse lineage (cross-source)

| What Prizm collects   | Description                                                                                                         |
| :-------------------- | :------------------------------------------------------------------------------------------------------------------ |
| **Raw lineage facts** | Source and target table names extracted per task run, via Query Tag parsing or OpenLineage events                   |
| **Responsible task**  | Which Airflow task performed the transformation — edges are tagged at the task level, not just the DAG level        |
| **Resolved edges**    | Raw facts matched against cataloged Snowflake assets to produce clickable lineage edges (Snowflake only; see below) |

**Warehouse support:**

| Warehouse      | Raw facts | Visible edges |
| :------------- | :-------- | :------------ |
| **Snowflake**  | ✅         | ✅             |
| **Databricks** | ✅         | ❌             |
| **MSSQL**      | ✅         | ❌             |

### Cross-DAG lineage

Prizm inspects each DAG's Python source code to detect control-flow dependencies between DAGs. Requires source code extraction to be enabled.

| Pattern detected                                | What Prizm records                                                   |
| :---------------------------------------------- | :------------------------------------------------------------------- |
| **`TriggerDagRunOperator`**                     | Edge from the triggering task to the root tasks of the triggered DAG |
| **`ExternalTaskSensor` / `ExternalTaskMarker`** | Edge from the watched upstream task to the sensing task              |

### What Prizm does not collect

* Data from your warehouses — Prizm reads metadata only, not row-level data
* Airflow connection secrets or variable values outside the plugin variables listed in [Plugin Setup](/connectors/airflow/plugin-setup)
* Lineage for runs that fail, are killed, or end as zombies (no completed run = no lineage event)

For full details on lineage methods, resolution logic, and PR impact analysis, see the [Lineage](/connectors/airflow/lineage) page.

***

## Sync frequency

| Sync type            | API Pull                                 | CLI                          | Plugin                     |
| :------------------- | :--------------------------------------- | :--------------------------- | :------------------------- |
| **Technical sync**   | Daily at 02:30 UTC (default, adjustable) | When you run the CLI command | On each DAG run completion |
| **Operational sync** | After each Technical sync                | After each CLI upload        | After each plugin push     |

## Next Steps

<CardGroup cols={2}>
  <Card title="Setup" icon="plug" href="/sources/airflow/setup">
    Create an Airflow source and configure API Pull or CLI mode.
  </Card>

  <Card title="What We Collect" icon="database" href="/sources/airflow/what-we-collect">
    Full breakdown of every field Prizm catalogs from DAG, task, and run metadata.
  </Card>
</CardGroup>