> ## Documentation Index
> Fetch the complete documentation index at: https://openmetadata-docs-release-watchers.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Extract MWAA Metadata

> Ingest pipeline metadata from Amazon MWAA. Use the REST API connection for an external MWAA environment, or run the ingestion inside MWAA when OpenMetadata cannot reach AWS.

# Extract MWAA Metadata

There are two ways to extract metadata from an Amazon MWAA environment, depending on where the ingestion process runs:

* **[REST API connection](/v2.0.x/connectors/pipeline/airflow/rest-api-connection#4-mwaa-configuration-amazon-managed-workflows-for-apache-airflow) (recommended).** The ingestion runs from OpenMetadata (UI-scheduled or via the CLI) and reads DAGs through the MWAA web server, using AWS credentials. No access to the underlying database is needed, and nothing has to be installed inside MWAA for metadata extraction (see below for the additional OpenLineage setup required for table-level lineage).
* **Running the ingestion inside MWAA.** A DAG deployed in MWAA extracts metadata from the local Airflow backend and pushes it to OpenMetadata. Use this when your MWAA environment runs an Airflow version older than 2.4.3, or when the ingestion cannot be given AWS credentials.

<Warning>
  **Do not use the `Backend` connection type from outside MWAA.** `Backend` reads the metadata database of the Airflow environment the ingestion workflow is *running inside*. OpenMetadata uses the `hostPort` field only to build clickable DAG links, never to fetch metadata. If you create an Airflow service pointing at your MWAA URL with a `Backend` connection and run it from OpenMetadata's ingestion pipeline, you will silently ingest the DAGs of OpenMetadata's own Airflow instead of MWAA's. `Backend` is only valid when the workflow executes inside MWAA itself (option 2 below).

  Direct database connections (`Postgres`/`MySQL`) aren't a general alternative either. MWAA's metadata database lives in an AWS-managed private network, reachable only from inside that VPC or a peered network — the same constraint a private MWAA web server has for the REST API connection below. See the ECS approach below for a working example from inside the VPC. It isn't an option for ingestion running entirely outside AWS.
</Warning>

## 1. REST API Connection (Recommended)

OpenMetadata calls the MWAA control plane's `InvokeRestApi` action directly with your AWS credentials — no login token is created or exchanged. That action proxies the request to the Airflow REST API to read DAGs, runs, and task states. OpenMetadata always uses the Airflow v1 REST API for MWAA connections. MWAA handles the Airflow 2/3 version difference internally, so no client-side version detection happens.

### Requirements

* The MWAA environment must run **Airflow 2.4.3 or newer** (the minimum version supported by the MWAA `InvokeRestApi` API).
* The IAM identity used in the connection needs `airflow:InvokeRestApi`, scoped to the environment's Airflow role, not the environment itself:

```json theme={null}
{
  "Effect": "Allow",
  "Action": "airflow:InvokeRestApi",
  "Resource": "arn:aws:airflow:<region>:<account-id>:role/<environment-name>/<airflow-role>"
}
```

Replace `<airflow-role>` with the Apache Airflow role to grant: `Admin`, `Op`, `User`, `Viewer`, or `Public`.

* If the MWAA environment uses a **private** web server, `InvokeRestApi` can't be called from outside its VPC. The ingestion runtime must run inside, or have network connectivity to, that VPC.

### Configuration

In the UI, create an **Airflow** pipeline service, select the **REST API Connection** type, and choose **MWAA** authentication. The equivalent YAML:

```yaml theme={null}
source:
  type: airflow
  serviceName: airflow_mwaa
  serviceConnection:
    config:
      type: Airflow
      hostPort: https://<environment-id>.c2.airflow.<region>.on.aws
      numberOfStatus: 10
      connection:
        type: RestAPI
        authConfig:
          mwaaConfig:
            mwaaEnvironmentName: <environment-name>
            awsConfig:
              awsRegion: <region>
              awsAccessKeyId: <access-key-id>
              awsSecretAccessKey: <secret-access-key>
              awsSessionToken: <session-token>  # only for temporary (STS) credentials
        apiVersion: auto
        verifySSL: true
  sourceConfig:
    config:
      type: PipelineMetadata
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: INFO
  openMetadataServerConfig:
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>
```

When the ingestion runs on AWS infrastructure with an attached IAM role (EC2, ECS, Lambda), omit the access key, secret, and session token. OpenMetadata picks up the role's credentials automatically. Cross-account access is supported via `assumeRoleArn`.

For the full parameter reference, see the [REST API Connection documentation](/v2.0.x/connectors/pipeline/airflow/rest-api-connection).

<Info>
  The REST API alone doesn't capture table-level lineage. It requires the OpenLineage provider configured in MWAA. For the required setup, see the [OpenLineage Setup Summary](/v2.0.x/connectors/pipeline/airflow/rest-api-connection#openlineage-setup-summary).
</Info>

## 2. Running the Ingestion Inside MWAA

If the REST API connection isn't an option, run the ingestion as a DAG inside MWAA using the `Backend` connection, which reads MWAA's own metadata database directly. The OpenMetadata server must be reachable **from** MWAA. For more information about running connectors from MWAA, see [Run the ingestion from AWS MWAA](/v2.0.x/deployment/ingestion/external/mwaa). There are three supported approaches:

* Install the `openmetadata-ingestion` package as a requirement in the Airflow environment, then run the process using a `PythonOperator`.
* Configure an ECS cluster and run the ingestion as an ECS Operator.
* Install a plugin and run the ingestion with the `PythonVirtualenvOperator`.

### 2.1 Extracting MWAA Metadata with the PythonOperator

As the ingestion process will be happening locally in MWAA, we can prepare a DAG with the following YAML
configuration:

```yaml theme={null}
source:
  type: airflow
  serviceName: airflow_mwaa
  serviceConnection:
    config:
      type: Airflow
      hostPort: http://localhost:8080
      numberOfStatus: 10
      connection:
        type: Backend
  sourceConfig:
    config:
      type: PipelineMetadata
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: INFO
  openMetadataServerConfig:
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>
```

### 2.2 Extracting MWAA Metadata with the ECS Operator

After setting up the ECS Cluster, you'll need first to check the MWAA database connection

#### Getting MWAA Database Connection

To extract MWAA information we will need to take a couple of points in consideration:

1. How to get the underlying database connection info, and
2. How to make sure we can reach such database.

The happy path would be going to the `Airflow UI > Admin > Configurations` and finding the `sql_alchemy_conn` parameter.

However, MWAA is not providing this information. Instead, we need to create a DAG to get the connection details
once. The DAG can be deleted afterwards. We want to use a Python Operator that will retrieve the Airflow's Session data:

```python theme={null}
import logging
import os
from datetime import timedelta

import yaml
from airflow import DAG

try:
    from airflow.operators.python import PythonOperator
except ModuleNotFoundError:
    from airflow.operators.python_operator import PythonOperator

from airflow.utils.dates import days_ago

from airflow.configuration import conf

default_args = {
    "owner": "user_name",
    "email": ["username@org.com"],
    "email_on_failure": False,
    "retries": 3,
    "retry_delay": timedelta(minutes=5),
    "execution_timeout": timedelta(minutes=60),
}

def get_data():
    from airflow.settings import Session

    logging.info("SQL ALCHEMY CONN")
    sqlalchemy_conn = conf.get("core", "sql_alchemy_conn", fallback=None)
    logging.info(sqlalchemy_conn)


with DAG(
    "airflow_database_connection",
    default_args=default_args,
    description="An example DAG which pushes Airflow data to OM",
    start_date=days_ago(1),
    is_paused_upon_creation=True,
    schedule_interval="@once",
    catchup=False,
) as dag:
    ingest_task = PythonOperator(
        task_id="db_connection",
        python_callable=get_data,
    )
```

After running the DAG, we can store the connection details and remove the dag file from S3.

Note that trying to log the `conf.get("core", "sql_alchemy_conn", fallback=None)` details might either result in:

1. An empty string, depending on the Airflow version: If that's the case, you can use update the line to be `conf.get("database", "sql_alchemy_conn", fallback=None)`.
2. The password masked in `****`. If that's the case, you can use `sqlalchemy_conn = list(conf.get("core", "sql_alchemy_conn", fallback=None))`,
   which will return the results separated by commas.

#### Preparing the metadata extraction

Then, prepare the YAML config with the information you retrieved above. For example:

```yaml theme={null}
source:
  type: airflow
  serviceName: airflow_mwaa_ecs_op
  serviceConnection:
    config:
      type: Airflow
      hostPort: http://localhost:8080
      numberOfStatus: 10
      connection:
        type: Postgres
        username: adminuser
        password: ...
        hostPort: vpce-075b93a08ef7a8ffc-c88g16we.vpce-svc-0dfc8a341f816c134.us-east-2.vpce.amazonaws.com:5432
        database: AirflowMetadata
  sourceConfig:
    config:
      type: PipelineMetadata
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  openMetadataServerConfig:
    enableVersionValidation: false
    hostPort: https://sandbox.open-metadata.org/api
    authProvider: openmetadata
    securityConfig:
      jwtToken: ...
```

### 2.3 Extracting MWAA Metadata with the Python Virtualenv Operator

This will be similar as the first step, where you just need the simple `Backend` connection YAML:

```yaml theme={null}
source:
  type: airflow
  serviceName: airflow_mwaa
  serviceConnection:
    config:
      type: Airflow
      hostPort: http://localhost:8080
      numberOfStatus: 10
      connection:
        type: Backend
  sourceConfig:
    config:
      type: PipelineMetadata
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: INFO
  openMetadataServerConfig:
    hostPort: <OpenMetadata host and port>
    authProvider: <OpenMetadata auth provider>
```