Airflow setup

This feature is in Beta mode, please reach out if you want to activate it on your instance.

Catalog currently integrates through Marquez and OpenLineage.

Configure Airflow

• If using Airflow 2.3 to 2.6 : add openlineage-airflow to the requirements / packages to install within the Airflow instance

• If using Airflow 2.7 : nothing to install

• Add the following environment variables:

  • OPENLINEAGE_URL : the URL of the HTTP backend

  • OPENLINEAGE_API_KEY : the API key to allow Airflow to write to Marquez

  • OPENLINEAGE_NAMESPACE : the namespace associated to the Airflow instance

Credentials

You must provide Catalog with following elements:

• Marquez API URL and API key (OPENLINEAGE_URL, OPENLINEAGE_API_KEY)

• namespace configured for Airflow (OPENLINEAGE_NAMESPACE)

• namespace for the main warehouse source (set by OpenLineage)

If you do not have your own Marquez server, please reach out to [email protected]