Add support for Airflow 2.x as pipeline runtime

[ptitzler]

Is your feature request related to a problem? Please describe.

Support for Airflow 1.10 is now limited, as EOL was reached June 2021.

Describe the solution you'd like
Add support for Apache Airflow 2.0

[jhgoebbert]

Are there any news on Apache Airflow 2.x support?

I read that it "might" work but it is not tested in the Elyra documentation.

[ptitzler]

@lresende

[lresende]

I was able to get Airflow "runtime specific components" working well, but was not able to try the generic components.

NOT A CONTRIBUTION

[jhgoebbert]

I was able to get Airflow "runtime specific components" working well, but was not able to try the generic components.

That are good news. So there is a high chance that Apache Airflow 2.x actually might work out-of-the-box.
Or did you have to patch Elyra for this?

[shalberd]

Yes, it seems the generic Airflow operators are working (Airflow Package Catalog Connector), while the Airflow Provider Package Catalog Connectors are not, especially with providers written for Airflow 2.x. E.g. in its version for Airflow 2.x, the https://pypi.org/project/apache-airflow-providers-amazon/#files is not working, too.

I was interested in using the DataRobot Provider for Airflow 2.x in the Elyra context, but this, too, is not yet possible.

datarobot/airflow-provider-datarobot#8