tracker: Airflow 2.x support

[shalberd]

Is your feature request related to a problem? Please describe.
Several issues with Airflow 2.x. The intent of this ticket is to track efforts related to solving the issues one by one

#3109 fix provider package catalog connector code to be Airflow 2 compatible

#2392 generic issue Airflow 2 support request

#3166 generic pipeline editor needs airflow 2 related fixes

#2124 package catalog connector import fixes needed for airflow 2

Describe the solution you'd like
Gradually, enable airflow 2.x support from Elyra components
Specifically, Airflow 2.6.0, which came out in May 2023, or higher.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Starting with generic pipeline editor (python files, scripts) and rendering to DAG.

#3166

[ianonavy]

@shalberd For future contributors to be aware, is backwards compatibility with 1.x a goal given that Airflow 1.x is no longer supported upstream?

[lresende]

@ianonavy I believe we were mentioning that this would be a new runtime, which then supports new operators, etc which I believe is incompatible in nature from Airflow 1.x otherwise we could have just used 1.x.

[shalberd]

I believe we were mentioning that this would be a new runtime

@lresende do you mean you still want to support Airflow 1.x in parallel? I mean, who really is still using Airflow 1.x or the last intermediate 1.x release with backport functionality? It's been ages ago.

Just to confirm, do you propose having parallel logic / processors and classes for airflow 1.x as well as greater than 2.x?

[shalberd]

@ianonavy backward compatibility with Airflow 1.x is no longer the goal for the next major Elyra release 4.x that the solutions to the tracker tickets will be a part of.