Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Airflow override lineage option is not working properly #18678

Open
mramirsha opened this issue Nov 18, 2024 · 1 comment · May be fixed by #18757
Open

Airflow override lineage option is not working properly #18678

mramirsha opened this issue Nov 18, 2024 · 1 comment · May be fixed by #18757

Comments

@mramirsha
Copy link

mramirsha commented Nov 18, 2024 •
edited
Loading

Affected module
It impacts the Ingestion Framework?

Describe the bug
When we are enabling the override lineage option for airflow ingestion pipeline we are facing the following error:
Airflow ingestion pipelines log:

Failed to ingest override_lineage=True lineage_request=AddLineageRequest(edge=EntitiesEdge(fromEntity=EntityReference(id=Uuid(root=UUID('fdcc19e4-cebc-4c31-a937-8e3c3bed3fd2')), type='table', name=None, fullyQualifiedName=None, description=None, displayName=None, deleted=None, inherited=None, href=None), toEntity=EntityReference(id=Uuid(root=UUID('b3d86767-7aef-4753-b819-4d3d39f188fa')), type='table', name=None, fullyQualifiedName=None, description=None, displayName=None, deleted=None, inherited=None, href=None), description=None, lineageDetails=LineageDetails(sqlQuery=None, columnsLineage=None, pipeline=EntityReference(id=Uuid(root=UUID('9b900446-5c4f-4199-9da7-3e310bd5d327')), type='pipeline', name=None, fullyQualifiedName=None, description=None, displayName=None, deleted=None, inherited=None, href=None), description=None, source=<Source.PipelineLineage: 'PipelineLineage'>))) due to api request failure: 500 Server Error: Internal Server Error for url: https://openmetadata.sophie.free-now.com/api/v1/lineage/pipeline/9b900446-5c4f-4199-9da7-3e310bd5d327/type/PipelineLineage

OpenMetadata server log:

ERROR [2024-11-18 12:58:33,969] [dw-334 - DELETE /api/v1/lineage/pipeline/ee120d9c-d9ed-4907-b45b-62e14db1c2c9/type/PipelineLineage] o.o.s.r.l.LineageResource - Got exception: [UnableToExecuteStatementException] / message [org.postgresql.util.PSQLException: ERROR: syntax error at or near "ORDER"

  | Nov 18, 2024 @ 13:58:33.970 | Position: 182 [statement:"/* EntityRelationshipDAO.deleteLineageBySourcePipeline */ DELETE FROM entity_relationship WHERE  json->'pipeline'->>'id' =:toId OR toId = :toId AND relation = :relation AND json->>'source' = :source ORDER BY toId", arguments:{positional:{1:pipeline,2:PipelineLineage,3:13}, named:{toId:ee120d9c-d9ed-4907-b45b-62e14db1c2c9,toEntity:pipeline,source:PipelineLineage,relation:13}, finder:[]}]

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.LazyHandleSupplier.invokeInContext(LazyHandleSupplier.java:75)

  | Nov 18, 2024 @ 13:58:33.970 | at jdk.proxy2/jdk.proxy2.$Proxy175.deleteLineageBySourcePipeline(Unknown Source)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.Server.handle(Server.java:516)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)

  | Nov 18, 2024 @ 13:58:33.970 | at jdk.internal.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)

  | Nov 18, 2024 @ 13:58:33.970 | ... 94 common frames omitted

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.Update.execute(Update.java:57)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.statement.internal.SqlUpdateHandler.invoke(SqlUpdateHandler.java:32)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.Jdbi.callWithExtension(Jdbi.java:502)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlets.HeaderFilter.doFilter(HeaderFilter.java:117)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)

  | Nov 18, 2024 @ 13:58:33.970 | at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:318)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)

  | Nov 18, 2024 @ 13:58:33.970 | at jdk.proxy2/jdk.proxy2.$Proxy170.execute(Unknown Source)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.Update.execute(Update.java:69)

  | Nov 18, 2024 @ 13:58:33.970 | at org.openmetadata.service.jdbi3.LineageRepository.deleteLineageBySource(LineageRepository.java:399)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.SqlObjectFactory.lambda$attach$2(SqlObjectFactory.java:108)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$invoke$5(OnDemandExtensions.java:99)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

  | Nov 18, 2024 @ 13:58:33.970 | at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:118)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1789)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.invoke(SqlObjectInitData.java:135)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.OnDemandExtensions.invoke(OnDemandExtensions.java:99)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$2(OnDemandExtensions.java:83)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/java.lang.reflect.Method.invoke(Method.java:569)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.SqlLoggerUtil.wrap(SqlLoggerUtil.java:31)

  | Nov 18, 2024 @ 13:58:33.970 | Position: 182 [statement:"/* EntityRelationshipDAO.deleteLineageBySourcePipeline */ DELETE FROM entity_relationship WHERE  json->'pipeline'->>'id' =:toId OR toId = :toId AND relation = :relation AND json->>'source' = :source ORDER BY toId", arguments:{positional:{1:pipeline,2:PipelineLineage,3:13}, named:{toId:ee120d9c-d9ed-4907-b45b-62e14db1c2c9,toEntity:pipeline,source:PipelineLineage,relation:13}, finder:[]}]] / related resource location: [org.openmetadata.service.resources.lineage.LineageResource.deleteLineageByType](LineageResource.java:475)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.result.ResultProducers.lambda$returningUpdateCount$0(ResultProducers.java:48)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.exceptions.Unchecked.lambda$function$4(Unchecked.java:76)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.Jdbi.withExtension(Jdbi.java:489)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:292)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

  | Nov 18, 2024 @ 13:58:33.970 | Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "ORDER"

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:180)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.statement.internal.SqlUpdateHandler.lambda$configureReturner$2(SqlUpdateHandler.java:91)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.statement.internal.CustomizingStatementHandler.invoke(CustomizingStatementHandler.java:196)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.call(SqlObjectInitData.java:141)

  | Nov 18, 2024 @ 13:58:33.970 | at org.openmetadata.service.resources.lineage.LineageResource.deleteLineageByType(LineageResource.java:475)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/java.lang.Thread.run(Thread.java:840)

  | Nov 18, 2024 @ 13:58:33.970 | Position: 182

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1787)

  | Nov 18, 2024 @ 13:58:33.970 | org.jdbi.v3.core.statement.UnableToExecuteStatementException: org.postgresql.util.PSQLException: ERROR: syntax error at or near "ORDER"

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.lambda$invoke$0(SqlObjectInitData.java:135)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$3(OnDemandExtensions.java:83)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/java.lang.reflect.Method.invoke(Method.java:569)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

To Reproduce

Trying to delete a lineage via the following api

curl -X DELETE -H "Authorization: Bearer token" -H "Accept: application/json" https://localhost/api/v1/lineage/pipeline/cb6fe35c-5cb4-4ff8-9e68-ee50229e3bdf/type/PipelineLineage

Expected behavior
Overriding the airflow lineage data

Version:

  • OpenMetadata backened Database: Postgres
  • OpenMetadata version: [1.5.10]
  • OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[airflow]==1.5.10.0]

Additional context
It seems that OpenMetadata is using a ORDER BY in delete statement which is not valid.
@mramirsha
Copy link
Author

I found the place that causing the issue https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java#L1160.
This query will not work on Postgresql, https://www.postgresql.org/docs/current/sql-delete.html

@mramirsha mramirsha linked a pull request Nov 24, 2024 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixes #18678: Remove ORDER BY from the postgres DELETE statement mramirsha/OpenMetadata
1 participant
@mramirsha