-
Notifications
You must be signed in to change notification settings - Fork 642
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] grpc SubStreamHandler exception #4804
Comments
when will this problem be solved? |
Since you checked the |
After calling the @9997766 Do you have an idea for this issue? |
NO, reply as follws
At 2024-04-02 14:35:11, "Pil0tXia" ***@***.***> wrote:
If you set the sender to null in the senderOnComplete() method, each subsequent message will require the sender to be recreated in sendSubscription(), increasing the performance overhead.
1、 if dont set sender to null, you will receive a halfclose exception;
2、 and, there is still a endless exception problem;
After calling the close() method, the SubStreamHandler instance will never run again unless the EventMeshGrpcConsumer is restarted, which is also a problem.
3、so this is another problem,i think you will find a method to resolve it;
Do you have a better idea please?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
ok,
it is easy to reproduce this problem;
you can run in debug mode,and put the point on the receive logic, hold it, stop it to run, about 1 minute ,release it ,let it run continuely, it will show the problem;
it will show a endless exception,and print the exception about every 30s;
the code i show in the issue,have already modify the problem;
At 2024-04-01 17:21:06, "Pil0tXia" ***@***.***> wrote:
Since you checked the Yes I am willing to submit a PR! box, we assumed you would submit a PR to fix the problem. I will try to reproduce this issue.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
I readed the code,you solved the problem。 But the main branche, do not have the 4807 code yet。 when will 4807 be merged to the main branch? |
no problem. but i am not so familar with developping with github。you submit a merge request to me ? |
and i find another problem 2024-04-10 17:33:49.374 ERROR 18488 [] [grpc-default-executor-0] org.apache.eventmesh.client.grpc.consumer.SubStreamHandler Received Server side error io.grpc.StatusRuntimeException: INTERNAL: RST_STREAM closed stream. HTTP/2 error code: INTERNAL_ERROR |
You may refer to this guide: https://eventmesh.apache.org/community/contribute/contribute/
I mean that at least two committers are required to merge a PR, and #4807 is waiting for one more committer's approval to be merged into master.
It is better to submit this problem to a new issue, since merging #4807 will close this issue and lose tracking. It seems that this log is reported by SDK, and you have to check Runtime for detailed reason. |
) * Handle exception loop by closeOnError * Lombok optimization * some format optimization * Avoid closing multiple times * Remove redundant set null * Revert "Avoid closing multiple times" This reverts commit 774397f. * Use synchronized latch to keep senderOnComplete called once * Use boolean to prevent latch called by somebody else * Remove the unique callee/caller close() of onCompleted()
* [ISSUE #4812] Set up Admin Endpoints v2 (#4813) * Remove redundant overloaded methods * Simplify write() result param * Add writeJson(); Add PUT; Add JavaDoc * Rename EventHttpHandler to EventMeshHttpHandler * Correct server thread name * Clean up messy & non-hierarchical overloading * No need to set headers manually any more * Set up v1&v2 endpoints * Set up v1&v2 response dto * Introduce fastjson2 * Fix fastjson2 "level too large : 2048" error caused by IPAddress * Correct @ConfigField naming * Return properties format json key * Add format option to query string * Introduce Result * Reduce duplicate builder code * Fix all checkstyle warnings in eventmesh-runtime * Add known dependency * [ISSUE #4814] Migrate from fastjson 1.2.83 to fastjson2 (#4819) * [Enhancement] Migrate from fastjson 1.2.83 to fastjson2 #4814 * fix_dependencies_problem * fix_check * [ISSUE #4551] modify the logic of time-consumption statistics (#4822) * init connector runtime v2 * [ISSUE #4804] Fix SubStreamHandler exception loop by closeOnError (#4807) * Handle exception loop by closeOnError * Lombok optimization * some format optimization * Avoid closing multiple times * Remove redundant set null * Revert "Avoid closing multiple times" This reverts commit 774397f. * Use synchronized latch to keep senderOnComplete called once * Use boolean to prevent latch called by somebody else * Remove the unique callee/caller close() of onCompleted() * [ISSUE #4838] Deprecate unused `eventMesh.connector.plugin.type` etc. properties (#4839) * Remove all references of `eventMesh.connector.plugin.type` * Deprecate `eventMesh.connector.plugin.type` and sort properties * Remove misconfigured & not-used `registerIntervalInMills`, `fetchRegistryAddrIntervalInMills` * Remove 'defibus' related un-used usages * Supplement #4809 for `null != object` * [ISSUE #4832] Downgrade stale bot to v8 to resolve state cache reserving error (#4833) * Revert stale bot to v8 to resolve state cache reserving error * Reduce operations-per-run to default value to ease pressure * Unify yaml to yml * [ISSUE #4820] Bug fix EventHandler not return json (#4821) * bug fix * bug fix * bug fix * update runtime v2 * update connector runtime * update connector runtime * update connector runtime * update connector runtime * update connector runtime --------- Co-authored-by: Pil0tXia <[email protected]> Co-authored-by: Zaki <[email protected]> Co-authored-by: Karson <[email protected]>
* EventMesh function admin (#4851) * own * dependency * finish registry * EventMesh function admin (#4853) * own * dependency * finish registry * init * Eventmesh function admin (#4854) * own * dependency * finish registry * init * 0419 * 0419 * more discovery and move gRPC * fix dependency * EventMesh function connector runtime (#4858) * [ISSUE #4812] Set up Admin Endpoints v2 (#4813) * Remove redundant overloaded methods * Simplify write() result param * Add writeJson(); Add PUT; Add JavaDoc * Rename EventHttpHandler to EventMeshHttpHandler * Correct server thread name * Clean up messy & non-hierarchical overloading * No need to set headers manually any more * Set up v1&v2 endpoints * Set up v1&v2 response dto * Introduce fastjson2 * Fix fastjson2 "level too large : 2048" error caused by IPAddress * Correct @ConfigField naming * Return properties format json key * Add format option to query string * Introduce Result * Reduce duplicate builder code * Fix all checkstyle warnings in eventmesh-runtime * Add known dependency * [ISSUE #4814] Migrate from fastjson 1.2.83 to fastjson2 (#4819) * [Enhancement] Migrate from fastjson 1.2.83 to fastjson2 #4814 * fix_dependencies_problem * fix_check * [ISSUE #4551] modify the logic of time-consumption statistics (#4822) * init connector runtime v2 * [ISSUE #4804] Fix SubStreamHandler exception loop by closeOnError (#4807) * Handle exception loop by closeOnError * Lombok optimization * some format optimization * Avoid closing multiple times * Remove redundant set null * Revert "Avoid closing multiple times" This reverts commit 774397f. * Use synchronized latch to keep senderOnComplete called once * Use boolean to prevent latch called by somebody else * Remove the unique callee/caller close() of onCompleted() * [ISSUE #4838] Deprecate unused `eventMesh.connector.plugin.type` etc. properties (#4839) * Remove all references of `eventMesh.connector.plugin.type` * Deprecate `eventMesh.connector.plugin.type` and sort properties * Remove misconfigured & not-used `registerIntervalInMills`, `fetchRegistryAddrIntervalInMills` * Remove 'defibus' related un-used usages * Supplement #4809 for `null != object` * [ISSUE #4832] Downgrade stale bot to v8 to resolve state cache reserving error (#4833) * Revert stale bot to v8 to resolve state cache reserving error * Reduce operations-per-run to default value to ease pressure * Unify yaml to yml * [ISSUE #4820] Bug fix EventHandler not return json (#4821) * bug fix * bug fix * bug fix * update runtime v2 * update connector runtime * update connector runtime * update connector runtime * update connector runtime * update connector runtime --------- Co-authored-by: Pil0tXia <[email protected]> Co-authored-by: Zaki <[email protected]> Co-authored-by: Karson <[email protected]> * [ISSUE #4931]Add Registry Module for Discovery AdminServer * [ISSUES #4933]Add Admin Module * [ISSUE #4935] Add and Move the Pojo Used By Both Runtime and Admin to Common * [ISSUE #4937]fix gradle dependecy and add runtime v2 * [ISSUES #4939]add canal connector * fix missing apache header * fix missing apache header * fix missing apache header * update gradle dependencies * fix admin server ci check error * fix admin server ci check error * fix ci checkStyle error * fix ci check error --------- Co-authored-by: sodaRyCN <[email protected]> Co-authored-by: Pil0tXia <[email protected]> Co-authored-by: Zaki <[email protected]> Co-authored-by: Karson <[email protected]>
* EventMesh function admin (#4851) * own * dependency * finish registry * EventMesh function admin (#4853) * own * dependency * finish registry * init * Eventmesh function admin (#4854) * own * dependency * finish registry * init * 0419 * 0419 * more discovery and move gRPC * fix dependency * EventMesh function connector runtime (#4858) * [ISSUE #4812] Set up Admin Endpoints v2 (#4813) * Remove redundant overloaded methods * Simplify write() result param * Add writeJson(); Add PUT; Add JavaDoc * Rename EventHttpHandler to EventMeshHttpHandler * Correct server thread name * Clean up messy & non-hierarchical overloading * No need to set headers manually any more * Set up v1&v2 endpoints * Set up v1&v2 response dto * Introduce fastjson2 * Fix fastjson2 "level too large : 2048" error caused by IPAddress * Correct @ConfigField naming * Return properties format json key * Add format option to query string * Introduce Result * Reduce duplicate builder code * Fix all checkstyle warnings in eventmesh-runtime * Add known dependency * [ISSUE #4814] Migrate from fastjson 1.2.83 to fastjson2 (#4819) * [Enhancement] Migrate from fastjson 1.2.83 to fastjson2 #4814 * fix_dependencies_problem * fix_check * [ISSUE #4551] modify the logic of time-consumption statistics (#4822) * init connector runtime v2 * [ISSUE #4804] Fix SubStreamHandler exception loop by closeOnError (#4807) * Handle exception loop by closeOnError * Lombok optimization * some format optimization * Avoid closing multiple times * Remove redundant set null * Revert "Avoid closing multiple times" This reverts commit 774397f. * Use synchronized latch to keep senderOnComplete called once * Use boolean to prevent latch called by somebody else * Remove the unique callee/caller close() of onCompleted() * [ISSUE #4838] Deprecate unused `eventMesh.connector.plugin.type` etc. properties (#4839) * Remove all references of `eventMesh.connector.plugin.type` * Deprecate `eventMesh.connector.plugin.type` and sort properties * Remove misconfigured & not-used `registerIntervalInMills`, `fetchRegistryAddrIntervalInMills` * Remove 'defibus' related un-used usages * Supplement #4809 for `null != object` * [ISSUE #4832] Downgrade stale bot to v8 to resolve state cache reserving error (#4833) * Revert stale bot to v8 to resolve state cache reserving error * Reduce operations-per-run to default value to ease pressure * Unify yaml to yml * [ISSUE #4820] Bug fix EventHandler not return json (#4821) * bug fix * bug fix * bug fix * update runtime v2 * update connector runtime * update connector runtime * update connector runtime * update connector runtime * update connector runtime --------- Co-authored-by: Pil0tXia <[email protected]> Co-authored-by: Zaki <[email protected]> Co-authored-by: Karson <[email protected]> * [ISSUE #4931]Add Registry Module for Discovery AdminServer * [ISSUES #4933]Add Admin Module * [ISSUE #4935] Add and Move the Pojo Used By Both Runtime and Admin to Common * [ISSUE #4937]fix gradle dependecy and add runtime v2 * [ISSUES #4939]add canal connector * fix missing apache header * fix missing apache header * fix missing apache header * update gradle dependencies * fix admin server ci check error * fix admin server ci check error * fix ci checkStyle error * fix ci check error * [ISSUE #4979]Canal Connector supports bidirectional data synchronization * add bash files for admin & runtime-v2 * fix ack offset read & persist * fix checkStyle error * [ISSUE #4979] Canal Connector supports bidirectional data synchronization (#5011) * [ISSUE #4979]Canal Connector supports bidirectional data synchronization * add bash files for admin & runtime-v2 * fix ack offset read & persist * fix checkStyle error * fix http source connector test error --------- Co-authored-by: sodaRyCN <[email protected]> Co-authored-by: Pil0tXia <[email protected]> Co-authored-by: Zaki <[email protected]> Co-authored-by: Karson <[email protected]>
Search before asking
Environment
Linux
EventMesh version
1.10.0
What happened
There is a problem with the class of SubStreamHandler. grpc is two-way communication. I have integrated grpc pub and sub functions in the project.
This code throws an exception, and every 30 seconds,
private void senderOnNext(final CloudEvent subscription) {
try {
synchronized (sender) {
sender.onNext(subscription);
}
} catch (Exception e) {
log.error("StreamObserver Error onNext", e);
}
}
I analyzed the reason, because when the exception is thrown, you should call onCompleted to close the sender, but that is not done here, resulting in
grpc thinks the connection is still there and keeps throwing exceptions in a loop.
I tried the method below,it solved the problem。
Will you change code quickly? I am using the project,but this problem stop me。
@slf4j
public class SubStreamHandler extends Thread implements Serializable {
...
public SubStreamHandler(final ConsumerServiceStub consumerAsyncClient, final EventMeshGrpcClientConfig clientConfig,
final ReceiveMsgHook listener) {
this.consumerAsyncClient = consumerAsyncClient;
this.clientConfig = clientConfig;
this.listener = listener;
}
}
How to reproduce
when exception happen ,this method will cause endless loop。
for example,grpc time out。
private void senderOnNext(final CloudEvent subscription) {
try {
synchronized (sender) {
sender.onNext(subscription);
}
} catch (Exception e) {
log.error("StreamObserver Error onNext", e);
}
}
Debug logs
no
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: