Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Function - service bus message sometimes take long time to recieve #2549

Open
petrkasnal opened this issue Oct 4, 2024 · 4 comments
Open

Comments

@petrkasnal
Copy link

petrkasnal commented Oct 4, 2024

Hello,

I have an application where there is one part WebApi (which as you can tell by the name is a webapi app service) and another part Worker (which is an Azure Function). They communicate with each other using Azure Service Bus. The WebApi service calls the Worker - the Started message, and the Worker calls back the WebApi - the Finished message. Everything works quite fine and as it should. The messages don't run extremely often. Mostly once an hour and they don't even run over each other.

However, once a day, typically at 06:00 am, the messages take an extremely long time to send. Roughly 1-2 minutes. Then because of this the state machine also malfunctions ) in my opinion some message expires. Otherwise messages are processed in seconds. For a while I thought it was due to cold start. But the message goes slowly to Worker and to WebApi. I'm getting desperate about this because I can't find any more info. These logs are info from the consumer method. Where there is no other code before the log.

image
image

I don't see any errors in the log. The only thing in the log is the occasional message. Which is a known problem and quite annoying
Result: Cancelled Exception: Grpc.Core.RpcException: Status(StatusCode="Cancelled", Detail="Call canceled by the client.", DebugException="System.OperationCanceledException: The operation was canceled.") ---> System.OperationCanceledException: The operation was canceled. --- End of inner exception stack trace --- at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.CompleteMessageAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken) in D:\a\_work\1\s\extensions\Worker.Extensions.ServiceBus\src\ServiceBusMessageActions.cs:line 78 at Worker.FunctionApp.DefaultServiceBusFunction.AsTopicResultWithCompleteMessage[TConsumer](ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, String topic, String subs, CancellationToken cancellationToken) in D:\a\1\s\Src\Worker.FunctionApp\DefaultServiceBusFunction.cs:line 50 at Worker.FunctionApp.Started.Run(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, CancellationToken cancellationToken) in D:\a\1\s\Src\Worker.FunctionApp\Started.cs:line 24 at Worker.FunctionApp.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\Src\Worker.FunctionApp\obj\Release\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 39 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 91 at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88 Stack: at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.CompleteMessageAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken) in D:\a\_work\1\s\extensions\Worker.Extensions.ServiceBus\src\ServiceBusMessageActions.cs:line 78 at Worker.FunctionApp.DefaultServiceBusFunction.AsTopicResultWithCompleteMessage[TConsumer](ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, String topic, String subs, CancellationToken cancellationToken) in D:\a\1\s\Src\Worker.FunctionApp\DefaultServiceBusFunction.cs:line 50 at Worker.FunctionApp.Started.Run(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, CancellationToken cancellationToken) in D:\a\1\s\Src\Worker.FunctionApp\Started.cs:line 24 at Worker.FunctionApp.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\Src\Worker.FunctionApp\obj\Release\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 39 at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13 at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 91 at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Thank you

@petrkasnal petrkasnal added the bug label Oct 4, 2024
@bhagyshricompany bhagyshricompany self-assigned this Oct 7, 2024
@bhagyshricompany
Copy link

I checked it seems fine working .please check now.

@petrkasnal
Copy link
Author

I checked it seems fine working .please check now.

It's still happening.
image

@bhagyshricompany
Copy link

@kshyju please comment and validate.

@kshyju
Copy link
Member

kshyju commented Nov 15, 2024

I suspect this is something specific to your application. Recommend opening a support ticket so that the team can take a look at your application. Thanks!

@JAdluri JAdluri self-assigned this Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants