NServiceBusMetadataReport.RunReportAsync exception handling can be improved

[ramonsmits]

Describe the suggested improvement

Is your improvement related to a problem? Please describe.

The reporting task sends messages to servicecontrol. The sending of these messages can fail if there is a connectivity issue with the transport. For example the following error with the Azure Service Bus transport:

System.TimeoutException at ServiceBus.Metrics.ServiceControl.ServiceControlReporting.NServiceBusMetadataReport+<RunReportAsync>d__1.MoveNext

Details:

"Microsoft.Azure.ServiceBus.ServiceBusTimeoutException","message":"The operation did not complete within the allocated time 00:00:59.9953352 for object sender165."

This results in a log entry with log level ERROR but without any information for operators that analyze the logs for problems.

Describe the suggested solution

The log entry should show information about the error and its impact. For example, we could catch exceptions and wrap them in a new exception that provides information or write a log entry with info like

Failed to dispatch metrics message for ServiceControl monitoring to transport

We could also consider to not have the log level set to ERROR but to WARN as metric messages are not critical for operation.

Additional Context

No response

[ramonsmits]

Workaround

An operator can suppress these ERROR log entries by excluding the NServiceBusMetadataReport logger

[ramonsmits]

Alternatively, we could add a circuit breaking. By default log with WARN but if errors are persistent to raise an ERROR log entry or maybe even more drastic to trigger an endpoint critical error

If there are persistent transport issues the circuit breaker of Core will likely already be set which will.

I think lowering the log level to WARN is sufficient and changing the log message:

• from: Error while sending metric data to {destination.Destination}.
• to: Error while sending metric data to {destination.Destination}. Occasional failures can be safely ignored.