Error when upgrading observability stack from Spring Boot version 3.1.3 to 3.2.0

[kcsurapaneni]

Description:

I am encountering an issue when upgrading the observability stack from Spring Boot version 3.1.3 to 3.2.0

Error Message:

2023-11-26T16:00:43.087Z  WARN [order service,,] 1 --- [order service] [BatchSpanProcessor_WorkerThread-1] [] i.o.exporter.zipkin.ZipkinSpanExporter   : Failed to export spans

org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "valid annotation timestamp required<EOL>"
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:103)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137)
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:932)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:881)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:781)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:663)
    at org.springframework.boot.actuate.autoconfigure.tracing.zipkin.ZipkinRestTemplateSender$RestTemplateHttpPostCall.doExecute(ZipkinRestTemplateSender.java:68)
    at org.springframework.boot.actuate.autoconfigure.tracing.zipkin.ZipkinRestTemplateSender$RestTemplateHttpPostCall.doEnqueue(ZipkinRestTemplateSender.java:75)
    at zipkin2.Call$Base.enqueue(Call.java:406)
    at io.opentelemetry.exporter.zipkin.ZipkinSpanExporter.export(ZipkinSpanExporter.java:82)
    at io.micrometer.tracing.otel.bridge.CompositeSpanExporter.lambda$export$3(CompositeSpanExporter.java:88)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at io.micrometer.tracing.otel.bridge.CompositeSpanExporter.export(CompositeSpanExporter.java:88)
    at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.exportCurrentBatch(BatchSpanProcessor.java:327)
    at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run(BatchSpanProcessor.java:245)
    at java.base/java.lang.Thread.run(Thread.java:833)

Environment:

• Spring Boot version: 3.2.0

Dependencies:

<dependencies>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.2.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-tracing-bridge-otel</artifactId>
    </dependency>

    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-zipkin</artifactId>
    </dependency>
</dependencies>

Observations:

• The issue is specific to Spring Boot version 3.2.0
• When using Brave dependencies with version 3.2.0, the application works without errors.

Sample Code:

• Repository: food-delivery-backend
• Affected Module: order (using version 3.2.0 with opentelemetry-java) - Link to Order API
• Affected Module: restaurant (using version 3.2.0 with Brave dependencies) - Link to Restaurant API

Additional Notes:

• The error is not present when using Spring Boot version 3.1.3 for both opentelemetry-java and brave dependencies - food-delivery-backend.
• The issue seems to be related to the interaction between Spring Boot 3.2.0 with opentelemetry-java.

[jkwatson]

Are you running with the otel javaagent, or some other way of incorporating otel into your application?

[trask]

hi @kcsurapaneni! it looks like you are using micrometer-tracing-bridge-otel which is a Spring project. you could try asking there or bypassing the micrometer tracing bridge and using OpenTelemetry directly: https://opentelemetry.io/docs/instrumentation/java/automatic/spring-boot/

[kcsurapaneni]

@jkwatson I'm using Spring Boot 3.x's built-in observability rather than using the java agent for instrumentation.

[kcsurapaneni]

@trask I've opened this bug based on a suggestion from a Stack Overflow user. You can find the related Stack Overflow question here.

I've been experimenting with different combinations of Micrometer Tracing and OTel Exporter Zipkin versions to identify the root cause of the error, but the issue persists. I'm unsure which specific dependency is triggering this exception.

Any insights or assistance in resolving this issue would be greatly appreciated. If you have any recommendations or ideas, feel free to share them here.

[trask]

I'm using Spring Boot 3.x's built-in observability rather than using the java agent for instrumentation.

the OpenTelemetry project also publishes a Spring Boot starter (which is separate from the Java agent): https://opentelemetry.io/docs/instrumentation/java/automatic/spring-boot/

[jkwatson]

@jkwatson I'm using Spring Boot 3.x's built-in observability rather than using the java agent for instrumentation.

I strongly recommend that you open a bug with the spring team, then.

[jonatan-ivanov]

Thanks to the collaboration on this, a bug was identified in Micrometer Tracing, fixed then released (1.2.1). Please upgrade your Micrometer Tracing version to use at least 1.2.1 (you can also wait till the next Boot release (3.2.1) using Brave since it should pick up the fixed version).

I think this issue should be closed.

[jkwatson]

closing this issue, as it is fixed in the micrometer tracing.