Exchange$ResponseBodySource.read java.lang.IllegalStateException - closed

[happysiddharth]

Describe the bug
A clear and concise description of what the bug is.

Fatal Exception: java.lang.IllegalStateException
closed
okhttp3.internal.connection.Exchange$ResponseBodySource.read (Exchange.kt:289)
okio.internal.-Buffer.commonWriteAll (-Buffer.java:1117)
okio.Buffer.writeAll (Buffer.kt:439)
okio.internal.-RealBufferedSource.commonReadByteArray (-RealBufferedSource.java:107)
okio.RealBufferedSource.readByteArray (RealBufferedSource.kt:75)
okhttp3.internal._ResponseBodyCommonKt.commonBytes (-ResponseBodyCommon.kt:29)
okhttp3.ResponseBody.bytes (ResponseBody.kt:128)
io.opentelemetry.exporter.sender.okhttp.internal.OkHttpHttpSender$1$1.responseBody (OkHttpHttpSender.java:161)
io.opentelemetry.exporter.internal.http.HttpExporter.onResponse (HttpExporter.java:86)
io.opentelemetry.exporter.internal.http.HttpExporter.lambda$export$0 (HttpExporter.java:66)
io.opentelemetry.exporter.sender.okhttp.internal.OkHttpHttpSender$1.onResponse (OkHttpHttpSender.java:144)
com.google.firebase.perf.network.InstrumentOkHttpEnqueueCallback.onResponse (InstrumentOkHttpEnqueueCallback.java:71)
okhttp3.internal.connection.RealCall$AsyncCall.run (RealCall.kt:529)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)

Steps to Reproduce

Initialize the app and configure OpenTelemetry with an OkHttp-based HTTP exporter.
Trigger an HTTP export process (e.g., by sending telemetry data).
Observe the application crash.
What Did You Expect to See?
The application should process the HTTP response body successfully without throwing exceptions or crashing.

What Did You See Instead?

The application crashes with the exception mentioned above.

What Version and What Artifacts Are You Using?

Artifacts:

opentelemetry-api
opentelemetry-sdk
opentelemetry-exporter-http (OkHttp-based HTTP exporter)
Version: 1.43.0
Environment

Compiler: Java
OS (build): Ubuntu 20.04
Runtime: OpenJDK 11
OS (runtime): Ubuntu 20.04

[jkwatson]

I see firebase in the stacktrace that you posted. That seems like it could be relevant.
Can you provide a repository with a minimal example that reproduces this issue?

[happysiddharth]

Thank you for looking into this issue.
We understand the importance of a reproducible example. However, we are facing significant challenges in reproducing this problem in a controlled environment due to the following reasons:

• This crash is occurring sporadically in production and is visible on Firebase Crashlytics. even we have tried on the product environment but not able to reproduce it.
• It happens during app startup, making it difficult to isolate the exact conditions under which it occurs.
• It seems to depend on specific runtime conditions.

[jkwatson]

Can you at least provide your full list of dependencies w/ versions? Otherwise, we really have nothing to go on.

[jack-berg]

I think I might understand what's happening.

The stacktrace shows that the exception originates from reading the body bytes here:

opentelemetry-java/exporters/sender/okhttp/src/main/java/io/opentelemetry/exporter/sender/okhttp/internal/OkHttpHttpSender.java

Lines 160 to 162 in 3a5222f

	if (bodyBytes == null) {
	bodyBytes = body.bytes();
	}

I found an issue in OkHttp showing a similar error, where a maintainer indicates that it probably originates from unsynchronized concurrent access to a byte buffer.

It shouldn't be possible for our exporter code to call responseBody() concurrently, but if we look at the stacktrace we also see the caller originating from com.google.firebase.perf.network.InstrumentOkHttpEnqueueCallback.onResponse (InstrumentOkHttpEnqueueCallback.java:71).

Note the calling class is InstrumentOkHttpEnqueueCallback. I suspect some instrumentation has injected itself into our OkHttp exporter's codepath, and is somehow accessing the response body's bytes at the same time the exporter's code is, causing this exception.

We could catch wrap / catch the exception here, but I suspect this is a problem with the instrumentation.

[happysiddharth]

+--- androidx.core:core-ktx:1.7.0 -> 1.10.1 (*) | | | +--- androidx.appcompat:appcompat:1.6.1 (*) | | | +--- com.google.android.material:material:1.9.0 (*) | | | +--- com.jakewharton.timber:timber:5.0.1 (*) | | | +--- com.groww.skynet:skynet:3.1.1 -> 4.0.1 (*) | | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.7.1 -> 1.8.0 (*) | | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-android:1.7.1 -> 1.8.0 (*) | | | +--- com.squareup.retrofit2:retrofit:2.9.0 -> 2.11.0 (*) | | | +--- io.opentelemetry:opentelemetry-exporter-otlp -> 1.44.1 | | | | +--- io.opentelemetry:opentelemetry-exporter-otlp-common:1.44.1 | | | | | +--- io.opentelemetry:opentelemetry-api-incubator:1.44.1-alpha | | | | | | \--- io.opentelemetry:opentelemetry-api:1.44.1 | | | | | | \--- io.opentelemetry:opentelemetry-context:1.44.1 | | | | | \--- io.opentelemetry:opentelemetry-exporter-common:1.44.1 | | | | | +--- io.opentelemetry:opentelemetry-api:1.44.1 (*) | | | | | \--- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.44.1 | | | | | \--- io.opentelemetry:opentelemetry-sdk:1.44.1 | | | | | +--- io.opentelemetry:opentelemetry-api:1.44.1 (*) | | | | | +--- io.opentelemetry:opentelemetry-sdk-common:1.44.1 | | | | | | \--- io.opentelemetry:opentelemetry-api:1.44.1 (*) | | | | | +--- io.opentelemetry:opentelemetry-sdk-trace:1.44.1 | | | | | | +--- io.opentelemetry:opentelemetry-api-incubator:1.44.1-alpha (*) | | | | | | +--- io.opentelemetry:opentelemetry-api:1.44.1 (*) | | | | | | \--- io.opentelemetry:opentelemetry-sdk-common:1.44.1 (*) | | | | | +--- io.opentelemetry:opentelemetry-sdk-metrics:1.44.1 | | | | | | +--- io.opentelemetry:opentelemetry-api-incubator:1.44.1-alpha (*) | | | | | | +--- io.opentelemetry:opentelemetry-api:1.44.1 (*) | | | | | | \--- io.opentelemetry:opentelemetry-sdk-common:1.44.1 (*) | | | | | \--- io.opentelemetry:opentelemetry-sdk-logs:1.44.1 | | | | | +--- io.opentelemetry:opentelemetry-api-incubator:1.44.1-alpha (*) | | | | | +--- io.opentelemetry:opentelemetry-api:1.44.1 (*) | | | | | \--- io.opentelemetry:opentelemetry-sdk-common:1.44.1 (*) | | | | +--- io.opentelemetry:opentelemetry-exporter-sender-okhttp:1.44.1 | | | | | +--- io.opentelemetry:opentelemetry-exporter-common:1.44.1 (*) | | | | | +--- io.opentelemetry:opentelemetry-sdk-common:1.44.1 (*) | | | | | \--- com.squareup.okhttp3:okhttp:4.12.0 -> 5.0.0-alpha.14 (*) | | | | +--- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.44.1 (*) | | | | +--- io.opentelemetry:opentelemetry-sdk-trace:1.44.1 (*) | | | | +--- io.opentelemetry:opentelemetry-sdk-metrics:1.44.1 (*) | | | | \--- io.opentelemetry:opentelemetry-sdk-logs:1.44.1 (*) | | | +--- io.opentelemetry:opentelemetry-exporter-logging -> 1.44.1 | | | | +--- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.44.1 (*) | | | | \--- io.opentelemetry:opentelemetry-sdk:1.44.1 (*) | | | +--- io.opentelemetry.android:android-agent:0.8.0-alpha | | | | +--- com.google.code.findbugs:jsr305:3.0.2 | | | | +--- com.google.auto.service:auto-service-annotations:1.1.1 | | | | +--- io.opentelemetry.instrumentation:opentelemetry-instrumentation-api -> 2.9.0 | | | | | +--- io.opentelemetry:opentelemetry-api-incubator:1.43.0-alpha -> 1.44.1-alpha (*) | | | | | +--- io.opentelemetry.semconv:opentelemetry-semconv:1.25.0-alpha | | | | | \--- io.opentelemetry:opentelemetry-api:1.43.0 -> 1.44.1 (*) | | | | +--- io.opentelemetry.android:core:0.8.0-alpha | | | | | +--- com.google.code.findbugs:jsr305:3.0.2 | | | | | +--- com.google.auto.service:auto-service-annotations:1.1.1 | | | | | +--- androidx.core:core:1.13.1 -> 1.10.1 (*) | | | | | +--- androidx.navigation:navigation-fragment:2.7.7 | | | | | | +--- androidx.fragment:fragment-ktx:1.6.2 (*) | | | | | | +--- androidx.navigation:navigation-runtime:2.7.7 | | | | | | | +--- androidx.activity:activity-ktx:1.7.1 -> 1.7.2 (*) | | | | | | | +--- androidx.annotation:annotation-experimental:1.1.0 -> 1.3.0 (*) | | | | | | | +--- androidx.collection:collection:1.0.0 -> 1.1.0 (*) | | | | | | | +--- androidx.lifecycle:lifecycle-runtime-ktx:2.6.2 -> 2.8.6 (*) | | | | | | | +--- androidx.lifecycle:lifecycle-viewmodel-ktx:2.6.2 -> 2.8.6 (*) | | | | | | | +--- androidx.navigation:navigation-common:2.7.7 | | | | | | | | +--- androidx.annotation:annotation:1.1.0 -> 1.8.0 (*) | | | | | | | | +--- androidx.collection:collection-ktx:1.1.0 (*) | | | | | | | | +--- androidx.core:core-ktx:1.1.0 -> 1.10.1 (*) | | | | | | | | +--- androidx.lifecycle:lifecycle-common:2.6.2 -> 2.8.6 (*) | | | | | | | | +--- androidx.lifecycle:lifecycle-runtime-ktx:2.6.2 -> 2.8.6 (*) | | | | | | | | +--- androidx.lifecycle:lifecycle-viewmodel-ktx:2.6.2 -> 2.8.6 (*) | | | | | | | | +--- androidx.lifecycle:lifecycle-viewmodel-savedstate:2.6.2 -> 2.8.6 (*) | | | | | | | | +--- androidx.profileinstaller:profileinstaller:1.3.0 -> 1.3.1 (*) | | | | | | | | +--- androidx.savedstate:savedstate-ktx:1.2.1 (*) | | | | | | | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.8.22 -> 2.0.21 (*) | | | | | | | | +--- androidx.navigation:navigation-fragment:2.7.7 (c) | | | | | | | | \--- androidx.navigation:navigation-runtime:2.7.7 (c) | | | | | | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.8.22 -> 2.0.21 (*) | | | | | | | +--- androidx.navigation:navigation-common:2.7.7 (c) | | | | | | | \--- androidx.navigation:navigation-fragment:2.7.7 (c) | | | | | | +--- androidx.slidingpanelayout:slidingpanelayout:1.2.0 (*) | | | | | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.8.22 -> 2.0.21 (*) | | | | | | +--- androidx.navigation:navigation-runtime:2.7.7 (c) | | | | | | \--- androidx.navigation:navigation-common:2.7.7 (c) | | | | | +--- androidx.lifecycle:lifecycle-process:2.8.6 (*) | | | | | +--- io.opentelemetry:opentelemetry-sdk -> 1.44.1 (*) | | | | | +--- io.opentelemetry:opentelemetry-api-incubator -> 1.44.1-alpha (*) | | | | | +--- io.opentelemetry:opentelemetry-exporter-logging -> 1.44.1 (*) | | | | | +--- io.opentelemetry.instrumentation:opentelemetry-instrumentation-api -> 2.9.0 (*) | | | | | +--- io.opentelemetry.semconv:opentelemetry-semconv-incubating:1.25.0-alpha | | | | | | \--- io.opentelemetry.semconv:opentelemetry-semconv:1.25.0-alpha | | | | | +--- io.opentelemetry.contrib:opentelemetry-disk-buffering:1.40.0-alpha | | | | | | +--- io.opentelemetry:opentelemetry-sdk:1.43.0 -> 1.44.1 (*) | | | | | | \--- com.squareup.wire:wire-runtime:5.1.0 | | | | | | \--- com.squareup.wire:wire-runtime-jvm:5.1.0 | | | | | | +--- com.squareup.okio:okio:3.9.0 (*) | | | | | | \--- org.jetbrains.kotlin:kotlin-stdlib:2.0.20 -> 2.0.21 (*) | | | | | +--- io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom:2.9.0 | | | | | | +--- io.opentelemetry:opentelemetry-bom:1.43.0 -> 1.44.1 | | | | | | | +--- io.opentelemetry:opentelemetry-api:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-exporter-logging:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-extension-kotlin:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-sdk:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-exporter-otlp:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-context:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-sdk-common:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-sdk-trace:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-sdk-metrics:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-sdk-logs:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-exporter-otlp-common:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-exporter-sender-okhttp:1.44.1 (c) | | | | | | | +--- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.44.1 (c) | | | | | | | \--- io.opentelemetry:opentelemetry-exporter-common:1.44.1 (c) | | | | | | \--- io.opentelemetry.instrumentation:opentelemetry-instrumentation-api:2.9.0 (c) | | | | | +--- io.opentelemetry:opentelemetry-api -> 1.44.1 (*) | | | | | \--- org.jetbrains.kotlin:kotlin-stdlib:2.0.21 (*)

+--- com.nextbillion.hoist:hoist-android:2.0.13-alpha.2 | | | +--- androidx.core:core-ktx:1.10.1 (*) | | | +--- com.google.code.gson:gson:2.11.0 (*) | | | +--- com.jakewharton.timber:timber:5.0.1 (*) | | | +--- com.squareup.okhttp3:okhttp:4.12.0 -> 5.0.0-alpha.14 (*) | | | +--- com.squareup.okhttp3:okhttp-brotli:4.12.0 -> 5.0.0-alpha.14 (*) | | | +--- com.tencent:mmkv:1.3.1 (*) | | | +--- com.ionspin.kotlin:bignum:0.3.3 | | | | \--- com.ionspin.kotlin:bignum-jvm:0.3.3 | | | | +--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.5.31 -> 2.0.21 (*) | | | | \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.5.31 -> 2.0.21 (*) | | | \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.25 -> 2.0.21 (*)

@jkwatson

[jkwatson]

It looks like this is running on Android, then, not on Ubuntu as you originally posted, @happysiddharth ?

[happysiddharth]

Ah, you’re absolutely right—by mistake, I posted that it was running on Ubuntu. This is actually running on Android OS. Pardon me for the mix-up.
If there’s anything specific I should check or update given the Android environment, please let me know. I really appreciate your understanding and assistance.

[jkwatson]

Ah, you’re absolutely right—by mistake, I posted that it was running on Ubuntu. This is actually running on Android OS. Pardon me for the mix-up. If there’s anything specific I should check or update given the Android environment, please let me know. I really appreciate your understanding and assistance.

did you see @jack-berg's reply? We both suspect that this is firebase interfering with okhttp instrumentation (or vice-versa), and not something related to the SDK itself. @breedx-splk I don't know if you have any Android specific ideas about what might be happening here?

@happysiddharth are you using otel's android instrumentation, or rolling your own?

[happysiddharth]

Yes we are using otel's android instrumentation

[breedx-splk]

I'm cool with us transferring this to the android repo then...

A couple of follow-ups:

1. How easily reproduced is it? Consistently/easily?
2. Do you have disk buffering enabled? If so, can you try it without?
3. Can you share the code that initializes the OpenTelemetryRum instance? Better still, can you provide a small sample that reproduces this?

Thanks.

[jack-berg]

Also see my response here to a similar issue.

To summarize, I don't think the presence of a stacktrace like this in logs alone is enough to indicate something is wrong. As I show in that comment, it could just represent a signal that an export failed, which is a routine part of dealing with unreliable networks like the internet.

[jack-berg]

We believe we found the root cause of this and #6946, and pushed #6969.

Thanks for the report and sorry for the confusion getting to the root cause.