[HADOOP-19341] HDFS Client's direct memory leaks with erasure coding enabled #7168

EungsopYoo · 2024-11-18T05:12:10Z

Description of PR

The direct memory allocated by ElasticByteBufferPool of HDFS client is never released.
We found that we already have the solution, WeakReferencedElasticByteBufferPool.
Just replace ElasticByteBufferPool with WeakReferencedElasticByteBufferPool.

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-11-18T11:25:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 20s		trunk passed
+1 💚	compile	8m 50s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	8m 16s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	2m 8s		trunk passed
+1 💚	mvnsite	3m 12s		trunk passed
+1 💚	javadoc	2m 52s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 54s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	5m 39s		trunk passed
+1 💚	shadedclient	21m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 59s		the patch passed
+1 💚	compile	8m 36s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	8m 36s		the patch passed
+1 💚	compile	8m 20s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	8m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 3s		the patch passed
+1 💚	mvnsite	3m 10s		the patch passed
+1 💚	javadoc	2m 46s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	3m 0s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	6m 8s		the patch passed
+1 💚	shadedclient	21m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 44s		hadoop-common in the patch passed.
+1 💚	unit	2m 8s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	200m 6s		hadoop-hdfs in the patch passed.
+1 💚	unit	2m 20s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		372m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/1/artifact/out/Dockerfile
GITHUB PR	#7168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 814316822d41 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a3c7fe7`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/1/testReport/
Max. process+thread count	4400 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur

How have you tested the patch ?

...p-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ElasticByteBufferPool.java

hadoop-yetus · 2024-11-19T06:54:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 45s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 53s		trunk passed
+1 💚	compile	10m 11s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	8m 53s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	2m 15s		trunk passed
+1 💚	mvnsite	2m 58s		trunk passed
+1 💚	javadoc	2m 32s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 44s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	5m 32s		trunk passed
+1 💚	shadedclient	24m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 33s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	compile	1m 23s	/patch-compile-root-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt	root in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌	javac	1m 23s	/patch-compile-root-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt	root in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌	compile	1m 22s	/patch-compile-root-jdkPrivateBuild-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga.txt	root in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga.
-1 ❌	javac	1m 22s	/patch-compile-root-jdkPrivateBuild-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga.txt	root in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 0s		root: The patch generated 0 new + 4 unchanged - 1 fixed = 4 total (was 5)
-1 ❌	mvnsite	0m 35s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	javadoc	1m 39s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 59s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
-1 ❌	spotbugs	0m 36s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	shadedclient	5m 19s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 2s		hadoop-common in the patch passed.
+1 💚	unit	1m 49s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	0m 34s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	unit	1m 55s		hadoop-azure in the patch passed.
-1 ❌	asflicense	0m 23s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		142m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/2/artifact/out/Dockerfile
GITHUB PR	#7168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d7b69a661eef 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b218980`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/2/testReport/
Max. process+thread count	3155 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-11-19T10:59:48Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 28s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 21s		trunk passed
+1 💚	compile	8m 59s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	8m 8s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	2m 11s		trunk passed
+1 💚	mvnsite	3m 14s		trunk passed
+1 💚	javadoc	2m 51s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 56s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	5m 46s		trunk passed
+1 💚	shadedclient	20m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 57s		the patch passed
+1 💚	compile	8m 36s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	8m 36s		the patch passed
+1 💚	compile	8m 13s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	8m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 3s		root: The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21)
+1 💚	mvnsite	3m 7s		the patch passed
+1 💚	javadoc	2m 40s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 57s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	6m 10s		the patch passed
+1 💚	shadedclient	20m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 10s		hadoop-common in the patch passed.
+1 💚	unit	2m 8s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	203m 32s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	2m 19s		hadoop-azure in the patch passed.
-1 ❌	asflicense	0m 42s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		374m 46s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommission
	hadoop.hdfs.TestDecommissionWithBackoffMonitor
	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/3/artifact/out/Dockerfile
GITHUB PR	#7168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e805bf59a9cd 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9d5bc0a`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/3/testReport/
Max. process+thread count	4396 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

EungsopYoo · 2024-11-19T23:46:56Z

I think the filed tests are not related to this PR, retrigger the checks.

hadoop-yetus · 2024-11-20T06:00:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 25s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 44s		trunk passed
+1 💚	compile	9m 41s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	9m 10s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	2m 19s		trunk passed
+1 💚	mvnsite	2m 56s		trunk passed
+1 💚	javadoc	2m 30s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 38s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	5m 39s		trunk passed
+1 💚	shadedclient	21m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 55s		the patch passed
+1 💚	compile	9m 25s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	9m 25s		the patch passed
+1 💚	compile	9m 12s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	9m 12s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 19s		root: The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21)
+1 💚	mvnsite	2m 52s		the patch passed
+1 💚	javadoc	2m 23s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 41s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	6m 9s		the patch passed
+1 💚	shadedclient	22m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 12s		hadoop-common in the patch passed.
+1 💚	unit	1m 56s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	196m 20s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	2m 18s		hadoop-azure in the patch passed.
-1 ❌	asflicense	0m 47s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		374m 20s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommissionWithBackoffMonitor
	hadoop.hdfs.TestDecommission

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/4/artifact/out/Dockerfile
GITHUB PR	#7168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux a929f86f2c83 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3fec6d2`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/4/testReport/
Max. process+thread count	4366 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur · 2024-11-20T18:27:06Z

...mmon-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestElasticByteBufferPool.java

+
+import java.nio.ByteBuffer;
+
+public class TestElasticByteBufferPool {


You are just testing the WeakReferencedElasticByteBufferPool here as well with just using the abstract base class. I think this is just duplicate and you can just change all the instances of base class in the TestWeakReferencedElasticByteBufferPool

hadoop-yetus · 2024-11-21T08:36:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 23s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 24s		trunk passed
+1 💚	compile	8m 55s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	8m 26s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	2m 4s		trunk passed
+1 💚	mvnsite	3m 14s		trunk passed
+1 💚	javadoc	2m 35s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 55s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	5m 37s		trunk passed
+1 💚	shadedclient	21m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 52s		the patch passed
+1 💚	compile	8m 29s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	8m 29s		the patch passed
+1 💚	compile	8m 18s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	8m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 9s		root: The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21)
+1 💚	mvnsite	3m 9s		the patch passed
+1 💚	javadoc	2m 46s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	3m 0s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	6m 18s		the patch passed
+1 💚	shadedclient	21m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 40s		hadoop-common in the patch passed.
+1 💚	unit	2m 5s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	202m 21s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	2m 15s		hadoop-azure in the patch passed.
-1 ❌	asflicense	0m 45s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		373m 37s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommission

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/5/artifact/out/Dockerfile
GITHUB PR	#7168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 76b61c60a64e 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `943fdc3`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/5/testReport/
Max. process+thread count	4014 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur · 2024-11-21T23:08:10Z

Not sure why yetus is giving the license error now. can you retrigger it and we are good to go.

hadoop-yetus · 2024-11-22T06:35:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 18s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 37s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 35s		trunk passed
+1 💚	compile	9m 2s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	8m 17s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	2m 6s		trunk passed
+1 💚	mvnsite	3m 15s		trunk passed
+1 💚	javadoc	2m 46s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	3m 1s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	5m 41s		trunk passed
+1 💚	shadedclient	21m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 1s		the patch passed
+1 💚	compile	8m 35s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	8m 35s		the patch passed
+1 💚	compile	8m 24s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	8m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 1s		root: The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21)
+1 💚	mvnsite	3m 7s		the patch passed
+1 💚	javadoc	2m 42s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	3m 0s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	6m 9s		the patch passed
+1 💚	shadedclient	21m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 53s		hadoop-common in the patch passed.
+1 💚	unit	2m 7s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	201m 42s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	2m 18s		hadoop-azure in the patch passed.
-1 ❌	asflicense	0m 48s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		374m 1s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommissionWithBackoffMonitor
	hadoop.hdfs.TestDecommission

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/6/artifact/out/Dockerfile
GITHUB PR	#7168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2f4e9f2ef64c 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a046e97`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/6/testReport/
Max. process+thread count	3803 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7168/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

replace ElasticByteBufferPool with WeakReferencedElasticByteBufferPool

a3c7fe7

github-actions bot added HDFS trunk Common TOOLS ABFS labels Nov 18, 2024

mukund-thakur reviewed Nov 18, 2024

View reviewed changes

...p-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ElasticByteBufferPool.java Show resolved Hide resolved

EungsopYoo changed the title ~~[HADOOP-19341] HDFS Client's direct memory leaks with erasure codind enabled~~ [HADOOP-19341] HDFS Client's direct memory leaks with erasure coding enabled Nov 19, 2024

terence-yoo added 2 commits November 19, 2024 13:43

add a test case for ElasticByteBufferPool

2151ea2

remove unused codes

9d5bc0a

EungsopYoo force-pushed the HADOOP-19341 branch from b218980 to 9d5bc0a Compare November 19, 2024 04:44

retrigger checks

3fec6d2

mukund-thakur requested changes Nov 20, 2024

View reviewed changes

reuse existing tests

943fdc3

mukund-thakur approved these changes Nov 21, 2024

View reviewed changes

retrigger

a046e97

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HADOOP-19341] HDFS Client's direct memory leaks with erasure coding enabled #7168

[HADOOP-19341] HDFS Client's direct memory leaks with erasure coding enabled #7168

EungsopYoo commented Nov 18, 2024

hadoop-yetus commented Nov 18, 2024

mukund-thakur left a comment

hadoop-yetus commented Nov 19, 2024

hadoop-yetus commented Nov 19, 2024

EungsopYoo commented Nov 19, 2024

hadoop-yetus commented Nov 20, 2024

mukund-thakur Nov 20, 2024

EungsopYoo Nov 21, 2024

hadoop-yetus commented Nov 21, 2024

mukund-thakur commented Nov 21, 2024

hadoop-yetus commented Nov 22, 2024


		import java.nio.ByteBuffer;

		public class TestElasticByteBufferPool {

[HADOOP-19341] HDFS Client's direct memory leaks with erasure coding enabled #7168

Are you sure you want to change the base?

[HADOOP-19341] HDFS Client's direct memory leaks with erasure coding enabled #7168

Conversation

EungsopYoo commented Nov 18, 2024

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Nov 18, 2024

mukund-thakur left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Nov 19, 2024

hadoop-yetus commented Nov 19, 2024

EungsopYoo commented Nov 19, 2024

hadoop-yetus commented Nov 20, 2024

mukund-thakur Nov 20, 2024

Choose a reason for hiding this comment

EungsopYoo Nov 21, 2024

Choose a reason for hiding this comment

hadoop-yetus commented Nov 21, 2024

mukund-thakur commented Nov 21, 2024

hadoop-yetus commented Nov 22, 2024