HDFS-17642. Add target node list, exclude source node list, and exclu… #7161

Jtdellaringa · 2024-11-14T21:22:15Z

…de target node list parameters to balancer (#7127)

HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (#7127)

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…de target node list parameters to balancer (apache#7127) HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (apache#7127) --------- Co-authored-by: Joseph DellAringa <[email protected]>

hadoop-yetus · 2024-11-15T01:48:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	34m 32s		branch-3.3 passed
+1 💚	compile	0m 50s		branch-3.3 passed
+1 💚	checkstyle	0m 32s		branch-3.3 passed
+1 💚	mvnsite	0m 55s		branch-3.3 passed
+1 💚	javadoc	1m 10s		branch-3.3 passed
+1 💚	spotbugs	1m 50s		branch-3.3 passed
+1 💚	shadedclient	22m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 48s		the patch passed
+1 💚	compile	0m 44s		the patch passed
+1 💚	javac	0m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 24s		the patch passed
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed
+1 💚	spotbugs	1m 50s		the patch passed
+1 💚	shadedclient	22m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	171m 38s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		265m 23s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
	hadoop.hdfs.server.datanode.TestLargeBlockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/1/artifact/out/Dockerfile
GITHUB PR	#7161
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ae7d204ca279 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `8493699`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/1/testReport/
Max. process+thread count	4768 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-11-18T19:13:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	14m 14s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 37s		branch-3.3 passed
+1 💚	compile	11m 14s		branch-3.3 passed
+1 💚	checkstyle	1m 33s		branch-3.3 passed
+1 💚	mvnsite	3m 14s		branch-3.3 passed
+1 💚	javadoc	2m 47s		branch-3.3 passed
+0 🆗	spotbugs	0m 34s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	22m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 10s	/patch-mvninstall-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	mvninstall	0m 19s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	mvninstall	0m 9s	/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch failed.
-1 ❌	compile	0m 27s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	javac	0m 27s	/patch-compile-root.txt	root in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 14s	/buildtool-patch-checkstyle-root.txt	The patch fails to run checkstyle in root
-1 ❌	mvnsite	0m 10s	/patch-mvnsite-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	mvnsite	0m 21s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	mvnsite	0m 11s	/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch failed.
-1 ❌	javadoc	0m 11s	/patch-javadoc-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	javadoc	0m 18s	/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	javadoc	0m 10s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch failed.
+0 🆗	spotbugs	0m 12s		hadoop-project has no data from spotbugs
-1 ❌	spotbugs	0m 11s	/patch-spotbugs-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	spotbugs	0m 19s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	spotbugs	0m 10s	/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch failed.
-1 ❌	shadedclient	0m 51s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 10s		hadoop-project in the patch passed.
-1 ❌	unit	0m 11s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	unit	0m 19s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	unit	0m 11s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch failed.
+1 💚	asflicense	0m 19s		The patch does not generate ASF License warnings.
		91m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/2/artifact/out/Dockerfile
GITHUB PR	#7161
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 1dd41a9d9b1f 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `cb7d9a3`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/2/testReport/
Max. process+thread count	551 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-11-18T23:54:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	34m 35s		branch-3.3 passed
+1 💚	compile	0m 50s		branch-3.3 passed
+1 💚	checkstyle	0m 33s		branch-3.3 passed
+1 💚	mvnsite	0m 56s		branch-3.3 passed
+1 💚	javadoc	1m 9s		branch-3.3 passed
+1 💚	spotbugs	1m 56s		branch-3.3 passed
+1 💚	shadedclient	22m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s		the patch passed
+1 💚	compile	0m 43s		the patch passed
+1 💚	javac	0m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 24s		the patch passed
+1 💚	mvnsite	0m 47s		the patch passed
+1 💚	javadoc	0m 59s		the patch passed
+1 💚	spotbugs	1m 53s		the patch passed
+1 💚	shadedclient	22m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	185m 11s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+0 🆗	asflicense	0m 29s		ASF License check generated no output?
		274m 55s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
	hadoop.hdfs.server.mover.TestStorageMover
	hadoop.hdfs.server.mover.TestMover
	hadoop.hdfs.server.blockmanagement.TestBlockManager
	hadoop.hdfs.TestDFSOutputStream
	hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.blockmanagement.TestReplicationPolicyExcludeSlowNodes
	hadoop.hdfs.server.namenode.TestReencryption
	hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
	hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
	hadoop.hdfs.server.blockmanagement.TestCorruptionWithFailover
	hadoop.hdfs.server.blockmanagement.TestBlockReportLease
	hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
	hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting
	hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId
	hadoop.hdfs.server.datanode.TestLargeBlockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/3/artifact/out/Dockerfile
GITHUB PR	#7161
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 46fd44746280 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `8493699`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/3/testReport/
Max. process+thread count	4796 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

dineshchitlangia

@Jtdellaringa +1 pending CI. It seems some of the tests are timing out. Pls try to kick off CI with an empty commit.

hadoop-yetus · 2024-11-22T22:14:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	33m 23s		branch-3.3 passed
+1 💚	compile	0m 50s		branch-3.3 passed
+1 💚	checkstyle	0m 34s		branch-3.3 passed
+1 💚	mvnsite	0m 55s		branch-3.3 passed
+1 💚	javadoc	1m 7s		branch-3.3 passed
+1 💚	spotbugs	1m 58s		branch-3.3 passed
+1 💚	shadedclient	22m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 47s		the patch passed
+1 💚	compile	0m 46s		the patch passed
+1 💚	javac	0m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 25s		the patch passed
+1 💚	mvnsite	0m 49s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed
+1 💚	spotbugs	1m 53s		the patch passed
+1 💚	shadedclient	22m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	172m 37s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		265m 2s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.datanode.TestLargeBlockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/4/artifact/out/Dockerfile
GITHUB PR	#7161
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1f936ef87865 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `502cb1e`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/4/testReport/
Max. process+thread count	4370 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7161/4/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Jtdellaringa · 2024-11-22T22:28:32Z

@dineshchitlangia I created this PR which just adds a log line and we see the same tests failing #7172

There will likely need to be some other change to resolve the tests failing with: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;

Jtdellaringa force-pushed the HDFS-17642-branch-3.3 branch from cb7d9a3 to 8493699 Compare November 18, 2024 19:19

dineshchitlangia self-requested a review November 22, 2024 17:44

dineshchitlangia reviewed Nov 22, 2024

View reviewed changes

empty commit to trigger ci

502cb1e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17642. Add target node list, exclude source node list, and exclu… #7161

HDFS-17642. Add target node list, exclude source node list, and exclu… #7161

Jtdellaringa commented Nov 14, 2024

hadoop-yetus commented Nov 15, 2024

hadoop-yetus commented Nov 18, 2024

hadoop-yetus commented Nov 18, 2024

dineshchitlangia left a comment

hadoop-yetus commented Nov 22, 2024

Jtdellaringa commented Nov 22, 2024

HDFS-17642. Add target node list, exclude source node list, and exclu… #7161

Are you sure you want to change the base?

HDFS-17642. Add target node list, exclude source node list, and exclu… #7161

Conversation

Jtdellaringa commented Nov 14, 2024

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Nov 15, 2024

hadoop-yetus commented Nov 18, 2024

hadoop-yetus commented Nov 18, 2024

dineshchitlangia left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Nov 22, 2024

Jtdellaringa commented Nov 22, 2024