Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-19357. ABFS: Optimizations for Retry Handling and Throttling #7216

Merged
merged 5 commits into from
Dec 17, 2024
Merged

HADOOP-19357. ABFS: Optimizations for Retry Handling and Throttling #7216

merged 5 commits into from
Dec 17, 2024
+22 −4

Conversation

manika137
Copy link
Contributor

@manika137 manika137 commented Dec 9, 2024 •
edited
Loading

Description of PR

JIRA: https://issues.apache.org/jira/browse/HADOOP-19357

Given the significant improvements on the service side over the past few years, the current client-side retry mechanism, which enforces 5-6 minutes of sleep time with jitter, is now excessive. Therefore, we are realigning the retry and throttling logic thresholds.
Since the service can now handle higher traffic rates, we aim to enhance driver performance by setting the following configurations as default:

Client-side throttling (CST): Off
Client Backoff - 500ms (reduced from 3sec)
Max Backoff - 25s (reduced from 30sec)
Min Backoff - 500ms (reduced from 3sec)

The test results are as follows:

============================================================
HNS-OAuth

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 645, Failures: 0, Errors: 0, Skipped: 85
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 32

============================================================
HNS-SharedKey

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 648, Failures: 0, Errors: 0, Skipped: 35
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 19

============================================================
NonHNS-SharedKey

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 632, Failures: 0, Errors: 0, Skipped: 275
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 20

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 653, Failures: 0, Errors: 0, Skipped: 87
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 32

@hadoop-yetus

This comment was marked as outdated.

Sign in to view
@manika137 manika137 marked this pull request as draft December 9, 2024 09:24
@hadoop-yetus

This comment was marked as outdated.

Sign in to view
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 18m 30s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 39m 18s trunk passed
+1 💚 compile 0m 39s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 34s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 0m 32s trunk passed
+1 💚 mvnsite 0m 39s trunk passed
+1 💚 javadoc 0m 40s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 33s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 6s trunk passed
+1 💚 shadedclient 37m 4s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 29s the patch passed
+1 💚 compile 0m 31s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 31s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 0m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 19s the patch passed
+1 💚 mvnsite 0m 30s the patch passed
+1 💚 javadoc 0m 28s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 3s the patch passed
+1 💚 shadedclient 37m 17s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 18s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
145m 10s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7216/3/artifact/out/Dockerfile
GITHUB PR #7216
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 1c71910b6cd5 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 469a451
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7216/3/testReport/
Max. process+thread count 528 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7216/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@manika137 manika137 marked this pull request as ready for review December 12, 2024 03:31
anujmodi2021
anujmodi2021 approved these changes Dec 16, 2024
View reviewed changes
Copy link
Contributor

@anujmodi2021 anujmodi2021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@anujmodi2021 anujmodi2021 merged commit fc42da7 into apache:trunk Dec 17, 2024
4 checks passed
@anujmodi2021
Copy link
Contributor

Thanks for the patch @manika137
Merged to trunk.
Can you please create a bakcport PR for this commit against branch-3.4 and run tests to validate changes there as well??

manika137 added a commit to manika137/hadoop that referenced this pull request Dec 18, 2024
@manika137
HADOOP-19357: [ABFS] Optimizations for Retry Handling and Client Side…
bcdf978 
… Throttling (apache#7216)

Default for following configs are changed:

Client-side throttling (CST): Off
Client Backoff - 500ms (reduced from 3sec)
Max Backoff - 25s (reduced from 30sec)
Min Backoff - 500ms (reduced from 3sec)

Contributed by Manika Joshi (@manika137)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anmolanmol1234 anmolanmol1234 anmolanmol1234 left review comments

@anujmodi2021 anujmodi2021 anujmodi2021 approved these changes

Assignees
No one assigned
Labels
ABFS TOOLS trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@manika137 @hadoop-yetus @anujmodi2021 @anmolanmol1234