[MINOR] feat(spark-client): Support set accessId by another config dynamically #2250
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2250 +/- ##
============================================
+ Coverage 52.65% 52.88% +0.23%
- Complexity 3376 3552 +176
============================================
Files 514 532 +18
Lines 27565 29315 +1750
Branches 2582 2724 +142
============================================
+ Hits 14513 15502 +989
- Misses 12092 12824 +732
- Partials 960 989 +29 ☔ View full report in Codecov by Sentry. |
jerqi
reviewed
Nov 18, 2024
| @@ -322,6 +322,9 @@ public class RssSparkConfig { | |||
|
|
|||
| public static final ConfigEntry<String> RSS_ACCESS_ID = | |||
| createStringBuilder(new ConfigBuilder("spark.rss.access.id")).createWithDefault(""); | |||
| public static final ConfigEntry<String> RSS_ACCESS_ID_PROVIDER_KEY = | |||
There was a problem hiding this comment.
Done.
I renamed to spark.rss.access.id.providerKey.
|
Could you add document, too? |
|
@jerqi Thanks for remind me, document added, PTAL. |
jerqi
approved these changes
Nov 18, 2024
|
@jerqi Thanks for your review, merge it. |
maobaolong
added a commit
to maobaolong/incubator-uniffle
that referenced
this pull request
Nov 21, 2024
…namically (apache#2250) ### What changes were proposed in this pull request? Support set accessId by another config dynamically. ### Why are the changes needed? Without this PR, we have to set accessId for each user belongs to `spark.yarn.queue`, after patch this feature, the `spark.access.id` set to the value of `spark.yarn.queue`, if we set `spark.rss.access.id.provider.key=spark.yarn.queue` ### Does this PR introduce _any_ user-facing change? Introduce spark new config `spark.rss.access.id.provider.key`. ### How was this patch tested? Updated UT. (cherry picked from commit b00b67c)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Support set accessId by another config dynamically.
Why are the changes needed?
Without this PR, we have to set accessId for each user belongs to
spark.yarn.queue, after patch this feature, thespark.access.idset to the value ofspark.yarn.queue, if we setspark.rss.access.id.provider.key=spark.yarn.queueDoes this PR introduce any user-facing change?
Introduce spark new config
spark.rss.access.id.provider.key.How was this patch tested?
Updated UT.