HDDS-11780. Increase client write retry when SCM is in safe mode

[ArafatKhan2198]

What changes were proposed in this pull request?

Root Cause:

• DataNode Registration Delays: Each DataNode requires approximately 30 seconds to register with the leader SCM due to the heartbeat interval.
• SCM Restart and Leadership Retention: In scenarios where the restarted SCM retains leadership after an election, the SCM loses its in-memory state. DataNodes and pipelines must re-register, leading to delays in exiting safe mode.
• Dependency on HealthyPipelineSafeModeRule: This rule requires DataNodes to report pipeline health, which can be delayed due to slow DataNode registration, network latency, or the time needed for pipelines to stabilize.
• These factors combined caused the SCM to take slightly over a minute to exit safe mode, impacting write operations during this transition.

Current Mechanism:

• The handleSubmitRequestAndSCMSafeModeRetry method manages write requests (e.g., block allocation or key creation) during SCM safe mode by:
• Catching the "SCM in safe mode" exception.
• Retrying the operation after a defined wait interval.
• Allowing limited retries to wait for the SCM to exit safe mode.

Proposed Change:

• Config Update: Increase BLOCK_ALLOCATION_RETRY_WAIT_TIME_MS to 1000 ms and BLOCK_ALLOCATION_RETRY_COUNT to 90.
• This extends the total wait time for retries from ~25–30 seconds to ~90 seconds.
• Impact: This ensures that write operations are not prematurely failed during scenarios where SCM takes longer to exit safe mode, improving client resilience.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11780

How was this patch tested?

[sumitagrawl]

LGTM