From 40338435b2dbbe1195e9714b851fbb0f6f7eb343 Mon Sep 17 00:00:00 2001 From: Hleb Kanonik Date: Fri, 11 Oct 2024 14:26:35 +0200 Subject: [PATCH 1/3] Update and rename ScalingUpReportPortalAPIService.md to ScalingReportPortalServices.md --- .../ScalingReportPortalServices.md | 41 +++++++ .../ScalingUpReportPortalAPIService.md | 106 ------------------ 2 files changed, 41 insertions(+), 106 deletions(-) create mode 100644 docs/installation-steps-advanced/ScalingReportPortalServices.md delete mode 100644 docs/installation-steps-advanced/ScalingUpReportPortalAPIService.md diff --git a/docs/installation-steps-advanced/ScalingReportPortalServices.md b/docs/installation-steps-advanced/ScalingReportPortalServices.md new file mode 100644 index 000000000..66bc17993 --- /dev/null +++ b/docs/installation-steps-advanced/ScalingReportPortalServices.md @@ -0,0 +1,41 @@ +--- +sidebar_position: 13 +sidebar_label: Scaling ReportPortal services +--- + +# Scaling ReportPortal services + +ReportPortal supports dynamic scaling of its API service during runtime to efficiently manage varying loads. This guide provides instructions on how to scale the API service up or down and discusses the implications of asynchronous reporting and queue management in RabbitMQ while scaling. + + ReportPortal also supports the scaling of UAT and UI services. However, it's not recommended to scale the Jobs service due to potential conflicts with cleaning cron jobs, which may lead to database locking issues. + +To effectively scale ReportPortal, you need to follow these steps: + +1. **Additional resources**: Increase capacity by deploying more instances or by enhancing the resources (CPU and memory) of existing ones. +2. **Load Balancing**: The Traefik (for Docker) and Ingress Controller (for Kubernetes) are already set up to automatically distribute incoming requests among all active services. +3. **AMQP settings:** Performance improvements can be achieved by increasing the queue count and adjusting the prefetch count per consumer. These adjustments allow for more efficient processing and resolution of messages within the queues. For more detailed information, refer to the article [Asynchronous Reporting](https://reportportal.io/docs/developers-guides/AsynchronousReporting/#exchanges-and-queues-for-reporting). + +## Kubernetes Configuration + +1. **Scaling Services**: To scale your [ReportPortal services in Kubernetes](https://github.com/reportportal/kubernetes), you need to increase the replica count parameter in the `values.yaml` file for the required services. For example, to scale the API service, adjust the `replicaCount` as shown below: + + ```yaml + serviceapi: + replicaCount: 2 + ``` + +2. **Load Balancing**: The Ingress Controller is already set up to automatically distribute incoming requests among all active services. However, to enhance control over idle TCP connections adjust the IDLE Timeout value to `300`. + +## Docker Configuration + +1. **Scaling Services**: To scale your [ReportPortal services in Docker](https://github.com/reportportal/reportportal/blob/master/docker-compose.yml), you need to add a replica parameter in the `docker-compose.yml` file for the required services. For example, to scale the API service, adjust the `replicas` as shown below: + + ```yaml + services: + + api: + deploy: + replicas: 2 + ``` + +2. **Load Balancing**: The Teafik is already set up to automatically distribute incoming requests among all active services. diff --git a/docs/installation-steps-advanced/ScalingUpReportPortalAPIService.md b/docs/installation-steps-advanced/ScalingUpReportPortalAPIService.md deleted file mode 100644 index 4c87f9395..000000000 --- a/docs/installation-steps-advanced/ScalingUpReportPortalAPIService.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -sidebar_position: 13 -sidebar_label: Scaling Up the ReportPortal Service API ---- - -# Scaling Up the ReportPortal Service API - -ReportPortal supports dynamic scaling of its API service during runtime to efficiently manage varying loads. This guide provides instructions on how to scale the API service up or down, and discusses the implications on asynchronous reporting and queue management in RabbitMQ while scaling. - -## Scaling Up the API Service - -### Steps to Scale Up -1. **Launch Additional Instances**: Increase the capacity by starting more instances of the API service. -2. **Load Balancing**: The load balancer will automatically distribute incoming requests among all active API service instances. - -## Scaling Down the API Service - -### Steps to Scale Down -1. **Shutdown Instances**: Decrease the scale by shutting down any of the API service instances. -2. **Message Redistribution**: Messages in the queues of the shutdown instance will automatically shift to the queues of the remaining active APIs. -3. **Queue Cleanup**: Inactive queues (those not receiving any new messages) will be removed after a few minutes. - -## Impact on Asynchronous Reporting and Queue Management - -### Considerations for Scaling Up -- **Message Rebalancing**: During periods of heavy asynchronous reporting, scaling up may cause messages to be rebalanced across different queues, despite using "Consistent Hashing Algorithm" for distribution. It may lead to an increased number of retries. It might take approx. 2 hours to restore order using retry logic with progressively increasing TTL for each message. -- **Avoid During Heavy Reporting**: Given the potential complexities in message handling when scaling up, it is advisable to refrain from doing so during extensive reporting activities to prevent hard-to-resolve situations and missed reporting items. - -### Considerations for Scaling Down -- **Continuity in Message Processing**: Shutting down an API instance leads to its queues redistributing their messages to the remaining queues, ensuring no disruption in processing. - -### Notable Effects -Scaling operations primarily affect asynchronous report processing and management within RabbitMQ queues: - -- **Order Processing Assurance**: To maintain correct order processing of reports for a specific Launch, all requests are directed to one particular queue and handled by only one consumer. - -### About RabbitMQ Queues -- **Scaling Limitations**: Currently, it is not possible to scale queues in RabbitMQ with spreading requests across multiple queues and consumers. - - -## Scaling up configuration for ReportPortal API Service - -### Kubernetes - -To scale your ReportPortal services in Kubernetes, you need to adjust the `replicaCount` and `queues.totalNumber` in your `values.yaml` file. - -1. **Update Replica Count**: - Change `replicaCount` from `1` to `2` for additional replication.
- [values.yaml replicaCount](https://github.com/reportportal/kubernetes/blob/master/reportportal/values.yaml#L91) - -2. **Edit Total Number of Queues**: - Modify `queues` from `10` to `20` to increase the total available queues.
- [values.yaml queues](https://github.com/reportportal/kubernetes/blob/master/reportportal/values.yaml#L159) - -### Docker - -To scale your ReportPortal services using Docker, update the environment variables and duplicate the API values block. - -- **Set Environment Variables**: - Add `RP_AMQP_QUEUES` and `RP_AMQP_QUEUESPERPOD` to your API environment variables.
- [docker-compose.yml environment](https://github.com/reportportal/reportportal/blob/v23.2/docker-compose.yml#L202)
- ```bash - version: '3.8' - services: - - api: - <...> - environment: - REPORTING_QUEUES_COUNT: 10 - <...> - ``` - -#### Docker Compose v2 -- **Duplicate API Values Block**: - Create a copy of the API values block and rename `api` to `api_replica_1` to facilitate scaling.
- [docker-compose.yml API values block](https://github.com/reportportal/reportportal/blob/v23.2/docker-compose.yml#L191-L241)
- ```bash - version: '3.8' - services: - - api: - <...> - environment: - REPORTING_QUEUES_COUNT: 10 - <...> - - api_replica_1: - <...> - environment: - REPORTING_QUEUES_COUNT: 10 - <...> - ``` - -#### Docker Compose v3.3+ -- **Add replicas**: - Add `deploy.replicas: 2` to your API: - ```bash - version: '3.8' - services: - - api: - <...> - deploy: - replicas: 2 - <...> - ``` From 1bae55347a11b48e1f10e1ec38d81f594e7770ab Mon Sep 17 00:00:00 2001 From: Hleb Kanonik Date: Fri, 11 Oct 2024 14:30:38 +0200 Subject: [PATCH 2/3] Update ScalingReportPortalServices.md --- docs/installation-steps-advanced/ScalingReportPortalServices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation-steps-advanced/ScalingReportPortalServices.md b/docs/installation-steps-advanced/ScalingReportPortalServices.md index 66bc17993..668521cc7 100644 --- a/docs/installation-steps-advanced/ScalingReportPortalServices.md +++ b/docs/installation-steps-advanced/ScalingReportPortalServices.md @@ -13,7 +13,7 @@ To effectively scale ReportPortal, you need to follow these steps: 1. **Additional resources**: Increase capacity by deploying more instances or by enhancing the resources (CPU and memory) of existing ones. 2. **Load Balancing**: The Traefik (for Docker) and Ingress Controller (for Kubernetes) are already set up to automatically distribute incoming requests among all active services. -3. **AMQP settings:** Performance improvements can be achieved by increasing the queue count and adjusting the prefetch count per consumer. These adjustments allow for more efficient processing and resolution of messages within the queues. For more detailed information, refer to the article [Asynchronous Reporting](https://reportportal.io/docs/developers-guides/AsynchronousReporting/#exchanges-and-queues-for-reporting). +3. **AMQP settings:** Performance improvements can be achieved by increasing the queue count and adjusting the prefetch count per consumer. These adjustments allow for more efficient processing and resolution of messages within the queues. For more detailed information, refer to the article [Asynchronous Reporting](/developers-guides/AsynchronousReporting/#exchanges-and-queues-for-reporting). ## Kubernetes Configuration From a0386d872af4bc36117783f232af0833fbb11884 Mon Sep 17 00:00:00 2001 From: Ilya Date: Fri, 11 Oct 2024 16:28:12 +0200 Subject: [PATCH 3/3] Update docusaurus.config.js --- docusaurus.config.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docusaurus.config.js b/docusaurus.config.js index bcfebaaf4..a89953512 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -525,7 +525,7 @@ const config = { from: '/installation-steps/ReportPortal23.1FileStorageOptions', }, { - to: '/installation-steps-advanced/ScalingUpReportPortalAPIService', + to: '/installation-steps-advanced/ScalingReportPortalServices', from: '/installation-steps/ScalingUpReportPortalAPIService', }, {