From 6cbdfbf1a71d96a68c744a090017d308d25b8bc8 Mon Sep 17 00:00:00 2001 From: Serge Mengaptche Date: Fri, 23 Jun 2023 05:59:39 -0700 Subject: [PATCH 1/2] FUS-3536 update microservices updates Fusion 5.6.x --- survival_guide/1_concepts.adoc | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/survival_guide/1_concepts.adoc b/survival_guide/1_concepts.adoc index b7a976e..633b1b8 100644 --- a/survival_guide/1_concepts.adoc +++ b/survival_guide/1_concepts.adoc @@ -90,8 +90,6 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |`connector-plugin-` | HTTP/TCP |Deployment |`analytics` or `system` |Yes (CPU or custom metric) | Deployment for each connector plugin type. *Note:* There is a base deployment, `connector-plugin` with 0 replicas. This is used as a deployment template for each connector plugin type. It should not be deleted or scaled. -|`devops-ui` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves static Web assets for the DevOps UI. - |`indexing` |REST/HTTP |Deployment |`search` or `analytics` depending on write-volume |Yes (CPU or custom metric) |Processes indexing requests. |`insights` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves the App Insights UI @@ -100,10 +98,6 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |`job-rest-server` |REST/HTTP |Deployment |`analytics` |Not required; only 1 pod should be sufficient for most clusters |Performs admin tasks for creating and running Spark jobs. -|`jupyter` |HTTP |Deployment |`analytics` |Not required; only 1 pod should be sufficient for most clusters |Jupyter notebook for ad hoc analytics and visualization. - -|`logstash` |HTTP |StatefulSet |`system` |Not required. Minimum of 1 but 2 pods are recommended for HA |Collects logs from the other microservices and either indexes into system_logs or ships them to an external service like Elastic - |`milvus` |REST/HTTP |Deployment |`analytics` or `system`| Not required; only 1 pod should be sufficient for most clusters |Dense Vector Search Engine for ML models active. |`milvus-mysql` |REST/HTTP |Deployment |`analytics` or `system` |Not required; only 1 pod should be sufficient for most clusters |Handles metadata for Milvus service active. @@ -114,10 +108,6 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |`proxy` / `api-gateway` |HTTP |Deployment |`search` |Not required. Minimum of 1 but 2 pods are recommended for HA |Performs authentication, authorization, and traffic routing. -|`pulsar-bookkeeper` |HTTP |StatefulSet | `search` | At least 3 nodes in HA, you need to run 3 or 5 to ensure a quorum | Write Ahead Log (WAL) used for persistent message storage. - -|`pulsar-broker` |HTTP and TCP | Deployment | `search` | At least 3 nodes in HA | Contains REST API for managing administration and dispatcher for handling all message transfers. - |`query` |REST/HTTP |Deployment |`search` |Yes (CPU or custom metric) |Processes query requests. |`rules-ui` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Serves static Web assets for the Rules UI. @@ -134,10 +124,6 @@ The table below lists the Fusion microservices deployed by our Helm chart. Recog |`spark-executor` |n/a |one or more pods launched by the Spark driver for executing job tasks |`analytics` or dedicated Node Pool for Spark jobs |depends on job configuration; controlled by the spark.executor.instances setting |Executes tasks for a Spark job -|`sql-service` |REST/HTTP and JDBC |Deployment |`analytics` |Not required; only 1 pod should be sufficient for most clusters |Performs admin tasks for creating and managing SQL catalog assets. - -Exposes a JDBC endpoint for the SQL service. - |`templating` |Web |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Retrieves and renders Predictive Merchandiser templates. |`tikaserver` |HTTP |Deployment |`system` |Not required; only 1 pod should be sufficient for most clusters |Is responsible for text extraction using Tika Server Parser @@ -146,6 +132,8 @@ Exposes a JDBC endpoint for the SQL service. |`zookeeper` |TCP |StatefulSet |`system` |No, you need to run 1,3, or 5 Zookeeper pods to ensure a quorum; HPA should not be used for scaling ZK |Stores centralized configuration and performs distributed coordination tasks. +|`kafka` |TCP |StatefulSet |`system` |Yes, required, only 1 pod should be sufficient for most clusters|Topic are used to hold incoming data for solr + |=== == Ports used by Fusion @@ -185,6 +173,7 @@ Below you will find the list of ports required to access Fusion services. |`templating` | 5250 |`webapps` | 8780 |`zookeeper` | 2181, 2888, 3888, 2281, 9141 +|`kafka` | 9092, 9093 |=== From 6433cbb3e76a4b6fa865ec8cd8680c2be9ee0395 Mon Sep 17 00:00:00 2001 From: Serge Mengaptche Date: Fri, 23 Jun 2023 13:43:35 -0700 Subject: [PATCH 2/2] FUS-3536 update microservices remove pulsar --- survival_guide/1_concepts.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/survival_guide/1_concepts.adoc b/survival_guide/1_concepts.adoc index 633b1b8..1a2411c 100644 --- a/survival_guide/1_concepts.adoc +++ b/survival_guide/1_concepts.adoc @@ -162,7 +162,6 @@ Below you will find the list of ports required to access Fusion services. |`ml-model-service milvus` | 3000 |`ml-model-service mysql` | 3306 |`pm-ui` | 8080 -|`pulsar` | 8080, 8443, 6650, 6651, 9527 |`query-pipeline` | 8787 |`rest-service` | 9010 |`rpc-service` | 8771, 8772