diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 94b8a0f..9a103a1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,21 +1,115 @@ -## How to contribute to a Solace Project +# How to contribute to a Solace Project -#### **Did you find a bug?** +We'd love for you to contribute and welcome your help. Here are some guidelines to follow: -* **Ensure the bug was not already reported** by searching on GitHub under [Issues](https://github.com/SolaceLabs/solace-gcp-quickstart/issues). +- [Issues and Bugs](#issue) +- [Submitting a fix](#submitting) +- [Feature Requests](#features) +- [Questions](#questions) -* If you're unable to find an open issue addressing the problem, [open a new one](https://github.com/SolaceLabs/solace-gcp-quickstart/new). Be sure to include a **title and clear description**, as much relevant information as possible, and a **code sample** or an **executable test case** demonstrating the expected behavior that is not occurring. +## Did you find a issue? -#### **Did you write a patch that fixes a bug?** +- **Ensure the bug was not already reported** by searching on GitHub under [Issues](https://github.com/SolaceProducts/solace-kubernetes-quickstart/issues). -* Open a new GitHub pull request with the patch. +- If you're unable to find an open issue addressing the problem, [open a new one](https://github.com/SolaceProducts/solace-kubernetes-quickstart/issues). Be sure to include a **title and clear description**, as much relevant information as possible, and a **code sample** or an **executable test case** demonstrating the expected behavior that is not occurring. -* Ensure the PR description clearly describes the problem and solution. Include the relevant issue number if applicable. +## Did you write a patch that fixes a bug? -#### **Do you intend to add a new feature or change an existing one?** +Open a new GitHub pull request with the patch following the steps outlined below. Ensure the PR description clearly describes the problem and solution. Include the relevant issue number if applicable. -* Open a GitHub [enhancement request issue](https://github.com/SolaceLabs/solace-gcp-quickstart/issues/new) and describe the new functionality. +Before you submit your pull request consider the following guidelines: -#### **Do you have questions about the source code?** +- Search [GitHub](/https://github.com/SolaceProducts/solace-kubernetes-quickstart/pulls) for an open or closed Pull Request + that relates to your submission. You don't want to duplicate effort. -* Ask any question about the code or how to use Solace messaging in the [Solace community](http://dev.solace.com/community/). +### Submitting a Pull Request + +Please follow these steps for all pull requests. These steps are derived from the [GitHub flow](https://help.github.com/articles/github-flow/). + +#### Step 1: Fork + +Fork the project and clone your fork +locally. + +```sh +prompt:~$ git clone https://github.com/SolaceProducts/solace-kubernetes-quickstart +``` + +#### Step 2: Branch + +Make your changes on a new git branch in your fork of the repository. + +```sh +prompt:~$ git checkout -b my-fix-branch master +``` + +#### Step 3: Commit + +Commit your changes using a descriptive commit message. + +```sh +prompt:~$ git commit -a -m "Your Commit Message" +``` + +Note: the optional commit `-a` command line option will automatically "add" and "rm" edited files. + +#### Step 4a: Rebase (if you have not yet pushed your branch to origin, else goto step 4b.) + +Assuming you have not yet pushed your branch to origin, use `git rebase` (not `git merge`) to synchronize your work with the main +repository. + +If you have not set the upstream, do so as follows: + +```sh +prompt:~$ git remote add upstream https://github.com/SolaceProducts/solace-kubernetes-quickstart +``` + +then: + +```sh +prompt:~$ git fetch upstream +prompt:~$ git rebase upstream/master +``` + +#### Step 4b: Merge (if you have already pushed our branch to origin) + +Assuming you have already pushed your branch to origin, use `git merge` (not `git rebase`) to synchronize your work with the main +repository. + +First ensure there are not any changes to master that you need to pick up, then merge in your changes. +You may need to resolve any conflicts on either of the merge steps. + +```sh +prompt:~$ git merge master +prompt:~$ git checkout master +prompt:~$ git merge my-fix-branch +``` + + +#### Step 5: Push + +Push your branch to your fork in GitHub: + +```sh +prompt:~$ git push origin my-fix-branch +``` + +#### Step 6: Pull Request + +In GitHub, send a pull request to `solace-samples-semp:master`. + +When fixing an existing issue, use the [commit message keywords](https://help.github.com/articles/closing-issues-via-commit-messages/) to close the associated GitHub issue. + +- If we suggest changes then: + - Make the required updates. + - Commit these changes to your branch (ex: my-fix-branch) + +That's it! Thank you for your contribution! + +## **Do you have an ideas for a new feature or a change to an existing one?** + +- Open a GitHub [enhancement request issue](https://github.com/SolaceProducts/solace-kubernetes-quickstart/issues) and describe the new functionality. + +## Do you have questions about the source code? + +- Ask any question about the code or how to use Solace messaging in the [Solace community](http://dev.solace.com/community/). diff --git a/README.md b/README.md index 988ecd8..a564281 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,94 @@ -# Install a Solace Message Router onto a Google Compute Engine Linux Virtual Machine +# Install a Solace PubSub+ Software Message Broker onto Google Compute Engine Linux Virtual Machines -The Solace Virtual Message Router (VMR) provides enterprise-grade messaging capabilities deployable in any computing environment. The VMR provides the same rich feature set as Solace’s proven hardware appliances, with the same open protocol support, APIs and common management. The VMR can be deployed in the datacenter or natively within all popular private and public clouds. +This repository explains how to install a Solace PubSub+ Software Message Broker in various configurations onto Google Compute Engine (GCE) Linux Virtual Machines. This guide is intended for development and demo purposes. -# How to Deploy a VMR +# Description of the Solace PubSub+ Software Message Broker -* [Optional] By default this project installs the Solace PubSub+ software message router, Standard Edition. If you want to install a different version you will have to provide link to version and accompanying md5sum files with the --url and --md5sum flags for solos-install. Visit http://dev.solace.com/downloads/ to see options and read release notes to understand differences. +The Solace PubSub+ software message broker meets the needs of big data, cloud migration, and Internet-of-Things initiatives, and enables microservices and event-driven architecture. Capabilities include topic-based publish/subscribe, request/reply, message queues/queueing, and data streaming for IoT devices and mobile/web apps. The message broker supports open APIs and standard protocols including AMQP, JMS, MQTT, REST, and WebSocket. Moreover, it can be deployed in on-premise datacenters, natively within private and public clouds, and across complex hybrid cloud environments. -* Go to your Google Cloud Platform console and create a Compute Engine instance. Select standard 2 vCPU machine type, and at least 6 GB of memory, a CentOS 7 OS, and a disk with a -size of at least 30 GB depolyed on Centos7 OS: +Solace PubSub+ software message brokers can be deployed in either a 3-node High-Availability (HA) cluster, or as a single-node deployment. For simple test environments that need only to validate application functionality, a single instance will suffice. Note that in production, or any environment where message loss cannot be tolerated, an HA cluster is required. -![alt text](https://raw.githubusercontent.com/SolaceLabs/solace-gcp-quickstart/master/images/gce_launch_1.png "GCE Image creation 1") +# How to deploy a message broker -* Expand the the Management tab to expose the Automation Startup script panel +In this quick start we go through the steps to set up a message broker either as a single stand-alone instance, or in a 3-node HA cluster. -![alt text](https://raw.githubusercontent.com/SolaceLabs/solace-gcp-quickstart/master/images/gce_launch_2.png "GCE Image creation 2") +## Step 1 (Optional): Obtain a reference to the Docker image of the Solace PubSub+ message broker to be deployed -Cut and paste the code into the panel, replace -link to VMR Docker Image- with the URL you received in step one. +First, decide which [Solace PubSub+ message broker](https://docs.solace.com/Solace-SW-Broker-Set-Up/Setting-Up-SW-Brokers.htm ) and version is suitable to your use case. + +**Note:** You can skip the rest of this step if you're using the default settings. By default this project installs the Standard edition of the Solace PubSub+ software message broker from the latest Docker image available from Docker Hub. + +The Docker image reference can be: + +* A public or accessible private Docker registry repository name with an optional tag. This is the recommended option if using PubSub+ Standard. The default is to use the latest message broker image [available from Docker Hub](https://hub.docker.com/r/solace/solace-pubsub-standard/ ) as `solace/solace-pubsub-standard:latest`, or use a specific version [tag](https://hub.docker.com/r/solace/solace-pubsub-standard/tags/ ). + +* A Docker image download URL + * If using Solace PubSub+ Enterprise Evaluation Edition, go to the Solace Downloads page. For the image reference, copy and use the download URL in the Solace PubSub+ Enterprise Evaluation Edition Docker Images section. + + | PubSub+ Enterprise Evaluation Edition
Docker Image + | :---: | + | 90-day trial version of PubSub+ Enterprise | + | [Get URL of Evaluation Docker Image](http://dev.solace.com/downloads#eval ) | + + * If you have purchased a Docker image of Solace PubSub+ Enterprise, Solace will give you information for how to download the compressed tar archive package from a secure Solace server. Contact Solace Support at support@solace.com if you require assistance. You can then host this tar archive together with its MD5 on a file server and use the download URL as the image reference. + +## Step 2: Create the required GCE Compute Engine instances + +The single stand-alone instance requires 1 Compute Engine instance, and the HA deployment requires 3 instances for the Primary, Backup, and Monitor nodes. + +Repeat these instructions for all instances required, and follow the specific requirements for HA setup as applicable. + +### Step 2a: Select instance machine type and parameters + +* Go to your Google Cloud Platform console > Compute Engine > VM instances screen and create a Compute Engine instance by clicking on "Create instance". + +![alt text](/images/gce_createinstance_1.png "GCE Create VM Instance") + +Select standard 2 vCPU machine type, and at least 6 GB of memory, a CentOS 7 OS, and a disk with a size of at least 30 GB deployed on Centos7 OS: + +**Note:** In an HA deployment it's recommended to choose a different availability zone for each node. Also, the Monitor node requires only 1 vCPU and the standard 10 GB of disk space. + +![alt text](/images/gce_launch_1.png "GCE Image creation 1") + +### Step 2b: (HA cluster deployment only) Customise your IP addresses + +* If you are configuring 3 HA nodes, expand the Networking tab to edit the Network interfaces panel and customise your IP addresses. You need to pick 3 available internal IPs. + +> Tip: Gather all 3 IP addresses before continuing by trying availability (there is feedback if the entered address is being used by another resource), and designating each one to one of the Primary, Backup, and Monitor nodes. + +Take note of the configured IP addresses: ``, `` and ``, as they will be used in subsequent steps. + +![alt text](/images/gce_launch_3.png "GCE Image creation 3") + +### Step 2c: Add automated startup script + +* Expand the Management tab to expose the "Automation" "Startup script" panel + +![alt text](/images/gce_launch_2.png "GCE Image creation 2") + +Cut and paste the following code according to your deployment configuration into the panel, replace the value of the variable `SOLACE_DOCKER_IMAGE_REFERENCE` if required to the reference from [Step 1](#step-1-optional-obtain-a-reference-to-the-docker-image-of-the-solace-pubsub-message-broker-to-be-deployed ), and replace `` with the desired password for the management `admin` user. + +**Note:** For an HA deployment, additional environment variables are required (see the script section "Add here environment variables..." near the beginning), which is discussed below. ``` #!/bin/bash -if [ ! -f /var/lib/solace ]; then +################################## +# Update following variables as needed: +SOLACE_DOCKER_IMAGE_REFERENCE="solace/solace-pubsub-standard:latest" # default to pull latest PubSub+ standard from docker hub +ADMIN_PASSWORD="" +GITHUB_BRANCH="SolaceProducts/solace-gcp-quickstart/master" +################################## +# Add here environment variables for HA deployment, not required for single-node deployment. +# export ... see next section HA deployment environment variables +################################## +# +if [ ! -d /var/lib/solace ]; then mkdir /var/lib/solace cd /var/lib/solace yum install -y wget LOOP_COUNT=0 while [ $LOOP_COUNT -lt 3 ]; do - wget https://raw.githubusercontent.com/SolaceLabs/solace-gcp-quickstart/master/solos-install.sh + wget https://raw.githubusercontent.com/$GITHUB_BRANCH/scripts/install-solace.sh if [ 0 != `echo $?` ]; then ((LOOP_COUNT++)) else @@ -36,38 +99,173 @@ if [ ! -f /var/lib/solace ]; then echo "`date` ERROR: Failed to download initial install script exiting" exit 1 fi - chmod +x /var/lib/solace/solos-install.sh - /var/lib/solace/solos-install.sh -p + chmod +x /var/lib/solace/install-solace.sh + /var/lib/solace/install-solace.sh -p $ADMIN_PASSWORD -i $SOLACE_DOCKER_IMAGE_REFERENCE fi ``` -Now hit the "Create" button on the bottom of this page. This will start the process of starting the GCE instance, installing Docker and finally download and install the VMR. It is possible to access the VM before the entire Solace solution is up. You can monitor /var/lib/solace/install.log for the following entry: "'date' INFO: Install is complete" to indicate when the install has completed. +#### HA deployment environment variables for the startup script + +The environment variables are specific to the role of the nodes, i.e. Primary, Backup, and Monitor. + +Assuming ``, `` and `` IP addresses for the nodes, depending on the role, here are the environment variables to be added to the beginning of above startup script: + +**Note:** Ensure that you replace the ``, `` and `` values according to your settings. + +Primary: +``` +##These are example values for configuring a primary node +export baseroutername=gcevmr +export nodetype=message_routing +export routername=gcevmr1 +export configsync_enable=yes +export redundancy_activestandbyrole=primary +export redundancy_enable=yes +export redundancy_group_password=gruyerecheese +export redundancy_group_node_gcevmr0_connectvia= +export redundancy_group_node_gcevmr0_nodetype=monitoring +export redundancy_group_node_gcevmr1_connectvia= +export redundancy_group_node_gcevmr1_nodetype=message_routing +export redundancy_group_node_gcevmr2_connectvia= +export redundancy_group_node_gcevmr2_nodetype=message_routing +export redundancy_matelink_connectvia= +``` + +Backup: +``` +##These are example values for configuring a backup node +export baseroutername=gcevmr +export nodetype=message_routing +export routername=gcevmr2 +export configsync_enable=yes +export redundancy_activestandbyrole=backup +export redundancy_enable=yes +export redundancy_group_password=gruyerecheese +export redundancy_group_node_gcevmr0_connectvia= +export redundancy_group_node_gcevmr0_nodetype=monitoring +export redundancy_group_node_gcevmr1_connectvia= +export redundancy_group_node_gcevmr1_nodetype=message_routing +export redundancy_group_node_gcevmr2_connectvia= +export redundancy_group_node_gcevmr2_nodetype=message_routing +export redundancy_matelink_connectvia= +``` + +Monitor: +``` +##These are example values for configuring a monitoring node +export baseroutername=gcevmr +export nodetype=monitoring +export routername=gcevmr0 +export redundancy_enable=yes +export redundancy_group_password=gruyerecheese +export redundancy_group_node_gcevmr0_connectvia= +export redundancy_group_node_gcevmr0_nodetype=monitoring +export redundancy_group_node_gcevmr1_connectvia= +export redundancy_group_node_gcevmr1_nodetype=message_routing +export redundancy_group_node_gcevmr2_connectvia= +export redundancy_group_node_gcevmr2_nodetype=message_routing +``` + + +### Step 2d: Submit the create request + +Now hit the "Create" button at the bottom of this page. This will begin the process of starting the GCE instance, installing Docker, and finally downloading and installing the message broker. + +It's possible to access the VM before the entire Solace solution is up. You can monitor `/var/lib/solace/install.log` for the following entry: `'date' INFO: Install is complete` to indicate when the installation has completed: + +* On the Google Cloud Platform console VM instances screen, locate the instance you've just started and wait for its status to become green (running). + +* In the Connect column, select a way to SSH into the VM and connect to it. + +* Check the logs: -# Set up network security to allow access -Now that the VMR is instantiated, the network security firewall rule needs to be set up to allow access to both the admin application and data traffic. Under the "Networking -> VPC network -> Firewall rules" tab add a new rule to your project exposing the required ports: +``` +[test@gcp-qs-test ~]$ sudo su +[root@gcp-qs-test test]# cd /var/lib/solace/ +[root@gcp-qs-test solace]# ls +install.log install-solace.sh swap +[root@gcp-qs-test solace]# tail -f install.log + Requires=docker.service + After=docker.service +[Service] + Restart=always + ExecStart=/usr/bin/docker start -a solace + ExecStop=/usr/bin/docker stop solace +[Install] + WantedBy=default.target +Fri Nov 23 15:33:55 UTC 2018 INFO: Start the Solace Message Router +Fri Nov 23 15:33:55 UTC 2018 INFO: Install is complete +``` + + +## Step 3: (HA cluster deployment only) Assert the primary message broker’s configuration + +As described in the [Solace documentation for configuring HA Group](https://docs.solace.com/Configuring-and-Managing/Configuring-HA-Groups.htm ) it's required to assert the primary message broker’s configuration after a Solace PubSub+ software message broker HA redundancy group is configured to support Guaranteed messaging. This can be done through Solace CLI commands as in the [documentation](https://docs.solace.com/Configuring-and-Managing/Configuring-HA-Groups.htm#Config-Config-Sync ) or running following commands at the Primary node (replace `` according to your settings): -![alt text](https://raw.githubusercontent.com/SolaceLabs/solace-gcp-quickstart/master/images/gce_network.png "GCE Firewall rules") +``` +# query redundancy status +curl -sS -u admin: http://localhost:8080/SEMP -d "" + +# wait until redundancy is up then execute the assert command: + +curl -sS -u admin: http://localhost:8080/SEMP -d "" +``` + +## Step 4: Set up network security to allow access + +Now that the message broker is instantiated, the network security firewall rule needs to be set up to allow access to both the admin application and data traffic. Under the "Networking -> VPC network -> Firewall rules" tab add a new rule to your project exposing the required ports: + +![alt text](/images/gce_network.png "GCE Firewall rules") `tcp:80;tcp:8080;tcp:1883;tcp:8000;tcp:9000;tcp:55003;tcp:55555` -For more information on the ports required for the message router see the [configuration defaults](http://docs.solace.com/Solace-VMR-Set-Up/VMR-Configuration-Defaults.htm) -. For more information on Google Cloud Platform Firewall rules see [Networking and Firewalls](https://cloud.google.com/compute/docs/networks-and-firewalls) +For more information on the ports required for the message broker see the [configuration defaults](https://docs.solace.com/Configuring-and-Managing/SW-Broker-Specific-Config/SW-Broker-Configuration-Defaults.htm ). For more information on Google Cloud Platform Firewall rules see [Networking and Firewalls](https://cloud.google.com/compute/docs/networks-and-firewalls ). + +**Note:** For troubleshooting, be aware that there may be existing firewall rules with the target "Apply to all", or otherwise applicable to the VMs you've created, and they will also be automatically applied. + +It may also be required to allow egress traffic to the internet for certain use cases. In this case, create an additional rule using similar steps. + +# Gaining admin access to the message broker + +Refer to the [Management Tools section](https://docs.solace.com/Management-Tools.htm ) of the online documentation to learn more about the available tools. The Solace PubSub+ Manager is the recommended way to administer the message broker for common tasks. + +## PubSub+ Manager, SolAdmin and SEMP access + +The Management IP will be the External IP associated with your GCE instance, and the port will be 8080 by default. -# Gaining admin access to the VMR +**Note:** If using the HA deployment, unless specifically required otherwise, use the GCE instance that is in the Active role (this is the Primary node at the initial setup, but can be the Backup node after a failover). -For persons used to working with Solace message router console access, this is still available with the google compute engine instance. Access the web ssh terminal window by clicking the [ssh] button next to your VMR instance, then launch a SolOS cli session: +![alt text](/images/gce_public_ip.png "getting started publish/subscribe") -![alt text](https://raw.githubusercontent.com/SolaceLabs/solace-gcp-quickstart/master/images/gce_console.png "GCE console with SolOS cli") -`sudo docker exec -it solace /usr/sw/loads/currentload/bin/cli -A` +## Solace CLI access -For persons who are unfamiliar with the Solace mesage router or would prefer an administration application, the imbedded PubSub+ Manager is available. For more information on PubSub+ Manager see the [PubSub+ Manager page](https://docs.solace.com/Solace-PubSub-Manager/PubSub-Manager-Overview.htm). Management IP will be the Public IP associated with youe GCE instance and port will be 8080 by default. +Access the web ssh terminal window by clicking the [ssh] button next to your message broker instance, then launch a Solace CLI session: -![alt text](https://raw.githubusercontent.com/SolaceLabs/solace-gcp-quickstart/master/images/gce_webui.png " PubSub+ Manager connection to gce") +```sh +$sudo docker exec -it solace /usr/sw/loads/currentload/bin/cli -A -# Testing data access to the VMR +Solace PubSub+ Standard Version 8.12.0.1007 -To test data traffic though the newly created VMR instance, visit the Solace developer portal and and select your preferred programming langauge to [send and receive messages](http://dev.solace.com/get-started/send-receive-messages/). Under each language there is a Publish/Subscribe tutorial that will help you get started. +The Solace PubSub+ Standard is proprietary software of +Solace Corporation. By accessing the Solace PubSub+ Standard +you are agreeing to the license terms and conditions located at +http://www.solace.com/license-software -![alt text](https://raw.githubusercontent.com/SolaceLabs/solace-gcp-quickstart/master/images/solace_tutorial.png "getting started publish/subscribe") +Copyright 2004-2018 Solace Corporation. All rights reserved. +To purchase product support, please contact Solace at: +http://dev.solace.com/contact-us/ + +Operating Mode: Message Routing Node + +solace-gcp-quickstart-master> +``` + +# Testing data access to the message broker + +To test data traffic though the newly created message broker instance, visit the Solace Developer Portal and select your preferred programming langauge to [send and receive messages](http://dev.solace.com/get-started/send-receive-messages/). Under each language there is a Publish/Subscribe tutorial that will help you get started and provide the specific default port to use. + +For single-node configuration, the IP to use will be the External IP associated with your GCE instance. For HA deployment the use of [Client Host List](https://docs.solace.com/Features/SW-Broker-Redundancy-and-Fault-Tolerance.htm#Failover ) is required for seamless failover - this will consist of the External IP addresses associated with your Primary and Backup node GCE instances. + +![alt text](/images/solace_tutorial.png "getting started publish/subscribe") ## Contributing @@ -75,7 +273,8 @@ Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduc ## Authors -See the list of [contributors](https://github.com/SolaceLabs/solace-gcp-quickstart/graphs/contributors) who participated in this project. +See the list of [contributors](https://github.com/SolaceProducts/solace-gcp-quickstart/graphs/contributors) who participated in this project. + ## License @@ -87,4 +286,4 @@ For more information about Solace technology in general please visit these resou - The Solace Developer Portal website at: http://dev.solace.com - Understanding [Solace technology.](http://dev.solace.com/tech/) -- Ask the [Solace community](http://dev.solace.com/community/). \ No newline at end of file +- Ask the [Solace community](http://dev.solace.com/community/). diff --git a/images/gce_createinstance_1.png b/images/gce_createinstance_1.png new file mode 100644 index 0000000..4671965 Binary files /dev/null and b/images/gce_createinstance_1.png differ diff --git a/images/gce_launch_2.png b/images/gce_launch_2.png index 5e10fd8..497ffcd 100644 Binary files a/images/gce_launch_2.png and b/images/gce_launch_2.png differ diff --git a/images/gce_launch_3.png b/images/gce_launch_3.png index b4b28ba..76e65c3 100755 Binary files a/images/gce_launch_3.png and b/images/gce_launch_3.png differ diff --git a/images/gce_public_ip.png b/images/gce_public_ip.png new file mode 100644 index 0000000..29dde88 Binary files /dev/null and b/images/gce_public_ip.png differ diff --git a/images/solace_tutorial.png b/images/solace_tutorial.png index e359c02..5414ceb 100644 Binary files a/images/solace_tutorial.png and b/images/solace_tutorial.png differ diff --git a/scripts/install-solace.sh b/scripts/install-solace.sh new file mode 100644 index 0000000..1240205 --- /dev/null +++ b/scripts/install-solace.sh @@ -0,0 +1,278 @@ +#!/bin/bash + +SOLACE_DOCKER_IMAGE_REF="solace/solace-pubsub-standard:latest" +USERNAME=admin +PASSWORD=admin +LOG_FILE=install.log +SWAP_FILE=swap +#cloud init vars +#array of all available cloud init variables to attempt to detect and pass to docker image creation +#see https://docs.solace.com/Configuring-and-Managing/SW-Broker-Specific-Config/Cloud-And-Machine-Tasks/Initializing-Config-Keys-With-Cloud-Init.htm +cloud_init_vars=( routername nodetype service_semp_port system_scaling_maxconnectioncount configsync_enable redundancy_activestandbyrole redundancy_enable redundancy_group_password redundancy_matelink_connectvia service_redundancy_firstlistenport ) + +# check if routernames contain any dashes or underscores and abort execution, if that is the case. +if [[ $routername == *"-"* || $routername == *"_"* || $baseroutername == *"-"* || $baseroutername == *"_"* ]]; then + echo "Dashes and underscores are not allowed in routername(s), aborting..." | tee -a ${LOG_FILE} + exit -1 +fi + +if [ ! -z "${baseroutername}" ]; then + cloud_init_vars+=( redundancy_group_node_${baseroutername}0_nodetype ) + cloud_init_vars+=( redundancy_group_node_${baseroutername}0_connectvia ) + cloud_init_vars+=( redundancy_group_node_${baseroutername}1_nodetype ) + cloud_init_vars+=( redundancy_group_node_${baseroutername}1_connectvia ) + cloud_init_vars+=( redundancy_group_node_${baseroutername}2_nodetype ) + cloud_init_vars+=( redundancy_group_node_${baseroutername}2_connectvia ) +fi + + +while [[ $# -gt 1 ]] +do + key="$1" + case $key in + -i|--url) + SOLACE_DOCKER_IMAGE_REF="$2" + shift # past argument + ;; + -l|--logfile) + LOG_FILE="$2" + shift # past argument + ;; + -p|--password) + PASSWORD="$2" + shift # past argument + ;; + -u|--username) + USERNAME="$2" + shift # past argument + ;; + *) + # unknown option + ;; + esac + shift # past argument or value +done + +echo "`date` INFO: Validate we have been passed a Solace Docker Image reference" &>> ${LOG_FILE} +# ----------------------------------------------------- +if [ -z "$SOLACE_DOCKER_IMAGE_REF" ] +then + echo "USAGE: install-solace.sh -i " + exit 1 +else + echo "`date` INFO: Solace Docker image reference is ${SOLACE_DOCKER_IMAGE_REF}" &>> ${LOG_FILE} +fi + + +echo "`date` INFO: Get repositories up to date" &>> ${LOG_FILE} +# --------------------------------------- + +yum -y update +yum -y install lvm2 + +echo "`date` INFO:Set up Docker Repository" &>> ${LOG_FILE} +# ----------------------------------- +tee /etc/yum.repos.d/docker.repo <<-EOF +[dockerrepo] +name=Docker Repository +baseurl=https://yum.dockerproject.org/repo/main/centos/7/ +enabled=1 +gpgcheck=1 +gpgkey=https://yum.dockerproject.org/gpg +EOF +echo "`date` INFO:/etc/yum.repos.d/docker.repo =\n `cat /etc/yum.repos.d/docker.repo`" &>> ${LOG_FILE} + +echo "`date` INFO:Intall Docker" &>> ${LOG_FILE} +# ------------------------- +yum -y install docker-engine + +echo "`date` INFO:Configure Docker as a service" &>> ${LOG_FILE} +# ---------------------------------------- +mkdir /etc/systemd/system/docker.service.d &>> install.log +tee /etc/systemd/system/docker.service.d/docker.conf <<-EOF +[Service] + ExecStart= + ExecStart=/usr/bin/dockerd --iptables=false --storage-driver=devicemapper +EOF +echo "`date` INFO:/etc/systemd/system/docker.service.d =\n `cat /etc/systemd/system/docker.service.d`" &>> ${LOG_FILE} + +systemctl enable docker +systemctl start docker + +## First make sure Docker is actually up +docker_running="" +loop_guard=10 +loop_count=0 +while [ ${loop_count} != ${loop_guard} ]; do + docker_running=`service docker status | grep -o running` + if [ ${docker_running} != "running" ]; then + ((loop_count++)) + echo "`date` WARN: Tried to launch Solace but Docker in state ${docker_running}" &>> ${LOG_FILE} + sleep 5 + else + echo "`date` INFO: Docker in state ${docker_running}" &>> ${LOG_FILE} + break + fi +done + +echo "`date` INFO: Get the solace image" &>> ${LOG_FILE} +# ------------------------------------------------ +# Determine first if SOLACE_DOCKER_IMAGE_REF is a valid docker registry uri +## Remove any existing solace image +if [ "`docker images | grep solace-`" ] ; then + echo "`date` INFO: Removing existing Solace images from local docker repo" &>> ${LOG_FILE} + docker rmi -f `docker images | grep solace- | awk '{print $3}'` +fi +## Try to load SOLACE_DOCKER_IMAGE_REF as a docker registry uri +echo "`date` Testing ${SOLACE_DOCKER_IMAGE_REF} for docker registry uri:" &>> ${LOG_FILE} +if [ -z "`docker pull ${SOLACE_DOCKER_IMAGE_REF}`" ] ; then + # If NOT in this branch then load was successful + echo "`date` INFO: Found that ${SOLACE_DOCKER_IMAGE_REF} was not a docker registry uri, retrying if it is a download link" &>> ${LOG_FILE} + if [[ ${SOLACE_DOCKER_IMAGE_REF} == *"solace.com/download"* ]]; then + REAL_LINK=${SOLACE_DOCKER_IMAGE_REF} + # the new download url + wget -O ${solace_directory}/solos.info -nv ${SOLACE_DOCKER_IMAGE_REF}_MD5 + else + REAL_LINK=${SOLACE_DOCKER_IMAGE_REF} + # an already-existing load (plus its md5 file) hosted somewhere else (e.g. in an s3 bucket) + wget -O ${solace_directory}/solos.info -nv ${SOLACE_DOCKER_IMAGE_REF}.md5 + fi + IFS=' ' read -ra SOLOS_INFO <<< `cat ${solace_directory}/solos.info` + MD5_SUM=${SOLOS_INFO[0]} + SolOS_LOAD=${SOLOS_INFO[1]} + if [ -z ${MD5_SUM} ]; then + echo "`date` ERROR: Missing md5sum for the Solace load - exiting." | tee /dev/stderr &>> ${LOG_FILE} + exit 1 + fi + echo "`date` INFO: Reference md5sum is: ${MD5_SUM}" &>> ${LOG_FILE} + + echo "`date` INFO: Now download from URL provided and validate, trying up to 5 times" &>> ${LOG_FILE} + LOOP_COUNT=0 + while [ $LOOP_COUNT -lt 5 ]; do + wget -q -O ${solace_directory}/${SolOS_LOAD} ${REAL_LINK} || echo "There has been an issue with downloading the Solace load" + ## Check MD5 + LOCAL_OS_INFO=`md5sum ${solace_directory}/${SolOS_LOAD}` + IFS=' ' read -ra SOLOS_INFO <<< ${LOCAL_OS_INFO} + LOCAL_MD5_SUM=${SOLOS_INFO[0]} + if [ -z "${MD5_SUM}" ] || [ "${LOCAL_MD5_SUM}" != "${MD5_SUM}" ]; then + echo "`date` WARN: Possible corrupt Solace load, md5sum do not match" &>> ${LOG_FILE} + else + echo "`date` INFO: Successfully downloaded ${SolOS_LOAD}" &>> ${LOG_FILE} + break + fi + ((LOOP_COUNT++)) + done + if [ ${LOOP_COUNT} == 3 ]; then + echo "`date` ERROR: Failed to download the Solace load, exiting" &>> ${LOG_FILE} + exit 1 + fi + ## Load the image tarball + docker load -i ${solace_directory}/${SolOS_LOAD} +fi +## Image details +export SOLACE_IMAGE_ID=`docker images | grep solace | awk '{print $3}'` +if [ -z "${SOLACE_IMAGE_ID}" ] ; then + echo "`date` ERROR: Could not load a valid Solace docker image - exiting." &>> ${LOG_FILE} + exit 1 +fi +echo "`date` INFO: Successfully loaded ${SOLACE_DOCKER_IMAGE_REF} to local docker repo" &>> ${LOG_FILE} +echo "`date` INFO: Solace message broker image and tag: `docker images | grep solace | awk '{print $1,":",$2}'`" &>> ${LOG_FILE} + + +echo "`date` INFO:Set up swap" &>> ${LOG_FILE} +# ----------------------------------------- +# Decide which scaling tier applies based on system memory +# and set maxconnectioncount, ulimit, devshm and swap accordingly +MEM_SIZE=`cat /proc/meminfo | grep MemTotal | tr -dc '0-9'` +if [ ${MEM_SIZE} -lt 4000000 ]; then + # 100 if mem<4GiB + maxconnectioncount="100" + shmsize="1g" + ulimit_nofile="2448:6592" + SWAP_SIZE="1024" +elif [ ${MEM_SIZE} -lt 12000000 ]; then + # 1000 if 4GiB<=mem<12GiB + maxconnectioncount="1000" + shmsize="2g" + ulimit_nofile="2448:10192" + SWAP_SIZE="2048" +elif [ ${MEM_SIZE} -lt 29000000 ]; then + # 10000 if 12GiB<=mem<28GiB + maxconnectioncount="10000" + shmsize="2g" + ulimit_nofile="2448:42192" + SWAP_SIZE="2048" +elif [ ${MEM_SIZE} -lt 58000000 ]; then + # 100000 if 28GiB<=mem<56GiB + maxconnectioncount="100000" + shmsize="3380m" + ulimit_nofile="2448:222192" + SWAP_SIZE="2048" +else + # 200000 if 56GiB<=mem + maxconnectioncount="200000" + shmsize="3380m" + ulimit_nofile="2448:422192" + SWAP_SIZE="2048" +fi +echo "`date` INFO: Based on memory size of ${MEM_SIZE}KiB, determined maxconnectioncount: ${maxconnectioncount}, shmsize: ${shmsize}, ulimit_nofile: ${ulimit_nofile}, SWAP_SIZE: ${SWAP_SIZE}" &>> ${LOG_FILE} + +echo "`date` INFO: Creating Swap space" &>> ${LOG_FILE} +mkdir /var/lib/solace +dd if=/dev/zero of=/var/lib/solace/swap count=${SWAP_SIZE} bs=1MiB +mkswap -f /var/lib/solace/swap +chmod 0600 /var/lib/solace/swap +swapon -f /var/lib/solace/swap +grep -q 'solace\/swap' /etc/fstab || sudo sh -c 'echo "/var/lib/solace/swap none swap sw 0 0" >> /etc/fstab' + +echo "`date` INFO:Create a Docker instance from Solace Docker image" &>> ${LOG_FILE} +# ------------------------------------------------------------- +SOLACE_CLOUD_INIT="--env SERVICE_SSH_PORT=2222" +[ ! -z "${USERNAME}" ] && SOLACE_CLOUD_INIT=${SOLACE_CLOUD_INIT}" --env username_admin_globalaccesslevel=${USERNAME}" +[ ! -z "${PASSWORD}" ] && SOLACE_CLOUD_INIT=${SOLACE_CLOUD_INIT}" --env username_admin_password=${PASSWORD}" +for var_name in "${cloud_init_vars[@]}"; do + [ ! -z ${!var_name} ] && SOLACE_CLOUD_INIT=${SOLACE_CLOUD_INIT}" --env $var_name=${!var_name}" +done + +echo "SOLACE_CLOUD_INIT set to:" | tee -a ${LOG_FILE} +echo ${SOLACE_CLOUD_INIT} | tee -a ${LOG_FILE} + +docker create \ + --uts=host \ + --shm-size=${shmsize} \ + --ulimit core=-1 \ + --ulimit memlock=-1 \ + --ulimit nofile=${ulimit_nofile} \ + --cap-add=IPC_LOCK \ + --cap-add=SYS_NICE \ + --net=host \ + --restart=always \ + --env "system_scaling_maxconnectioncount=${maxconnectioncount}" \ + ${SOLACE_CLOUD_INIT} \ + --name=solace ${SOLACE_IMAGE_ID} + +docker ps -a + +echo "`date` INFO:Construct systemd for Solace PubSub+" &>> ${LOG_FILE} +# -------------------------------------- +tee /etc/systemd/system/solace-docker.service <<-EOF +[Unit] + Description=solace-docker + Requires=docker.service + After=docker.service +[Service] + Restart=always + ExecStart=/usr/bin/docker start -a solace + ExecStop=/usr/bin/docker stop solace +[Install] + WantedBy=default.target +EOF +echo "`date` INFO:/etc/systemd/system/solace-docker.service =/n `cat /etc/systemd/system/solace-docker.service`" &>> ${LOG_FILE} + +echo "`date` INFO: Start the Solace Message Router" &>> ${LOG_FILE} +# -------------------------- +systemctl daemon-reload +systemctl enable solace-docker +systemctl start solace-docker + +echo "`date` INFO: Install is complete" &>> ${LOG_FILE} \ No newline at end of file diff --git a/solos-install.sh b/solos-install.sh deleted file mode 100644 index ea87f8d..0000000 --- a/solos-install.sh +++ /dev/null @@ -1,212 +0,0 @@ -#!/bin/bash - -URL="https://products.solace.com/download/PUBSUB_DOCKER_STAND" -MD5SUM="https://products.solace.com/download/PUBSUB_DOCKER_STAND_MD5" -USERNAME=admin -PASSWORD=admin -LOG_FILE=install.log -SWAP_FILE=swap -SOLACE_HOME=`pwd` -#cloud init vars -#array of all available cloud init variables to attempt to detect and pass to docker image creation -#see http://docs.solace.com/Solace-VMR-Set-Up/Initializing-Config-Keys-With-Cloud-Init.htm -cloud_init_vars=( routername nodetype service_semp_port system_scaling_maxconnectioncount configsync_enable redundancy_activestandbyrole redundancy_enable redundancy_group_password redundancy_matelink_connectvia service_redundancy_firstlistenport ) - -# check if routernames contain any dashes or underscores and abort execution, if that is the case. -if [[ $routername == *"-"* || $routername == *"_"* || $baseroutername == *"-"* || $baseroutername == *"_"* ]]; then - echo "Dashes and underscores are not allowed in routername(s), aborting..." | tee -a ${LOG_FILE} - exit -1 -fi - -#remove all dashes and underscores from routernames -#[ ! -z "${routername}" ] && routername=${routername/-/} -#[ ! -z "${routername}" ] && routername=${routername/_/} -#[ ! -z "${baseroutername}" ] && baseroutername=${baseroutername/-/} -#[ ! -z "${baseroutername}" ] && baseroutername=${baseroutername/_/} - -if [ ! -z "${baseroutername}" ]; then - cloud_init_vars+=( redundancy_group_node_${baseroutername}0_nodetype ) - cloud_init_vars+=( redundancy_group_node_${baseroutername}0_connectvia ) - cloud_init_vars+=( redundancy_group_node_${baseroutername}1_nodetype ) - cloud_init_vars+=( redundancy_group_node_${baseroutername}1_connectvia ) - cloud_init_vars+=( redundancy_group_node_${baseroutername}2_nodetype ) - cloud_init_vars+=( redundancy_group_node_${baseroutername}2_connectvia ) -fi - - -while [[ $# -gt 1 ]] -do - key="$1" - case $key in - -i|--url) - URL="$2" - shift # past argument - ;; - -l|--logfile) - LOG_FILE="$2" - shift # past argument - ;; - -m|--md5sum) - MD5SUM="$2" - shift # past argument - ;; - -p|--password) - PASSWORD="$2" - shift # past argument - ;; - -u|--username) - USERNAME="$2" - shift # past argument - ;; - *) - # unknown option - ;; - esac - shift # past argument or value -done - -echo "`date` INFO: Validate we have been passed a VMR url" &>> ${LOG_FILE} -# ----------------------------------------------------- -if [ -z "$URL" ] -then - echo "USAGE: vmr-install.sh --url " &>> ${LOG_FILE} - exit 1 -else - echo "`date` INFO: VMR URL is ${URL}" &>> ${LOG_FILE} -fi - - -echo "`date` INFO:Get repositories up to date" &>> ${LOG_FILE} -# --------------------------------------- - -yum -y update &>> ${LOG_FILE} -yum -y install lvm2 &>> ${LOG_FILE} - -echo "`date` INFO:Set up Docker Repository" &>> ${LOG_FILE} -# ----------------------------------- -tee /etc/yum.repos.d/docker.repo <<-EOF -[dockerrepo] -name=Docker Repository -baseurl=https://yum.dockerproject.org/repo/main/centos/7/ -enabled=1 -gpgcheck=1 -gpgkey=https://yum.dockerproject.org/gpg -EOF -echo "`date` INFO:/etc/yum.repos.d/docker.repo =\n `cat /etc/yum.repos.d/docker.repo`" &>> ${LOG_FILE} - -echo "`date` INFO:Intall Docker" &>> ${LOG_FILE} -# ------------------------- -yum -y install docker-engine &>> ${LOG_FILE} - -echo "`date` INFO:Configure Docker as a service" &>> ${LOG_FILE} -# ---------------------------------------- -mkdir /etc/systemd/system/docker.service.d &>> install.log -tee /etc/systemd/system/docker.service.d/docker.conf <<-EOF -[Service] - ExecStart= - ExecStart=/usr/bin/dockerd --iptables=false --storage-driver=devicemapper -EOF -echo "`date` INFO:/etc/systemd/system/docker.service.d =\n `cat /etc/systemd/system/docker.service.d`" &>> ${LOG_FILE} - -systemctl enable docker &>> ${LOG_FILE} -systemctl start docker &>> ${LOG_FILE} - -echo "`date` INFO:Set up swap for < 6GB machines" &>> ${LOG_FILE} -# ----------------------------------------- -MEM_SIZE=`cat /proc/meminfo | grep MemTotal | tr -dc '0-9'` &>> ${LOG_FILE} -if [ ${MEM_SIZE} -lt 6087960 ]; then - echo "`date` WARN: Not enough memory: ${MEM_SIZE} Creating 2GB Swap space" &>> ${LOG_FILE} - mkdir /var/lib/solace &>> ${LOG_FILE} - dd if=/dev/zero of=/var/lib/solace/swap count=2048 bs=1MiB &>> ${LOG_FILE} - mkswap -f /var/lib/solace/swap &>> ${LOG_FILE} - chmod 0600 /var/lib/solace/swap &>> ${LOG_FILE} - swapon -f /var/lib/solace/swap &>> ${LOG_FILE} - grep -q 'solace\/swap' /etc/fstab || sudo sh -c 'echo "/var/lib/solace/swap none swap sw 0 0" >> /etc/fstab' &>> ${LOG_FILE} -else - echo "`date` INFO: Memory size is ${MEM_SIZE}" &>> ${LOG_FILE} -fi - - -echo "`date` INFO:Get the md5sum of the expected solace image" &>> ${LOG_FILE} -# ------------------------------------------------ -wget -O /tmp/solos.info -nv ${MD5SUM} -IFS=' ' read -ra SOLOS_INFO <<< `cat /tmp/solos.info` -MD5_SUM=${SOLOS_INFO[0]} -SolOS_LOAD=${SOLOS_INFO[1]} -if [ -z ${MD5_SUM} ]; then - echo "`date` ERROR: Missing md5sum for the Solace load" | tee /dev/stderr | tee /dev/stderr &>> ${LOG_FILE} - exit 1 -fi -echo "`date` INFO: Reference md5sum is: ${MD5_SUM}" &>> ${LOG_FILE} - - -echo "`date` INFO:Get the solace image" &>> ${LOG_FILE} -# ------------------------------------------------ -wget -q -O /tmp/${SolOS_LOAD} ${URL} -LOCAL_OS_INFO=`md5sum /tmp/${SolOS_LOAD}` -IFS=' ' read -ra SOLOS_INFO <<< ${LOCAL_OS_INFO} -LOCAL_MD5_SUM=${SOLOS_INFO[0]} -if [ ${LOCAL_MD5_SUM} != ${MD5_SUM} ]; then - echo "`date` ERROR: Possible corrupt Solace load, md5sum do not match" | tee /dev/stderr &>> ${LOG_FILE} - exit 1 -else - echo "`date` INFO: Successfully downloaded ${SolOS_LOAD}" &>> ${LOG_FILE} -fi - -docker load -i /tmp/${SolOS_LOAD} &>> ${LOG_FILE} -export SOLOS_VERSION=`docker images | grep solace | awk '{print $3}'` &>> ${LOG_FILE} -echo "`date` INFO: Solace message broker image: ${SOLOS}" &>> ${LOG_FILE} - - -echo "`date` INFO:Create a Docker instance from Solace Docker image" &>> ${LOG_FILE} -# ------------------------------------------------------------- -VMR_VERSION=`docker images | grep solace | awk '{print $2}'` - -SOLACE_CLOUD_INIT="--env SERVICE_SSH_PORT=2222" -[ ! -z "${USERNAME}" ] && SOLACE_CLOUD_INIT=${SOLACE_CLOUD_INIT}" --env username_admin_globalaccesslevel=${USERNAME}" -[ ! -z "${PASSWORD}" ] && SOLACE_CLOUD_INIT=${SOLACE_CLOUD_INIT}" --env username_admin_password=${PASSWORD}" -for var_name in "${cloud_init_vars[@]}"; do - [ ! -z ${!var_name} ] && SOLACE_CLOUD_INIT=${SOLACE_CLOUD_INIT}" --env $var_name=${!var_name}" -done - -echo "SOLACE_CLOUD_INIT set to:" | tee -a ${LOG_FILE} -echo ${SOLACE_CLOUD_INIT} | tee -a ${LOG_FILE} - -docker create \ - --uts=host \ - --shm-size 2g \ - --ulimit core=-1 \ - --ulimit memlock=-1 \ - --ulimit nofile=2448:38048 \ - --cap-add=IPC_LOCK \ - --cap-add=SYS_NICE \ - --net=host \ - --restart=always \ - ${SOLACE_CLOUD_INIT} \ - --name=solace ${SOLOS_VERSION} &>> ${LOG_FILE} - -docker ps -a &>> ${LOG_FILE} - -echo "`date` INFO:Construct systemd for Solace PubSub+" &>> ${LOG_FILE} -# -------------------------------------- -tee /etc/systemd/system/solace-docker.service <<-EOF -[Unit] - Description=solace-docker - Requires=docker.service - After=docker.service -[Service] - Restart=always - ExecStart=/usr/bin/docker start -a solace - ExecStop=/usr/bin/docker stop solace -[Install] - WantedBy=default.target -EOF -echo "`date` INFO:/etc/systemd/system/solace-docker.service =/n `cat /etc/systemd/system/solace-docker.service`" &>> ${LOG_FILE} - -echo "`date` INFO: Start the Solace Message Router" -# -------------------------- -systemctl daemon-reload &>> ${LOG_FILE} -systemctl enable solace-docker &>> ${LOG_FILE} -systemctl start solace-docker &>> ${LOG_FILE} - -echo "`date` INFO: Install is complete" &>> ${LOG_FILE} \ No newline at end of file