Skip to content

Latest commit

 

History

History
330 lines (259 loc) · 7.95 KB

README.adoc

File metadata and controls

330 lines (259 loc) · 7.95 KB
Raw

Crunchy Watch

crunchy logo

Overview

Crunchy Watch is an application that watches a PostgreSQL master and looks for a failure, at which point it will perform a failover scenario.

Failover scenarios are extensible. Sample failover scenarios are provided including:

  • trigger a failover on a random replica

  • trigger a failover on a replica using metadata labels

  • trigger a failover on a replica that is further ahead than others

Crunchy Watch is packaged into a Docker container which can execute in a pure Docker 1.12, Kubernetes 1.7, and Openshift 3.6 environments.

You can also run Crunchy Watch outside of a container as a binary.

Crunchy provides a commercially supported version of this container built upon RHEL 7 and the Crunchy supported PostgreSQL. Contact Crunchy for more details at link:http://www.crunchydata.com.

Usage

Crunchy Watch is designed to operate on multiple platforms. Therefore, it is necessary to specify the platform at startup.

$> crunchy-watch <platform>

Supported Platforms:

Name Value

Kubernetes

kube

Openshift

openshift

Example:

$> crunchy-watch kube

Options

All options can be configured via a command-line flag or an environment variable.

Flag values will take precedence over values defined by an environment variable.

There are general options that apply across all platforms. As well, each platform provides their own specific options. The details for each are provided below.

General

Option Environment Variable Default Description

--primary

CRUNCHY_WATCH_PRIMARY

host of the primary PostgreSQL instance

--primary-port

CRUNCHY_WATCH_PRIMARY_PORT

5432

port of the primary PostreSQL instance

--replica

CRUNCHY_WATCH_REPLICA

host of the replica PostgreSQL instance

--replica-port

CRUNCHY_WATCH_REPLICA_PORT

5432

port of the replica PostgreSQL instance

--username

CRUNCHY_WATCH_USERNAME

postgres

login user to connect to PostgreSQL

--password

CRUNCHY_WATCH_PASSWORD

login user’s password to connect to PostgreSQL

--database

CRUNCHY_WATCH_DATABASE

postgres

database to connect

--timeout

CRUNCHY_WATCH_TIMEOUT

10s

connection timeout - valid time units are "ns", "us", "ms", "s", "m", "h"

--max-failures

CRUNCHY_WATCH_MAX_FAILURES

0

maximum number of failures before performing a failover

--healthcheck-interval

CRUNCHY_WATCH_HEALTHCHECK_INTERVAL

10s

interval between healthchecks - valid time units are "ns", "us", "ms", "s", "m", "h"

--failover-wait

CRUNCHY_WATCH_FAILOVER_WAIT

50s

time to wait for failover to process - valid time units are "ns", "us", "ms", "s", "m", "h"

--pre-hook

CRUNCHY_WATCH_PRE_HOOK

failover hook to execute before processing failover

--post-hook

CRUNCHY_WATCH_POST_HOOK

failover hook to execute after processing failover

Kubernetes

Name Environment Variable Default Description

--kube-namespace

CRUNCHY_WATCH_KUBE_NAMESPACE

default

the kubernetes namespace

--kube-failover-strategy

CRUNCHY_WATCH_KUBE_FAILOVER_STRATEGY

default

the kubernetes failover strategy

Openshift

Option Environment Variable Default Description

--openshift-project

CRUNCHY_WATCH_OPENSHIFT_PROJECT

default

the openshift project (namespace)

--openshift-failover-strategy

CRUNCHY_WATCH_OPENSHIFT_FAILOVER_STRATEGY

default

the openshift failover strategy

Build

Building crunchy-watch, supporting plugin modules and docker image are accomplished using make and the provide Makefile.

Requirements

Note: The $> make setup target (below) will retrieve the requisite kubernetes and openshift client binaries.

Centos Build Steps

These steps assume your normal userid is someuser and you are installing on a clean minimal Centos7 install.

Install Docker

sudo yum -y install docker
sudo groupadd docker
sudo systemctl enable docker
sudo systemctl start docker
sudo usermod -a -G docker someuser
newgrp docker
docker ps

Install Build Dependencies

sudo yum -y install gettext git golang

Setup Project Settings and Structure

export GOPATH=$HOME/odev
mkdir -p $GOPATH/src $GOPATH/bin $GOPATH/pkg
mkdir -p $GOPATH/src/github.com/crunchydata/
export PATH=$PATH:$GOPATH/bin
export CCP_BASEOS=centos7
export CCP_PGVERSION=10
export CCP_PG_FULLVERSION=10.1
cd $GOPATH/src/github.com/crunchydata
git clone https://github.com/CrunchyData/crunchy-watch.git
cd crunchy-watch

Get Project Dependencies

go get github.com/Masterminds/glide
glide update
make setup

Build from Source

make

Build the Docker Image

Note
 To build the RHEL based image, you will need the Crunchy repo keys to be copied to the $GOPATH/src/github.com/crunchydata/crunchy-watch directory. This is because the RHEL image is based on the Crunchy RPM packages. 
cp CRUNCHY-GPG-KEY.public  $GOPATH/src/github.com/crunchydata/crunchy-watch
cp crunchypg*.repo $GOPATH/src/github.com/crunchydata/crunchy-watch
make docker-image

Targets

Target Description

all

(default) calls clean, resolve and build targets

build

builds crunchy-watch binary

modules

builds all plugin modules

kube-module

builds kubernetes plugin module

openshift-module

builds openshift plugin module

clean

cleans all build related artifacts, including dependencies.

resolve

resolves all build related dependencies

docker-image

build docker image - Note: requires CCP_BASEOS, CCP_PGVERSION,

CCP_PG_FULLVERSION and CCP_VERSION to be defined.

setup

Extending Crunchy Watch

Crunchy Watch is designed with extension of its function and supported platforms in mind.

Extending by Plugin

Crunchy Watch makes use of the golang plugin package. Therefore it is possible to build support for new platforms separate from each other.

To integrate with the plugin system the following interface must be met:

type FailoverHandler interface {
	Failover() error
	SetFlags(*flag.FlagSet)
}

Failover() is called to process the failover logic for the platform that the plugin supports.

SetFlags(*flag.FlagSet) is called immediately after the plugin is loaded. This allows for plugin to define options/flags that are unique to its operation.

As well, it must be built with the -buildmode=plugin option. See an example of this in the project Makefile

Extending by Hook

Crunchy Watch provides both a pre and post failover hook. These hooks will be executed in a shell environment created by the crunchy-watch process. Therefore they can be any executable or script that can be called by the user running the crunchy-watch process.

To configure the execution of these hooks, a fully qualified path to the executable or script must be provided by either the --pre-hook or --post-hook flags. Or by defining the CRUNCHY_WATCH_PRE_HOOK or CRUNCHY_WATCH_POST_HOOK environment variables.

Example:

$> crunchy-watch kube --pre-hook=/tmp/watch-pre-hook

Or,

$> CRUNCHY_WATCH_PRE_HOOK=/tmp/watch-pre-hook crunchy-watch kube

Examples

Various examples are provided in the examples directory and described in the documentation.

To run the examples, you will need to set the CCP_IMAGE_TAG environment variable which indicates which version of the container you will pull down and execute, for example:

$> export CCP_IMAGE_TAG=centos7-10.1-1.7.1
$> ./run.sh

Or,

$> CCP_IMAGE_TAG=centos7-10.1-1.7.1 ./run.sh