Maestro: Conduct your clouds

Maestro is a cloud provisioning, configuration, and management utility for your Ruby and Ruby On Rails applications. Simply declare the structure of your clouds via configuration files, create Chef recipes to configure the nodes in your clouds, and Maestro takes care of the rest.

Maestro currently supports the Amazon Web Services cloud. Support for other cloud providers is on the roadmap.

Maestro currently supports the following Linux distributions:

Ubuntu (10.04, 9.10, 9.04, 8.10, 8.04)
Debian (5.0)
Fedora (8)
CentOS (5.4)

Maestro has been tested with the following Ruby versions (thanks RVM!):

MRI (1.8.7, 1.9.1, 1.9.2) JRuby (1.5.2)

Using Maestro with Ruby On Rails¶ ↑

Installation with Ruby On Rails 3¶ ↑

Add the following gem dependency to your Rails project’s Gemfile:

gem 'the-maestro', '0.4.0', :require => 'maestro'

Then run the following to install the Maestro gem:

bundle install

Installation with Ruby On Rails 2¶ ↑

Add the following gem dependency to your Rails project’s config/environment.rb file:

Rails::Initializer.run do |config|

  config.gem "the-maestro", :lib => "maestro", :version => "0.4.0", :source => "http://gemcutter.org"

end

Then run the following to install the Maestro gem:

rake gems:install

You may optionally unpack Maestro into vendor/gems by running:

rake gems:unpack:dependencies

You’ll interact with Maestro through custom Rake tasks that Maestro adds to your Rails project. Due to this issue in Rails 2 (fixed in Rails 3), you’ll need to add this to your Rails project’s Rakefile:

require 'maestro/tasks'

Make sure the Maestro gem is successfully installed before adding this line to your Rails project’s Rakefile.

Configuration¶ ↑

To create the Maestro configuration directory structure, run the following Rake task:

rake maestro:create_config_dirs

This will create the following directory structure within your Rails project’s config/ directory:

YOUR_RAILS_APP/config/maestro/clouds/ - the directory which contains your cloud configuration files
YOUR_RAILS_APP/config/maestro/cookbooks/ - the directory which contains your Chef cookbooks
YOUR_RAILS_APP/config/maestro/roles/ - the directory which contains your Chef JSON Roles files

If these directories already exist when running rake maestro:create_config_dirs, no action is taken.

Declare your clouds¶ ↑

To declare a cloud, simply create a configuration file within your Rails project’s YOUR_RAILS_APP/config/maestro/clouds/ directory with the same name as your cloud. For example, to create “dev”, “staging”, and “production” clouds, your YOUR_RAILS_APP/config/maestro/clouds/ directory would contain the following files:

YOUR_RAILS_APP/config/maestro/clouds/dev.rb
YOUR_RAILS_APP/config/maestro/clouds/staging.rb
YOUR_RAILS_APP/config/maestro/clouds/production.rb

A cloud configuration file is a Ruby file containing a simple DSL for describing the structure of a cloud for a given cloud provider. Currently the Amazon Web Services cloud is the only supported provider.

Amazon Web Services Cloud pre-requisites¶ ↑

To use the Amazon Web Services cloud provider, the following are minimally required:

• An Amazon Web Services Account

• EC2 enabled in your account

• S3 enabled in your account

• AWS Account ID

• AWS Access Key

• AWS Secret Access Key

• AWS Keypair name and keypair file stored locally

You may optionally utilize all other features of the Amazon Web Services cloud infrastructure, such as ELB, RDS, etc, assuming that these capabilities are enabled in your Amazon Web Services account. See the Amazon Web Services documentation for more information.

Anatomy of an Amazon Web Services cloud configuration file¶ ↑

aws_cloud :dev do

  keypair_name "XXXXXXX-keypair"
  keypair_file "/path/to/id_rsa-XXXXXXX-keypair"
  aws_account_id "XXXX-XXXX-XXXX"
  aws_access_key "XXXXXXXXXXXXXXXXXXXX"
  aws_secret_access_key "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  chef_bucket "maestro.yourdomain.com"

  roles do
    role "web" do
      public_ports [80, 443]
    end
  end

  nodes do
    ec2_node "web-1" do
      roles ["web"]
      ami "ami-bb709dd2"
      ssh_user "ubuntu"
      instance_type "m1.small"
      availability_zone "us-east-1b"
    end

    ...etc...
  end
end

Let’s look at each of these sections in more detail:

aws_cloud cloud_name¶ ↑

The aws_cloud method declares an Amazon Web Services cloud with the given name. The cloud name can be either a symbol or a string.

aws_cloud :dev do
   ...
end

aws_cloud "dev" do
   ...
end

A cloud name may only contain alphanumerics and dashes.

aws_cloud attributes¶ ↑

The following attributes must be present within your aws_cloud declaration:

• keypair_name: A keypair associated with your Amazon Web Services account

• keypair_file: Fully qualified path to the keypair file on your computer matching the above keypair_name

• aws_account_id: Your Amazon Web Services Account ID

• aws_access_key: Your Amazon Web Services Access Key

• aws_secret_access_key: Your Amazon Web Services Secret Access Key

• chef_bucket: The name of the Amazon S3 Bucket where you want your Chef recipes and roles stored. If this Bucket does not exist, Maestro will create it for you. Please note that the Bucket itself, and all objects stored within the Bucket use the canned ‘private’ access policy, meaning no one but the Bucket owner (i.e. you) may access the Bucket or the objects within it. Maestro uses query string authentication when accessing the objects to ensure the integrity of your data.

roles¶ ↑

The roles method allows you to apply configuration to all of the nodes in your cloud which are in a given role, rather than repeating that configuration for all nodes individually.

role role_name¶ ↑

A role corresponds to a Chef role that the configurable nodes in your Cloud can play. A role name may either be a symbol or a string.

role "web" do
  public_ports [80, 443]
end

A role name may only contain alphanumerics and dashes.

role attributes¶ ↑

A role may configure the following attributes:

• public_ports: The ports that should be open to the public internet, for example ports 80 and 443 for a web role. The value of this attribute must be an array of integers.

nodes¶ ↑

The nodes method is where you define the nodes (servers, machine, VMs, devices, etc…) in your cloud.

AWS Node types¶ ↑

The following are the AWS node types supported by Maestro:

ec2_node node-name¶ ↑

Creates an Elastic Cloud Compute (EC2) node. An EC2 node name may only contain alphanumerics and dashes.

The following attributes must be present:

• roles: An array of Chef roles to apply to this node. The role names in the array must match the Chef roles that are defined in config/maestro/roles.

• ami: The AMI identifier to use.

• ssh_user: The name of the user to use to ssh to the instance.

• instance_type: The instance type to use for this EC2 node.

• availability_zone: The availability zone to launch the instance in.

The following attributes are optional:

• cookbook_attributes: Chef cookbook attribute overrides, in JSON format (see the Chef documentation for more information). Since the attribute JSON format necessitates lots of “ characters, the use of heredocs is recomended to avoid manual escaping of ” characters and improve readability. See the example below.

• elastic_ip: The Elastic IP address to associate with this node.

• ebs_volume_id: The id of the EBS Volume to attach to this node.

• ebs_device: The device to attach the EBS Volume to.

An example ec2_node:

ec2_node "node-1" do
  roles ["role-1"]
  ami "ami-bb709dd2"
  ssh_user "ubuntu"
  instance_type "m1.small"
  availability_zone "us-east-1b"
  cookbook_attributes <<-COOKBOOK_ATTRIBUTES
    "apache": {
      "listen_ports": ["81", "8181"],
      "worker" : {
        "startservers" : "5"
      }
    },
    "mysql": {
      "server_root_password": "XXXXXXXX",
      "tunable": {
        "max_connections": "500"
      }
    }
  COOKBOOK_ATTRIBUTES
  elastic_ip "123.45.678.90"
  ebs_volume_id "vol-xxxxxxxx"
  ebs_device "/dev/sdh"
end

elb_node node-name¶ ↑

Creates an Elastic Load Balancer (ELB) node. An ELB node name may only contain alphanumerics and dashes.

The following attributes must be present:

• availability_zones: An array of availability zone names this ELB node will operate in.

• listeners: An array of hashes configuring the ELB listeners.

• ec2_nodes: An array of EC2 node names that this ELB fronts.

• health_check: A hash configuring the health check of this ELB.

An example elb_node:

elb_node "lb-1" do
  availability_zones ["us-east-1b"]
  listeners [{:load_balancer_port => 80, :instance_port => 80, :protocol => "http"}]
  ec2_nodes ["app-1", "app-2"]
  health_check(:target => "TCP:80", :timeout => 15, :interval => 60, :unhealthy_threshold => 5, :healthy_threshold => 3)
end

rds_node node-name¶ ↑

Creates a Relational Database Service (RDS) node. An RDS node name may only contain alphanumerics and dashes.

The following attributes must be present:

• engine: The name of the database engine to use.

• db_instance_class: The instance class to use.

• master_username: The master user’s username.

• master_user_password: The master user’s password.

• port: The port this RDS node should listen on.

• allocated_storage: The storage to allocate (in GB).

• availability_zone: The availability zone to launch this RDS node in.

• preferred_maintenance_window: The preferred 4 hour window in which to run maintenance.

• preferred_backup_window: The preferred 2 hour window in which backups should run.

• backup_retention_period: The number of days to retain backups.

The following attributes are optional:

• db_parameters: An array of hashes containing “name” and “value” keys representing database parameters.

An example rds_node:

rds_node "db-2" do
  engine "MySQL5.1"
  db_instance_class "db.m1.small"
  master_username "root"
  master_user_password "password"
  port 3306
  allocated_storage 5
  availability_zone "us-east-1b"
  preferred_maintenance_window "Sun:03:00-Sun:07:00"
  preferred_backup_window "03:00-05:00"
  backup_retention_period 7
  db_parameters [{:name => "character_set_server", :value => "utf8"},
                 {:name => "collation_server", :value => "utf8_bin"},
                 {:name => "long_query_time", :value => "5"}]
end

A note about node definitions¶ ↑

If your cloud has a large number of nodes which are identical other than their name, defining each node individually with a node method would be tedious. Don’t forget that your cloud definition files are Ruby, which allows you to do things like the following:

nodes do
  10.times do |i|
    ec2_node "web-#{i+1}" do
      .......
    end
  end
end

This will create 10 different ec2_nodes, each with a distinct name (“web-1” through “web-10”).

Validate your configuration¶ ↑

At any time, you may run the following Rake task to validate your Maestro configurations:

rake maestro:validate_configs

This will validate all of the files in your Maestro config directory, and report any errors.

Conduct!¶ ↑

Maestro creates Rake tasks for all of your valid clouds. These tasks allow you to provision, configure, and deploy to your cloud. A very simplistic workflow:

rake maestro:validate_configs

rake maestro:mycloud:status
rake maestro:mycloud:start
rake maestro:mycloud:configure
rake maestro:mycloud:shutdown

If you do not see a maestro:cloud-name:… set of tasks for a named cloud when running rake -T, your cloud configuration file is not valid. Simply run…

rake maestro:validate_configs

…and you will be given a report as to what is wrong with the given cloud configuration file.

To see all available rake commands for all of your clouds:

rake -T | grep maestro

Configuring the nodes in your cloud¶ ↑

Maestro uses the Chef framework to configure the nodes in your cloud. Any roles defined in your ec2_nodes must map to Chef Roles with the same name. These Chef Roles are stored as JSON files within YOUR_RAILS_APP/config/maestro/roles/. When you run rake maestro:mycloud:configure, Maestro will apply the recipes defined in the Role JSON files on all nodes in that role. Your recipes should be placed in YOUR_RAILS_APP/config/maestro/cookbooks.

For a more in depth explanation of Chef, please consult the Chef documentation.

Logging¶ ↑

Maestro will log cloud-wide workflow messages to STDOUT. In addition, log files will be created for your cloud, as well as each of the nodes in your cloud. These can be found at the following location:

YOUR_RAILS_APP/log/maestro/clouds/cloud-name/
YOUR_RAILS_APP/log/maestro/clouds/cloud-name/cloud-name.log
YOUR_RAILS_APP/log/maestro/clouds/cloud-name/node-name.log

Node-specific messages are not logged to STDOUT, but only to the node’s log file. For troubleshooting problems configuring your nodes, please consult the individual node log files for information.

Using Maestro stand alone¶ ↑

Maestro can also be used in stand alone mode, as a simple command line cloud management utility.

Installation¶ ↑

Install the Maestro gem by running the following command:

gem install the-maestro

Requirements¶ ↑

Maestro requires the following in order to be run in stand alone mode:

• A base directory structure

• An environment variable MAESTRO_DIR set to the location of the base directory

• A Rake file

Create the base directory structure¶ ↑

Maestro requires a directory representing your project. This directory will house your Rake file, and a subdirectory named “config” where all of your Maestro configuration files will live.

For example, suppose we are running a Hadoop cluster in the cloud. We would create the following directory structure:

mkdir /home/bob/projects/hadoop-cluster
mkdir /home/bob/projects/hadoop-cluster/config

Set the MAESTRO_DIR environment variable¶ ↑

When run in stand alone mode, Maestro requires that an environment variable named MAESTRO_DIR be set, pointing at the base directory defined above. The mechanism for setting environment variables is operating system and/or shell specific, but as an example:

export MAESTRO_DIR="/home/bob/projects/hadoop-cluster"

Please note that if you wish to use Maestro with multiple stand alone projects, you may want to set the MAESTRO_DIR environment variable in shell/batch scripts.

Create a Rakefile¶ ↑

You’ll interact with Maestro through Rake tasks. Create a Rakefile in your base directory, and add the following:

require 'rubygems'
require 'rake'
require 'maestro/tasks'

Make sure the Maestro gem is successfully installed before adding these lines to your Rakefile.

Create the Maestro configuration directory structure¶ ↑

To create the Maestro configuration directory structure, run the following Rake task within the base directory containing your Rake file:

rake maestro:create_config_dirs

This will create the following directory structure within your project’s config/ directory:

$MAESTRO_DIR/config/maestro/clouds/ - the directory which contains your cloud configuration files
$MAESTRO_DIR/config/maestro/cookbooks/ - the directory which contains your Chef cookbooks
$MAESTRO_DIR/config/maestro/roles/ - the directory which contains your Chef JSON Roles files

If these directories already exist when running rake maestro:create_config_dirs, no action is taken.

Conduct!¶ ↑

At this point, you can use Maestro as described above. Define your clouds, create your Chef assets, and conduct!

Where To Get Help¶ ↑

The Maestro Users Google Group is the recommended place to get help.

Contributing¶ ↑

Please use the Issues page on the Maestro GitHub site to report bugs or feature requests.

To contribute fixes for Issues, please create a topic branch off of the appropriate branch and send a pull request indicating which Issue the commit is for. All bug fixes must have a corresponding unit and/or integration test (depending on the nature of the Issue) in order to be accepted.

To run the unit tests:

rake test:units

To run all integration tests:

rake test:integration

To run just the AWS integration tests (please see test/integration/base_aws.rb for instructions on how to supply your AWS credentials to run the tests):

rake test:integration:aws

Due to the nature of the AWS integration tests, they take quite a long time to run (well over an hour), so you may want to grab a beer or 3 to pass the time.

To run just the operating system integration tests:

rake test:integration:centos
rake test:integration:debian
rake test:integration:fedora
rake test:integration:ubuntu

Be mindful of the code coverage metrics. We try to maintain a > 85% coverage percentage, and any pull requests should not result in a lower percentage unless for a very good reason.

To run RCov:

rake rcov

To install the gem locally:

rake install

To generate RDoc locally:

rake rdoc

License¶ ↑

(The MIT License)

Copyright © 2010 Brian Ploetz <bploetz (at) gmail.com>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR