spiceweasel

Description

Spiceweasel is a command-line tool for batch loading Chef infrastructure. It provides a simple syntax in Ruby, JSON or YAML for describing and deploying infrastructure in order with the Chef command-line tool knife. This manifest may be bundled with a Chef repository to deploy the infrastructure contained within the repository and validate that the components listed are all present. The manifest may also be extracted from an existing repository.

The https://github.com/mattray/lab-repo provides a working example for bootstrapping a Chef repository with Spiceweasel.

The CHANGELOG.md covers current, previous and future development milestones and contains the features backlog.

Requirements

Spiceweasel currently depends on knife to run commands for it, and requires the chef gem for validating cookbook metadata. Berkshelf is a dependency for the Cookbook Berksfile support. Infrastructure files must end in .rb, .json or .yml to be processed.

Written and tested with the Chef 11.x series (previous versions of Chef may still work). It is tested with Ruby 1.9.3. Version 2.0 was the last version known to work with Ruby 1.8.7 due to the introduction of the Berkshelf dependency in 2.1. If you want to use Ruby 2.0, you will need to use the Chef 11.6 (or later) gem.

File Syntax

The syntax for the Spiceweasel file may be Ruby, JSON or YAML format of Chef primitives describing what is to be instantiated. Please refer to the test/examples/example.json or test/examples/example.yml for examples of the same infrastructure. Each subsection below shows the YAML syntax converted to knife commands.

Cookbooks

The cookbooks section of the manifest currently supports knife cookbook upload FOO where FOO is the name of the cookbook in the cookbooks directory. The default behavior is to download the cookbook as a tarball, untar it and remove the tarball. The --siteinstall option will allow for use of knife cookbook site install with the cookbook and the creation of a vendor branch if git is the underlying version control. Validation is done to ensure the cookbook matches the name (and version if given) in the metadata and that any cookbook dependencies are listed in the manifest. You may pass any additional options if necessary. Assuming the apt cookbook was not present, the example YAML snippet

cookbooks:
- apache2:
- apt:
    version: 1.2.0
    options: --freeze
- mysql:
- ntp:

produces the knife commands

knife cookbook upload apache2
knife cookbook site download apt 1.2.0 --file cookbooks/apt.tgz
tar -C cookbooks/ -xf cookbooks/apt.tgz
rm -f cookbooks/apt.tgz
knife cookbook upload apt --freeze
knife cookbook upload mysql ntp

Berkshelf

If you prefer to use Berkshelf for managing your cookbooks, Spiceweasel supports adding berksfile: and the ability to specify the path and any Berkshelf options. You may mix use of berksfile: with cookbooks: if you wish as well. Berkshelf-managed cookbooks will be included in the validation of roles, environments and run lists as well. You may simply use the YAML snippet

berksfile:

which produces the command

berks upload -b ./Berksfile

or you may use additional options like

berksfile:
    path: '/Users/mray/ws/lab-repo/Berksfile'
    options: '--skip_syntax_check --config some_config.json'

which produces the output

berks upload --skip_syntax_check --config some_config.json -b /Users/mray/ws/lab-repo/Berksfile

Environments

The environments section of the manifest currently supports knife environment from file FOO where FOO is the name of the environment file ending in .rb or .json in the environments directory. You may also pass a wildcard as an entry to load all matching environments (ie. "*" or prod*"). Validation is done to ensure the filename matches the environment and that any cookbooks referenced are listed in the manifest. The example YAML snippet

environments:
- development:
- qa:
- "prod*":

assuming the production environment exists, produces the knife commands

knife environment from file development.rb qa.rb production.rb

Roles

The roles section of the manifest currently supports knife role from file FOO where FOO is the name of the role file ending in .rb or .json in the roles directory. You may also pass a wildcard as an entry to load all matching roles (ie. "*" or data*"). Validation is done to ensure the filename matches the role name and that any cookbooks or roles referenced are listed in the manifest. The example YAML snippet

roles:
- base:
- "data*":
- iisserver:
- monitoring:
- webserver:

assuming the database1.json and database2.json roles exist, this produces the knife commands

knife role from file base.rb database1.json database2.json iisserver.rb monitoring.rb webserver.rb

Data Bags

The data bags section of the manifest currently creates the data bags listed with knife data bag create FOO where FOO is the name of the data bag. Individual items may be added to the data bag as part of a JSON or YAML sequence, the assumption is made that they .json files and in the proper data_bags/FOO directory. You may also pass a wildcard as an entry to load all matching data bags (ie. "*" or "data*"). Encrypted data bags are supported by using the secret: secret_key_filename attribute. Validation is done to ensure the JSON is properly formatted, the id matches and any secret key files are in the correct locations. Assuming the presence of dataA.json and dataB.json in the data_bags/data directory, the YAML snippet

data bags:
- users:
    items:
    - alice
    - bob
    - chuck
- data:
    items:
    - "*"
- passwords:
    secret: secret_key_filename
    items:
    - mysql
    - rabbitmq

produces the knife commands

knife data bag create users
knife data bag from file users alice.json bob.json chuck.json
knife data bag create data
knife data bag from file data dataA.json dataB.json
knife data bag create passwords
knife data bag from file passwords mysql.json rabbitmq.json --secret-file secret_key_filename

Nodes

The nodes section of the manifest bootstraps a node for each entry given a hostname or a provider with a count. Windows nodes need to specify either windows_winrm or windows_ssh depending on the protocol used, followed by the name of the node(s). Each node may have a run_list and options. The run_list may be space or comma-delimited. Validation is performed on the run_list components to ensure that only cookbooks and roles listed in the manifest are used. Validation on the options ensures that any environments referenced are also listed. You may specify multiple nodes to have the same configuration by listing them separated by a space. If you want to give your nodes names, simply pass -N NAME{{n}} or --node-name NAME{{n}} and the {{n}} will be substituted by a number. The example YAML snippet

nodes:
- serverA:
    run_list: role[base]
    options: -i ~/.ssh/mray.pem -x user --sudo
- serverB serverC:
    run_list: role[base]
    options: -i ~/.ssh/mray.pem -x user --sudo -E production
- rackspace 3:
    run_list: recipe[mysql],role[monitoring]
    options: --image 49 --flavor 2 -N db{{n}}
- windows_winrm winboxA:
    run_list: role[base],role[iisserver]
    options: -x Administrator -P 'super_secret_password'
- windows_ssh winboxB winboxC:
    run_list: role[base],role[iisserver]
    options: -x Administrator -P 'super_secret_password'

produces the knife commands

knife bootstrap serverA -i ~/.ssh/mray.pem -x user --sudo -r 'role[base]'
knife bootstrap serverB -i ~/.ssh/mray.pem -x user --sudo -E production -r 'role[base]'
knife bootstrap serverC -i ~/.ssh/mray.pem -x user --sudo -E production -r 'role[base]'
knife rackspace server create --image 49 --flavor 2 --node-name db1.example.com -r 'recipe[mysql],role[monitoring]'
knife rackspace server create --image 49 --flavor 2 --node-name db2.example.com -r 'recipe[mysql],role[monitoring]'
knife rackspace server create --image 49 --flavor 2 --node-name db3.example.com -r 'recipe[mysql],role[monitoring]'
knife bootstrap windows winrm winboxA -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'
knife bootstrap windows ssh winboxB -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'
knife bootstrap windows ssh winboxC -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'

Providers

The following knife plugins are currently supported as providers: bluebox, clodo, cs, digital_ocean, ec2, gandi, google, hp, joyent, kvm, linode, lxc, openstack, rackspace, slicehost, terremark, vagrant, voxel and vsphere.

Bulk node creation

You may also use the --parallel flag from the command line, allowing provider commands to run simultaneously for faster deployment. Using --parallel with the following block and the -N webserver{{n}}:

nodes:
- ec2 3:
  - role[webserver]
  - -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -N webserver{{n}}

produces the following

seq 3 | parallel -j 0 -v "knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -N webserver{} -r 'role[base],role[tc],role[users]'"

which generates nodes named "webserver1", "webserver2" and "webserver3".

Clusters

Clusters are not a type supported by Chef, this is a logical construct added by Spiceweasel to enable managing sets of infrastructure together. The clusters section is a special case of nodes, where each member of the named cluster in the manifest will be put in the same Environment to ensure that the entire cluster (every node in the Environment) may be created and destroyed in sync. The node syntax is the same as that under nodes, the only addition is the cluster name.

clusters:
- amazon:
  - ec2 1:
      run_list: role[mysql]
      options: -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-8af0f326 -f m1.medium
  - ec2 3:
      run_list: role[webserver] recipe[mysql::client]
      options: -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small

produces the knife commands

knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-8af0f326 -f m1.medium -E amazon -r 'role[mysql]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'

Another use of clusters is with the --cluster-file option, which will allow the use of a different file to define the members of the cluster. If there are any nodes or clusters defined in the primary manifest file, they will be removed and the content of the --cluster-file will be used instead. This allows you to switch the target destination of infrastructure by picking different --cluster-file endpoints.

options

The options section provides the ability to pass global options to all nodes being created.

options: -i ~/.ssh/mray.pem
nodes:
- serverA:
    run_list: role[base]
    options: -x user --sudo
clusters:
- amazon:
  - ec2 1:
      run_list: role[mysql]
      options: -S mray -x ubuntu -G default -I ami-8af0f326 -f m1.medium

produces the knife commands

knife bootstrap serverA -x user --sudo -r -i ~/.ssh/mray.pem 'role[base]'
knife ec2 server create -S mray -x ubuntu -G default -I ami-8af0f326 -f m1.medium -E amazon -i ~/.ssh/mray.pem -r 'role[mysql]'

knife

The knife section allows you to run arbitrary knife commands after you have deployed the infrastructure specified in the rest of the manifest. Validation is done to ensure that the knife commands called are installed on the system. The example YAML snippet

knife:
- ssh:
  - "'role:monitoring' 'sudo chef-client' -x user"
- rackspace server delete:
  - -y --node-name db3 --purge
- vsphere:
  - vm clone --bootstrap --template 'abc' my-new-webserver1
  - vm clone --bootstrap --template 'def' my-new-webserver2
- vsphere vm clone:
  - --bootstrap --template 'ghi' my-new-webserver3

assumes the knife-rackspace and knife-vsphere plugins are installed and produces the knife commands

knife ssh 'role:monitoring' 'sudo chef-client' -x user
knife rackspace server delete -y --node-name db3 --purge
knife vsphere vm clone --bootstrap --template 'abc' my-new-webserver1
knife vsphere vm clone --bootstrap --template 'def' my-new-webserver2
knife vsphere vm clone --bootstrap --template 'ghi' my-new-webserver3

Extract

Spiceweasel may be used to generate knife commands or Spiceweasel manifests in JSON or YAML from an existing Chef repository.

spiceweasel --extractlocal

When run in your Chef repository generates the knife commands required to upload all the existing infrastructure that is found in the cookbooks, roles, environments and data bags directories with validation.

spiceweasel --extractjson

When run in your Chef repository generates JSON-formatted output that may be captured and used as your Spiceweasel manifest file.

spiceweasel --extractyaml

When run in your Chef repository generates YAML-formatted output that may be captured and used as your Spiceweasel manifest file.

Usage

To parse a Spiceweasel manifest, run the following from your Chef repository directory:

spiceweasel path/to/infrastructure.yml

or to extract infrastructure

spiceweasel --extractlocal

This will generate the knife commands to build the described infrastructure. Infrastructure manifest files must end in either .json or .yml.

OPTIONS

--bulkdelete

When using the delete or rebuild commands, whether or not to attempt to delete all nodes managed by a provider. The assumption is that if Spiceweasel manages all the nodes, it is safe to delete them all.

-c/--knifeconfig

Specify a knife.rb configuration file to use with the knife commands.

--chef-client

This will generate the knife commands to run the chef-client on each of the nodes in the manifest. The nodes refreshed will be matched by node name, environment for clusters and by cloud provider run list otherwise.

knife ssh 'name:*' 'sudo chef-client' knife ssh 'role:base and recipe:apt::cacher-ng' 'uptime' knife ssh "role:web"

You may also use -a or --attribute to specify an attribute to use with knife ssh.

--cluster-file

Specify the file to use to override the nodes and clusters from the primary manifest file. This allows you to switch the target destination of infrastructure by picking different --cluster-file endpoints.

--debug

This provides verbose debugging messages.

-d/--delete

The delete option will generate the knife commands to delete the infrastructure described in the manifest. This includes each cookbook, environment, role, data bag and node listed. Node deletion will specify individual nodes and their clients, and attempt to pass the list of nodes to the cloud provider for deletion, and finish with knife node bulk delete. If you are mixing individual nodes with cloud provider nodes it is possible that nodes may be missed from cloud provider deletion and you should double-check (ie. knife ec2 server list). Some knife plugins do not use the node name as the ID, so they may fail at deletion (knife-rackspace for example).

-e/--execute

The execute option will directly execute the knife commands, creating (or deleting or rebuilding) the infrastructure described in the manifest.

--extractlocal

When run in a chef repository, this will print the knife commands to be run.

--extractjson

When run in a chef repository, this will print a new JSON manifest that may be used as input to spiceweasel.

--extractyaml

When run in a chef repository, this will print a new YAML manifest that may be used as input to spiceweasel.

-h/--help

Print the currently-supported usage options for spiceweasel.

--node-only

Loads from JSON or creates nodes on the server without bootstrapping, useful for pre-creating nodes. If you specify a run list with the node, it will override any run list specified within the JSON file.

--novalidation

Disable validation ensuring existence of the cookbooks, environments, roles, data bags and nodes in infrastructure file.

--parallel

Use the GNU 'parallel' command to execute 'knife VENDOR server create' commands that may be run simultaneously.

-r/--rebuild

The rebuild option will generate the knife commands to delete and recreate the infrastructure described in the manifest. This includes each cookbook, environment, role, data bag and node listed.

--siteinstall

Use the 'install' command with 'knife cookbook site' instead of the default 'download'.

-v/--version

Print the version of spiceweasel currently installed.

Testing

Spiceweasel uses RSpec for testing. You should run the following before commiting.

$ rspec test

License and Author

Author	Matt Ray ([e-mail])
Copyright	Copyright (c) 2011-2014, Chef Software, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.