linux_provision

Library for provisioning initial setup of Linux computer for Ruby/Rails development

Introduction

Why do we need virtualization in development?

• We want to have same environment for all developers, no matter on what platform they are working now.

• We are working on multiple projects on same computer unit. As a result, suddenly your computer has "hidden", hard-to-discover inter-project dependencies or different versions of the same library.

• We want to run Continuous Integration Server jobs that start services on same ports for different set of acceptance tests (isolated jobs).

• To overcome "It works on my machine!" syndrome - when development environment is different from production environment.

• Sometimes required software is not available on developer's platform. Example: 64-bit instant client for oracle was broken for almost two years on OSX >= 10.7.

• Development for PAAS, such as Heroku, Engine Yard etc. You can find and build virtualization that is pretty close to your platform.

We will take a look at how can we do provisioning for Vagrant and Docker. Both tools are built on top of VirtualBox.

Installing and configuring Vagrant

Vagrant is the wrapper around VirtualBox. It is a tool for managing virtual machines via simple to use command line interface. With it you can work in a clean environment based on a standard template - base box.

In order to use Vagrant you have to install these programs:

• VirtualBox. Download it from dedicated web site and install it as native program. You can use it in UI mode, but it's not required.

• Vagrant. Before it was distributed as ruby gem, now it's packaged as native application. Once installed, it will be accessible from command line as vagrant command.

You have to decide what linux image fits your needs. I our case we use Ubuntu 14.04 LTS 64-bit image - it is identified with "ubuntu/trusty64" key.

Download and install it:

vagrant box add ubuntu/trusty64 https://vagrantcloud.com/ubuntu/boxes/trusty64

Initialize it:

vagrant init ubuntu/trusty64

This command creates Vagrantfile file in the root of your project. Below is an example of such a file:

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "ubuntu/trusty64"
end

You can do various commands with vagrant tool. For example:

vagrant up        # starts up: creates and configures guest machine
vagrant suspend   # suspends the guest machine
vagrant halt      # shuts down the running machine
vagrant reload    # vagrant halt; vagrant up
vagrant destroy   # stops machine and destroys all related resources
vagrant provision # perform provisioning for machine
vagrant box remove ubuntu/trusty64 # removes a box from vagrant

You can package currently running VirtualBox environment into reusable box:

vagrant package --vagrantfile Vagrantfile --output linux_provision.box

After Vagrantfile is generated, you can start your base box:

vagrant up

Now you have a fully running virtual machine in VirtualBox. You can access it through vagrant ssh command:

vagrant ssh

or directly via ssh (use vagrant password for vagrant user and port 2222, this port is used as default by vagrant for ssh connections):

ssh vagrant@127.0.0.1 -p 2222

You can assign IP address for your linux box, e.g.:

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.network "private_network", ip: "22.22.22.22"
end

With this configuration you can access ssh on default port:

ssh vagrant@22.22.22.22

Your initial setup of linux box is completed now and ready to use.

Installing and configuring Docker

Docker helps you create and manage Linux containers - extremely lightweight VMs. Containers allow code to run in isolation from other containers. They safely share the machine's resources, all without the overhead of a hypervisor.

In order to use Docker you have to install these programs:

• VirtualBox.

• boot2docker. You need to install it only for non-Linux environment.

boot2docker is a lightweight Linux image made specifically to run Docker containers. It runs completely from RAM, weighs approximately 27 MB and boots in about 5 seconds.

We'll run the Docker client natively on OSX, but the Docker server will run inside our boot2docker VM. This also means that boot2docker, not OSX, is the Docker host.

This command will create boot2docker-vm virtual machine:

boot2docker init

Go to VirtualBox UI - new VM will be added.

Start it up:

boot2docker up

or shut it down:

boot2docker down

Upgrade Boot2docker VM image:

boot2docker stop
boot2docker download
boot2docker up

When docker daemon first started, it gives you recommendation about how to run docker client. It needs to know where docker is running, e.g.:

export DOCKER_HOST=tcp://192.168.59.103:2375

You have to setup it globally in .bash_profile file or specify it each time when docker client is started.

You can access boot2docker over ssh (user: docker, password: tcuser):

boot2docker ssh

Download the small base image named busybox:

docker pull busybox

Run and test docker as separate command:

docker run busybox echo "hello, linus!"

or interactively:

docker run -t -i busybox /bin/sh

Install and confige linux_provision gem

Both programs - Vagrant and Docker - have their own ways to serve provisioning. Vagrant is doing it with the help of provision attribute. Example with simple shell script:

Vagrant::Config.run do |config|
  config.vm.provision :shell, :path => "bootstrap.sh"
end

or with chef solo:

Vagrant::Config.run do |config|
  config.vm.provision :chef_solo do |chef|
    ...
  end
end

Docker also lets you do provisioning in form of RUN command:

# Dockerfile

RUN apt-get -y -q install postgresql-9.3

After multiple experiments with provisions both from Vagrant and Docker it was discovered that it is not convenient to use. It does not let you to easy install or uninstall separate packages. It's better to do it as set of independent scripts, separated completely from Docker and Vagrant.

linux_provision gem is the set of such shell scripts - they install various components like postgres server, rvm, ruby etc. with the help of thor or rake script. You can see other gems that are providing similar solutions: for Oracle Instant Client and for OSX.

In order to use gem add this line to your application's Gemfile:

gem 'linux_provision'

And then execute:

bundle

Before you can start using linux_provision gem within your project, you need to configure it. Do the following:

• Create configuration file (e.g. .linux_provision.json) in json format at the root of your project. It will define your environment:

{
  "node": {
   ...
  },

  "project": {
    "home": "#{node.home}/demo",
    "ruby_version": "1.9.3",
    "gemset": "linux_provision_demo"
  },

  "postgres": {
    "hostname": "localhost", "user": "postgres", "password": "postgres",
    "app_user": "pg_user", "app_password": "pg_password",
    "app_schemas": [ "my_project_test", "my_project_dev", "my_project_prod"]
  }
}

Variables defined in this file are used by underlying shell scripts provided by the gem.

In node section you describe destination computer where you want to install this provision.

In project section you keep project-related info, like project home, project gemset name and ruby version.

Last postgres section contains information about your postgres server.

• Provide execution script

Library itself if written in ruby, but for launching its code it's more convenient to use rake or thor tool. Here I provide thor script as an example:

# thor/linux_install.thor

$: << File.expand_path(File.dirname(__FILE__) + '/../lib')

require 'linux_provision'

class LinuxInstall < Thor
  @installer = LinuxProvision.new self, ".linux_provision.json"

  class << self
    attr_reader :installer
  end

  desc "general", "Installs general packages"
  def general
    invoke :prepare

    invoke :rvm
    invoke :ruby

    invoke :postgres
    invoke :mysql
  end
end

You can execute separate commands from script directly with invoke thor command. Below is fragment of such script:

#!/bin/sh

#######################################
[prepare]
# Updates linux core packages

sudo apt-get update

sudo apt-get install -y curl
sudo apt-get install -y g++
sudo apt-get install -y subversion
sudo apt-get install -y git

#######################################
[rvm]
# Installs rvm

curl -L https://get.rvm.io | bash

#sudo chown -R vagrant /opt/vagrant_ruby

#######################################
[ruby]
# Installs ruby

USER_HOME="#{node.home}"

source $USER_HOME/.rvm/scripts/rvm

rvm install ruby-1.9.3

You can add your own scripts (e.g. demo_scripts.sh):

class LinuxInstall < Thor
  @installer = LinuxProvision.new self, 
    ".linux_provision.json", 
    [File.expand_path("demo_scripts.sh", File.dirname(__FILE__))]
  ...
end

We defined 2 new commands in demo_script.sh:

#!/bin/sh

##############################
[project]
# Installs demo sinatra project

USER_HOME="#{node.home}"

APP_HOME="#{project.home}"

cd $APP_HOME

source $USER_HOME/.rvm/scripts/rvm

rvm use #{project.ruby_version}@#{project.gemset} --create

bundle

rake db:migrate


##############################
[rackup]
# Starts sinatra demo application

USER_HOME="#{node.home}"

APP_HOME="#{project.home}"

cd $APP_HOME

source $USER_HOME/.rvm/scripts/rvm

rvm use #{project.ruby_version}@#{project.gemset}

rackup

Demo application with Vagrant

For testing purposes we have created demo web application (in demo folder) based on sinatra framework.

First, we need to inform Vagrant about the location of this application within virtual machine:

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.synced_folder "./demo", "/home/vagrant/demo"
end

Second, we need to configure linux_provision gem to point to right domain/port and use correct user name/password:

{
  "node": {
    "domain": "22.22.22.22", # remote host, see "config.vm.synced_folder"
    "port": "22",            # default ssh port
    "user": "vagrant",       # vagrant user name
    "password": "vagrant",   #
    "home": "/home/vagrant", # vagrant user password
    "remote": true
  }
}

Start your base box:

vagrant up

Access linux box and find out this demo application's home:

ssh vagrant@22.22.22.22

pwd # /home/vagrant

ls # demo

cd demo

ls # content of demo folder

These commands from linux_provision gem will build your environment for the demo project (install rvm, ruby, postgres, postgres user and posters tables):

thor linux_install:prepare
thor linux_install:rvm
thor linux_install:ruby

thor linux_install:postgres

thor linux_install:postgres_create_user
thor linux_install:postgres_create_schemas

Initialize demo project and run sinatra application:

thor linux_install:project

thor linux_install:rackup

Now you can access application from your favorite browser:

open http://22.22.22.22:9292

Demo application with Docker

You need to do very similar steps as with Vagrant. The only difference is in linux_provision.json file you have to point to different host, port and user:

{
  "node": {
    "domain": "192.168.59.103", # remote host, see boot2docker ip
    "port": "42222",            # ssh port in docker
    "user": "vagrant",          # vagrant user name
    "password": "vagrant",      #
    "home": "/home/vagrant",    # vagrant user password
    "remote": true
  }
}

Our Dockerfile is responsible for the following base steps:

• Install Ubuntu 14.4.

• Install sshd (for enabling ssh).

• Create vagrant user (just to in-synch with Vagrant example).

• Reveal project home as /home/vagrant/demo.

• Expose port 9292 (our sinatra application).

Here is example:

FROM ubuntu:14.04

MAINTAINER Alexander Shvets "[e-mail]"

# 1. Update system
RUN sudo apt-get update
RUN sudo locale-gen en_US.UTF-8

# 2. Install sshd

RUN sudo apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN echo 'root:root' |chpasswd
RUN sed --in-place=.bak 's/without-password/yes/' /etc/ssh/sshd_config

EXPOSE 22

CMD /usr/sbin/sshd -D

# 3. Create vagrant user
RUN groupadd vagrant
RUN useradd -d /home/vagrant -g vagrant -m -s /bin/bash vagrant
RUN sudo sed -i '$a vagrant    ALL=(ALL) NOPASSWD: ALL' /etc/sudoers
RUN echo vagrant:vagrant | chpasswd
RUN sudo chown -R vagrant /home/vagrant

# 4. Prepare directories for the project

# Add project dir to docker

ADD . /home/vagrant/demo
WORKDIR /home/vagrant/demo

EXPOSE 9292

Build docker image and run it:

docker build -t demo demo
docker run -d -p 42222:22 -p 9292:9292 --name demo demo

As you can see, we map port 22 inside docker to port 42222 outside. It means that when we hit port 42222 with regular telnet tool, we'll hit service inside the docker.

You can access virtual machine via ssh:

ssh vagrant@192.168.59.103 -p 42222

Now you can do your provision - it's exactly the same as with Vagrant example:

thor linux_install:prepare
thor linux_install:rvm
thor linux_install:ruby

thor linux_install:postgres

thor linux_install:postgres_create_user
thor linux_install:postgres_create_schemas

thor linux_install:project

thor linux_install:rackup

After provisioning and starting server try to access your application from the browser:

open http://192.168.59.103:9292

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request