humilis

humilis

|Build Status| |PyPI|

Helps you deploy AWS infrastructure with Cloudformation <https://aws.amazon.com/cloudformation/>__.

This project is originally based on the cumulus <https://github.com/cotdsa/cumulus>__ project. See CUMULUS_LICENSE <https://github.com/humilis/humilis/blob/master/CUMULUS_LICENSE>__ for license information.

Installation

Install the AWS CLI <https://aws.amazon.com/cli/>__:

::

pip install awscli

Configure the AWS CLI:

::

aws configure

humilis will use whatever credentials you introduced when configuring your AWS CLI installation.

You can now install the latest “stable” version of humilis:

::

pip install humilis

or the development version if you prefer that:

::

pip install git+https://github.com/humilis/humilis

After installation you need to configure humilis. To configure globally for your system:

::

humilis configure

The command above will store and read the configuration options from ~/.humilis.ini. You can also store the configuration in a .humilis.ini file stored in your current working directory by using:

::

humilis configure --local

humilis will always read the configuration first from a .humilis.ini file under your current work directory. If it is not found then it will read it from your system global config file ~/.humilis.

Development environment

Assuming you have virtualenv <https://virtualenv.readthedocs.org/en/latest/>__ installed:

::

make develop

. .env/bin/activate

Testing

At the moment, most tests are integration tests with the AWS SDK. This means that you will need to set up your system <http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html>__ to access AWS resources if you want to run the test suite.

::

py.test tests

Quickstart

Define your infrastructure environment following the examples in the examples directory <https://github.com/humilis/humilis/tree/master/examples>__. Then to create the environment:

::

humilis create examples/humilis-firehose.yaml

To update the environment after it has been deployed:

::

humilis update examples/humilis-firehose.yaml

And to delete it:

::

humilis delete examples/humilis-firehose.yaml

Humilis environments

A humilis environment is just a collection of cloudformation stacks that are required for an application. Instead of having a monolytic CF template for your complete application, humilis allows you to define infrastructure layers that are combined into an environment. Each humilis layer translates exactly into one CF template (therefore into one CF stack after the layer is deployed).

Breaking a complex infrastructure environment into smaller layers has at least two obvious advantages:

• Easier to maintain. It’s easier to maintain a simple layer that contains just a bunch of CF resources <http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html>__ than serve a well-defined purpose.

• Easier to reuse. You should strive to define your infrastructure layers in such a way that you can reuse them across various environments. For instance, many projects may require a base layer that defines a VPC, a few subnets, a gateway and some routing tables, and maybe a (managed) NAT. You can define a humilis layer with those resources and have a set of layer parameters (e.g. the VPC CIDR) that will allow you to easily reuse it across environments.

Environment anatomy

An environment definition file is a yaml <https://en.wikipedia.org/wiki/YAML>__ document that specifies the list of layers that form your enviroment. The file should be named as your environment. That is, for environment my-app-environment the environment description file should be called my-app-environment.yaml. The contents of the environment definition should be organized as follows:

::

---

my-app-environment: description: A description of what this environment is for layers: # The layers that you environment requires. They will be deployed in the # same order as you list them. Note that you can also pass parameters # to a layer (more on that later). - {layer: name_of_first_layer, layer_param: layer_value} - {layer: name_of_second_layer} - {layer: name_of_third_layer}

Layer anatomy

Anything associated to a given layer must be stored in a directory with the same name as the layer, within the same directory where the environment definition file is located. If we consider the my-app-environment environment we used above then your directory tree should look like this:

::

. ├── my-app-environment.yaml ├── name_of_first_layer │   ├── meta.yaml │   └── resources.yaml ├── name_of_second_layer │   ├── meta.json │   └── meta.yaml └── name_of_third_layer ├── resources.json.j2 └── resources.yaml.j2

A layer must contain at least two files:

• meta.yaml: Meta information about the layer such as a description, and layer parameters.
• resources.yaml: Basically a CF template with the resources that the layer contains.

Those two files can also be in .json format (meta.json and resources.json). Or you can add the extension .j2 if you want the files to be pre-processed with the Jinja2 <http://jinja.pocoo.org/>__ template compiler.

Below an example of how a layer meta.yaml may look like:

::

---

meta: description: Creates a VPC, that's it parameters: vpc_cidr: description: The CIDR block of the VPC value: 10.0.0.0/16

Above we declare only one layer parameter: vpc_cidr. humilis will make pass that parameter to Jinja2 when compiling any template contained in the layer. So the resources.yaml.j2 for that same layer may look like this:

::

---

resources: VPC: Type: "AWS::EC2::VPC" Properties: CidrBlock: {{ vpc_cidr }}

References

You can use references in your meta.yaml files to refer to thing other than resources within the same layer (to refer to resources within a layer you can simply use Cloudformation’s Ref <http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-ref.html>__ or GetAtt <http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html>__ functions). Humilis references are used by setting the value of a layer parameter to a dict that has a ref key. Below an a meta.yaml that refers to a resource (with a logical name VPC) that is contained in another layer (called vpc_layer):

::

---

meta: description: Creates an EC2 instance in the vpc created by the vpc layer dependencies: - vpc parameters: vpc: description: Physical ID of the VPC where the instance will be created value: ref: parser: layer parameters: layer_name: vpc_layer resource_name: VPC

Every reference must have a parser key that identifies the parser that should be used to parse the reference. There are also two optional keys:

• parameters: allows you to pass parameters to the reference parser. You can pass either named parameters (as a dict) or positional arguments (as a list).

• priority: the parsing priority. Parameters with a lower value in priority will be parsed before parameters with a higher value. This allows some reference parsers to refer internally to other parameters within the same layer. For example, the lambda parser, when parsing templated lambda code, it uses previously parsed layer parameters as template parameters.

More information on the reference parsers that are bundled with humilis below.

Available reference parsers

layer_resource references

``layer_resource`` references allow you to refer to the physical ID of a
resource that is part of another layer.

**Parameters**:

-  ``layer_name``: The name of the layer you are referring to
-  ``resource_name``: The logical name of the layer resource

**Example**:

Consider the following environment definition:

::

   ---
   my-environment:
       description:
           Creates a VPC with a NAT in the public subnet
       layers:
           - {layer: vpc}
           - {layer: nat}

Obviously the ``nat`` layer that takes care of deploying the NAT in the
public subnet will need to know the physical ID of that subnet. You
achieve this by declaring a ``layer_resource`` reference in the
``meta.yaml`` for the ``nat`` layer:

::

   ---
   meta:
       description:
           Creates a managed NAT in the public subnet of the NAT layer
       parameters:
           subnet_id:
               description:
                   The physical ID of the subnet where the NAT will be placed
               value:
                   ref:
                       parser: layer_resource
                       parameters:
                           layer_name: vpc
                           # The logical name of the subnet in the vpc layer
                           resource_name: PublicSubnet

When parsing ``meta.yaml`` humilis will replace this:

::

   ref:
       parser: layer_resource
       parameters:
           layer_name: vpc
           # The logical name of the subnet in the vpc layer
           resource_name: PublicSubnet

with the physical ID you need (something like ``subnet-bafa90cd``). You
can then use this physical ID in the ``resources.yaml.j2`` section of
the ``nat`` layer:

::

   {# Pseudo-content of layers/nat/resources.yaml.j2 #}
   resources:
       {# An Elastic IP reservation that will be associated to the NAT #}
       NatEip:
         Type: 'AWS::EC2::EIP'
         Properties: {}
       {# Custom resource deploying the NAT #}
       NatGateway:
         Type: 'Custom::NatGateway',
         Properties:
           {# The ARN of the Lambda function backing the custom resource #}
           ServiceToken: 'arn:aws:lambda:eu-west-1:XXXX:function:CreateNatGateway'
           {# Here we use the subnet_id reference defined in meta.yaml #}
           SubnetId: {{subnet_id}}
           AllocationId:
               Ref: NatEip

``environment_resource`` references

environment_output references allow you to refer to resources that belong to other humilis environments.

Parameters:

• environment_name: The name of the environment you are referring to
• layer_name: The name of the layer you are referring to
• resource_name: The logical name of the layer resource

layer_output references

``layer_output`` references allow you to refer to outputs produced by
another layer.

**Parameters**:

-  ``layer_name``: The name of the layer you are referring to
-  ``output_name``: The logical name of the output parameter

In general you should prefer using ``layer_output`` references over
``layer_resource`` references. The output parameters produced by a layer
define an informal *layer interface* that is more likely to remain
constant than the logical names of resources within a layer.

``boto3`` references
~~~~~~~~~~~~~~~~~~~~

``boto3`` references define arbitrary calls to
`boto3facade <https://github.com/InnovativeTravel/boto3facade>`__. The
latter is just a simpler facade interface on top of
`boto3 <https://github.com/boto/boto3>`__.

**Parameters**:

-  ``service``: The AWS service, e.g. \ ``ec2`` or ``cloudformation``.
   Note that only only AWS services that have a facade in
   `boto3facade <https://github.com/InnovativeTravel/boto3facade>`__ are
   supported.
-  ``call``: The corresponding facade method,
   e.g. \ ``get_ami_by_name``. The value of this parameter must be a
   dictionary with a ``method`` key (the name of the facade method to
   invoke) and an optional ``args`` key (the parameters to pass to the
   facade method). Best to look at the example below to understand how
   this works.
-  ``output_attribute``: Optional. If provided the reference parser will
   return the value of this attribute from the object returned by the
   facade method.

Below an example of a layer that uses a ``boto3`` reference:

::

   ---
   meta:
       description:
           Creates an EC2 instance using a named AMI
       # More stuff omitted for brevity
       ami:
           description: The AMI to use when launching the EC2 instance
           value:
               ref:
                   parser: boto3
                   parameters:
                       service: ec2
                       call:
                           method: get_ami_by_name
                           args:
                               - test-ami
                       output_attribute: id

``humilis`` will parse the reference using this code:

::

   # Import the Ec2 facade
   from boto3facade.ec2 import Ec2

   # Create a facade object
   ec2_facade = Ec2()

   # Make the call
   ami = ec2_facade.get_ami_by_name('test-ami')

   # Extract the requested attribute
   ref_value = ami.id

``file`` references
~~~~~~~~~~~~~~~~~~~

``file`` references allow you to refer to a local file. The file will be
uploaded to S3 and the reference will evaluate to the corresponding S3
path.

**Parameters**:

-  ``path``: The path to the file, relative to the layer root directory.

``lambda`` references
~~~~~~~~~~~~~~~~~~~~~

``lambda`` references allow you to refer to some Python code in your
local machine. If your code follows some simple conventions ``humilis``
will take care of building a `deployment
package <http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html>`__
for you, uploading it to S3, and the reference will evaluate to the S3
path of the deployment package.

**Parameters**:

-  ``path``: Path to either a completely self-contained ``.py`` file, or
   to the root directory of your lambda code. In the latter case your
   code needs to follow some simple conventions for this to work. More
   information below.

-  ``dependencies``: A list of dependencies to be included in the Lambda
   deployment package. Dependencies may be either pip installable
   packages, or paths to local Python packages or modules, or paths to
   local ``requirements`` files.

**Example**:

::

   ref:
       parser: lambda
       parameters:
           # Path to the root directory containing your lambda code
           path: dummy_function
           dependencies:
               # The Lambda code requires Pypi's pyyaml
               - pyyaml
               # It also requires a local package in this path
               - mycode/mypkgdir
               # And this local module
               - mycode/mymodule.py

which will evaluate to a S3 path such as:

::

   s3://[bucket_name]/[environment_name]/[stage_name]/[func_name]-[commithash].zip

**Code conventions**:

Following the example above, the contents of the layer responsible of
deploying the ``dummy_function`` lambda may look like this:

::

   .
   ├── dummy_function
   │   ├── dummy_function.py
   │   └── setup.py
   ├── meta.yaml
   ├── outputs.yaml.j2
   └── resources.yaml.j2

Basically all your code needs to be included under directory
``dummy_function``. In this case there is only one file:
``dummy_function.py``. External dependencies need to be specified in
your ``setup.py``.

``secret`` references
~~~~~~~~~~~~~~~~~~~~~

``secret`` references retrieve a secret using Python’s
`keyring <https://pypi.python.org/pypi/keyring>`__ module.

**Parameters**:

-  ``service``: The name of the service the secret is associated to.

-  ``key``: The key (e.g. the username) that identifies the secret.

**Example**:

::

   ref:
       parser: secret
       parameters: {"service": "mysqldb", "key": "adminuser"}

Custom Jinja2 filters
---------------------

Humilis defines the following `custom Jinja2
filters <http://jinja.pocoo.org/docs/dev/templates/#filters>`__:

-  ``uuid``: A random UUID. Example: ``{{''|uuid}}``.
-  ``password(length=8)``: A random password. Example:
   ``{{10|password}}``.

.. |Build Status| image:: https://travis-ci.org/humilis/humilis.svg?branch=master
   :target: https://travis-ci.org/humilis/humilis
.. |PyPI| image:: https://img.shields.io/pypi/v/humilis.svg?style=flat
   :target: https://pypi.python.org/pypi/humilis