@znemz/cfn-include

cfn-include

cfn-include is a preprocessor for CloudFormation templates which extends CloudFormation's intrinsic functions. For example, Fn::Include provides a convenient way to include files, which can be local, a URL or on an S3 bucket (with proper IAM authentication if necessary). It supports both JSON and YAML as input and output format. CloudFormation's tag syntax for YAML (e.g. !GetAtt) is supported as well.

cfn-include tries to be minimally invasive, meaning that the template will still look and feel like an ordinary CloudFormation template. This is what sets cfn-include apart from other CloudFormation preprocessors such as CFNDSL, StackFormation and AWSBoxen. There is no need to use a scripting language or adjust to new syntax. Check them out though, they might be a better fit for you.

Functions

• Fn::Include
• Fn::Flatten
• Fn::GetEnv
• Fn::Length
• Fn::Map
• Fn::Merge
• Fn::DeepMerge
• Fn::Outputs
• Fn::Sequence
• Fn::Stringify
• Fn::UpperCamelCase and Fn::LowerCamelCase

Tag-based syntax is available in YAML templates. For example,Fn::Include becomes !Include.

Installation

You can either install cfn-include or use a web service to compile templates.

npm install --global @znemz/cfn-include

The web service can be called with your favorite CLI tool such as curl.

curl https://api.netcubed.de/latest/template -XPOST -d @template.json

Synopsis

CLI

cfn-include <path> [options]

• path

  location of template. Either path to a local file, URL or file on an S3 bucket (e.g. s3://bucket-name/example.template)

Options:

• -m, --minimize minimize JSON output [false]
• --metadata add build metadata to output [false]
• -t, --validate validate compiled template [false]
• -y, --yaml output yaml instead of json [false]
• --bucket bucket name required for templates larger than 50k
• --prefix prefix for templates uploaded to the bucket ['cfn-include']
• --version print version and exit
• --context template full path. only utilized for stdin when the template is piped to this script example: cat examples/base.template | ./bin/cli.js --context examples/base.template
• --doEnv pre-process env vars and inject into templates as they are processed looks for $KEY or ${KEY} matches

cfn-include also accepts a template passed from stdin

cat mytemplate.yml | cfn-include

Example

Mappings:
  Region2AMI: !Include https://api.netcubed.de/latest/ami/lookup?platform=amzn2
Resources:
  Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !FindInMap [Region2AMI, !Ref AWS::Region, AMI]
      UserData:
        Fn::Base64:
          Fn::Sub: !Include { type: string, location: userdata.sh }

This is what the userdata.sh looks like:

#!/bin/bash
/opt/aws/bin/cfn-init -s ${AWS::StackId} -r MyInstance --region ${AWS::Region}

cfn-include synopsis.json > output.template
# you can also compile remote files
cfn-include https://raw.githubusercontent.com/nmccready/cfn-include/master/examples/synopsis.json > output.template

The output will be something like this:

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Mappings": {
    "Region2AMI": {
      "Metadata": {
        "Name": "amzn-ami-hvm-2016.09.0.20161028-x86_64-gp2",
        "Owner": "amazon",
        "CreationDate": "2016-10-29T00:49:47.000Z"
      },
      "us-east-2": {
        "AMI": "ami-58277d3d"
      },
      // ...
  } },
  "Resources": {
    "Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": {
          "FindInMap": [ "Region2AMI", { "Ref": "AWS::Region" }, "AMI" ]
        },
        "UserData": {
          "Fn::Base64": {
            "Fn::Sub": {
              "Fn::Join": ["", [
                  "#!/bin/bash\n",
                  "\"/opt/aws/bin/cfn-init -s ${AWS::StackId} -r MyInstance --region ${AWS::Region}\n",
                  ""
] ] } } } } } } }

Fn::Include

Place Fn::Include anywhere in the template and it will be replaced by the contents it is referring to. The function accepts an object. Parameters are:

• location: The location to the file can be relative or absolute. A relative location is interpreted relative to the template. Included files can in turn include more files, i.e. recursion is supported.
• type (optional): either json, string or api. Defaults to json. string will include the file literally which is useful in combination with Fn::Sub. api will call any AWS API and return the response which can be included in the template. Choose json for both JSON and YAML files. The literal type is deprecated and uses the infamous Fn::Join syntax.
• context (optional, deprecated): If type is literal a context object with variables can be provided. The object can contain plain values or references to parameters or resources in the CloudFormation template (e.g. { "Ref": "StackId" }). Use Mustache like syntax in the file. This option is deprecated in favor of the Fn::Sub syntax (see examples below).
• query (optional): If type is json a JMESPath query can be provided. The file to include is then queried using the value as a JMESPath expression.

Only applicable if type is api:

• service: Service to call (see AWSJavaScriptSDK, case sensitive, e.g. EC2, CloudFormation)
• action: Action to call (case sensitive, e.g. updateStack, describeRegions)
• parameters (optional): Parameters passed to action (e.g. { StackName: "MyStack" })
• region (optional): Either AWS_DEFAULT_REGION or this parameter have to be set which specifies the region where the API call is made.

You can also use a plain string if you want the default behavior, which is simply including a JSON file.

Examples

Include a file from a URL

!Include https://example.com/include.json

// equivalent to

Fn::Include:
  type: json
  location: https://example.com/include.json

Include a file from an S3 bucket. Authentication is handled by aws-sdk. See Setting AWS Credentials for details.

!Include s3://bucket-name/include1.json

Include a file in the same folder

!Include include.json

Include a file literally and make use of Fn::Sub:

Fn::Sub:
  Fn::Include:
    type: string
    location: https://example.com/userdata.txt

Include an AWS API response, e.g. loop through all regions and return the image id of a specific AMI:

Fn::Merge:
  Fn::Map:
    - Fn::Include:
        action: describeRegions
        query: "Regions[*].RegionName[]"
        service: EC2
        type: api
    - _:
        AMI:
          Fn::Include:
            action: describeImages
            parameters:
              Filters:
                - Name: manifest-location
                  Values:
                    - amazon/amzn-ami-hvm-2016.03.3.x86_64-gp2
            query: "Images[*].ImageId | [0]"
            region: _
            service: EC2
            type: api

Output as JSON:

{ "ap-south-1": { "AMI": "ami-ffbdd790" },
  "eu-west-1": {"AMI": "ami-f9dd458a" },
  "ap-southeast-1": { "AMI": "ami-a59b49c6" },
  ...
}

Fn::Map

Fn::Map is the equivalent of the JavaScript map() function allowing for the transformation of an input array to an output array. By default the string _ is used as the variable in the map function. A custom variable can be provided as a second parameter, see Fn::Flatten for an example. If a custom variable is used, the variable will also be replaced if found in the object key, see Fn::Merge for an example.

Fn::Map:
  - [80, 443]
  - CidrIp: 0.0.0.0/0
    FromPort: _
    ToPort: _
    IpProtocol: tcp

[
  {
    "CidrIp": "0.0.0.0/0",
    "FromPort": "80",
    "ToPort": "80"
  },
  {
    "CidrIp": "0.0.0.0/0",
    "FromPort": "443",
    "ToPort": "443"
  }
]

Custom variables can be specified as a single value, of as a list of up to three values. If a list is specified, the second variable is used as index and the third (if present) as size.

Fn::Map:
  - !Sequence [A, C]
  - [NET, INDEX, N]
  - Subnet${NET}:
      Type: 'AWS::EC2::Subnet'
      Properties:
        CidrBlock: !Select [INDEX, !Cidr [MyCIDR, N, 8]]

[{
  "SubnetA": {
    "Type": "AWS::EC2::Subnet",
    "Properties": {
      "CidrBlock": { "Fn::Select": [ 0, { "Fn::Cidr": [ "MyCIDR", 3, 8 ] } ] }
    }
  }
}, {
  "SubnetB": {
    "Type": "AWS::EC2::Subnet",
    "Properties": {
      "CidrBlock": { "Fn::Select": [ 1, { "Fn::Cidr": [ "MyCIDR", 3, 8 ] } ] }
    }
  }
}, {
  "SubnetC": {
    "Type": "AWS::EC2::Subnet",
    "Properties": {
      "CidrBlock": { "Fn::Select": [ 2, { "Fn::Cidr": [ "MyCIDR", 3, 8 ] } ] }
    }
  }
}]

Fn::Flatten

This function flattens an array a single level. This is useful for flattening out nested Fn::Map calls.

SecurityGroupIngress:
  Fn::Flatten:
    Fn::Map:
      - [80, 443]
      - $
      - Fn::Map:
          - [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16]
          - CidrIp: _
            FromPort: $
            ToPort: $
            IpProtocol: tcp

Results in:

{
  "SecurityGroupIngress": [
    {
      "CidrIp": "10.0.0.0/8",
      "FromPort": "80",
      "ToPort": "80",
      "IpProtocol": "tcp"
    },
    {
      "CidrIp": "172.16.0.0/12",
      "FromPort": "80",
      "ToPort": "80",
      "IpProtocol": "tcp"
    },
    {
      "CidrIp": "192.168.0.0/16",
      "FromPort": "80",
      "ToPort": "80",
      "IpProtocol": "tcp"
    },
    {
      "CidrIp": "10.0.0.0/8",
      "FromPort": "443",
      "ToPort": "443",
      "IpProtocol": "tcp"
    },
    {
      "CidrIp": "172.16.0.0/12",
      "FromPort": "443",
      "ToPort": "443",
      "IpProtocol": "tcp"
    },
    {
      "CidrIp": "192.168.0.0/16",
      "FromPort": "443",
      "ToPort": "443",
      "IpProtocol": "tcp"
    }
  ]
}

Fn::GetEnv

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !GetEnv [BUCKET_NAME, !Ref AWS::NoValue]

The second argument is optional and provides the default value and will be used of the environmental variable is not defined. If the second argument is omitted !GetEnv BUCKET_NAME and the environmental variable is not defined then the compilation will fail.

Fn::Length

Fn::Length returns the length of a list or expanded section.

Fn::Merge

Fn::Merge will merge an array of objects into a single object. See lodash / merge for details on its behavior. This function is useful if you want to add functionality to an existing template if you want to merge objects of your template that have been created with Fn::Map.

Fn::Merge accepts a list of objects that will be merged together. You can use other cfn-include functions such as Fn::Include to pull in template from remote locations such as S3 buckets.

Fn::Merge:
  - !Include s3://my-templates/my-template.json

  - !Include s3://my-templates/my-other-template.json

  - Parameters:
      MyCustomParameter:
        Type: String

    Resources:
      MyBucket:
        Type: AWS::S3::Bucket

Fn::DeepMerge

Fn::DeepMerge will deeply merge an array of objects and arrays into a single object. See deepmerge for details on its behavior. This function is useful if you want to add functionality to an existing template if you want to merge objects of your template that have been created with Fn::Map.

Fn::DeepMerge accepts a list of objects that will be merged together. You can use other cfn-include functions such as Fn::Include to pull in template from remote locations such as S3 buckets.

To understand it better besides the below example refer to this test. Note that all arrays are concatenated.

Why does Fn::Merge still exist? Answer: Backwards compatibility for expected behavior.

Fn::DeepMerge:
  - !Include s3://my-templates/my-template.json

  - !Include s3://my-templates/my-other-template.json

  - Parameters:
      MyCustomParameter:
        Type: String

    Resources:
      MyBucket:
        Type: AWS::S3::Bucket

This snippet shows how multiple subnets can be created for each AZ and then merged with the rest of the template.

Resources:
  IAMUser:
    Type: AWS::IAM::User
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
  Fn::Merge:
    Fn::Map:
      - [A, B]
      - AZ
      - Subnet${AZ}:
          Type: AWS::EC2::Subnet

{
  "Resources": {
    "SubnetA": {
      "Type": "AWS::EC2::Subnet"
    },
    "SubnetB": {
      "Type": "AWS::EC2::Subnet"
    },
    "SG": {
      "Type": "AWS::EC2::SecurityGroup"
    }
  }
}

Fn::Sequence

Fn::Sequence generates a sequence of numbers of characters. You can specify the start, end and step.

!Sequence [1, 4]

# generates
[1, 2, 3 4]

!Sequence [1, 10, 2]

# generates
[1, 3, 5, 7, 9]

!Sequence [a, d]

# generates
[a, b, c, d]

Fn::Sequence can be used in combination with Fn::Map to generate complex objects:

Fn::Map:
  - !Sequence [a, c]
  - AZ
  - Subnet${AZ}:
      Type: AWS::EC2::Subnet

Fn::Stringify

Fn::Stringify will take the passed value and transform it to a JSON string. This is useful for parameters that require a JSON document as a string. Using this function, you can keep writing your configuration in YAML and let the function transform it into a JSON string.

Another useful application is the use of this function in a config file passed as --cli-input-json parameter.

# stack.config.yml

StackName: MyStack

TemplateBody:
  Fn::Stringify: !Include mytemplate.yml

Parameters:
  - ParameterKey: Foo
    ParameterValue: Bar

You can then simply run the following command to deploy a stack:

cfn-include stack.config.yml > stack.config.json
aws cloudformation create-stack --cli-input-json file://stack.config.json

Fn::UpperCamelCase

Name: !UpperCamelCase foo-bar # yields FooBar

Fn::Outputs

This helper transformation simplifies the definition of output variables and exports.

Outputs:
  Fn::Outputs:
    Version: !GetEnv [VERSION, "1.0.0"]
    BucketArn: ${Bucket.Arn}
    BucketPolicy:
      Condition: HasBucketPolicy
      Value: ${BucketPolicy}
    Subnets:
      - ${SubnetA},${SubnetB},${Provided}
      - Provided: ${SubnetC}

This will translate into:

Outputs:
  Version:
    Value: !Sub "1.0.0"
    Export:
      Name: !Sub ${AWS::StackName}:Version

  BucketArn:
    Value: !Sub ${Bucket.Arn}
    Export:
      Name: !Sub ${AWS::StackName}:BucketArn

  BucketPolicy:
    Value: !Sub ${BucketPolicy}
    Condition: HasBucketPolicy
    Export:
      Name: !Sub ${AWS::StackName}:BucketPolicy

  Subnets:
    Value:
      Fn::Sub:
        - ${SubnetA},${SubnetB},${Provided}
        - Provided: ${SubnetC}
    Export:
      Name: !Sub ${AWS::StackName}:Subnets

More Examples

See /examples for templates that call an API Gateway endpoint to collect AMI IDs for all regions. There is also a good amount of tests that might be helpful.

A common pattern is to process a template, validate it against the AWS validate-template API, minimize it and upload the result to S3. You can do this with a single line of code:

cfn-include example.template -t -m | aws s3 cp - s3://bucket-name/output.template

Proxy Support

cfn-include honors proxy settings defined in the https_proxy environmental variable. The module will attempt to load proxy-agent. Make sure proxy-agent is installed since it is not a dependency for this module.

Compatibility

Node.js versions 8 and up are supported both on Windows and Linux.

Web Service

curl https://api.netcubed.de/latest/template?[options] -XPOST -d @<path>

• path

  the contents of path will be POSTed to the web service. See man curl for details.

Options:

Options are query parameters.

• validate=false do not validate template [true]

To compile the synopsis run the following command.

curl -Ssf -XPOST https://api.netcubed.de/latest/template -d '{"Fn::Include":"https://raw.githubusercontent.com/nmccready/cfn-include/master/examples/synopsis.json"}' > output.template