catops

CatOps

Highly trained cats for managing servers.

What is CatOps

CatOps is a very simple NoOps framework for deploying your own ChatOps bot.

Commands you wish to add to your CatOps implementation are added to a plugins folder in the same directory, and will then be automatically imported and callable using the function name.

Why CatOps

• NoOps.

  • Deploy, rewrite, and redeploy FaaS easily with no worrying about setting up and managing servers.
  • Only charged when CatOps is called.

• Codify common maintenance procedures.

  • Perform high level actions without intimate low level knowledge.
  • Prevent errors doing complicated but routine tasks.

• Unify documentation.

  • CatOps can act as a unified go-to location for help, merging/pooling all documentation into one place.

• Transparency.

  • Team members can see all actions taken by others in solving a problem. Organic learning.
  • No 'go-to' person for certain maintenance tasks.
  • Everyone aware of server changes. No-one surprised that the server is down if they see /meow restart server in the chat.
  • Spread knowledge; everyone becomes equally capable of solving problems.
  • Out of date help messages or documentation is more obvious to everyone.

• Context-aware suggestions, suggest actions and display help depending on context.

  • Docs/procedures/etc are useful, but can be too much to read through, hard to find, not up to date.
  • Reduce clutter when trying to figure out next actions.

• Reduce context switching.

  • No need for bash, Linux, ssh or VPN to fix most server issues.
  • No checking server logs.
  • Easily accesible and readble output.

• Control access.

  • Only gives necessary access, no unnecessary ssh-ing into production!

Features

• Completely NoOps.
• Easily extensible.
• Pay per invocation.
• Provider agnostic.

Quick Start

• Install catops pip install catops
• Run meow install [--template] [--target-dir]
• Adjust the template according to your needs e.g. add Slack OAuth tokens, adjust service names etc.
• Install serverless dependencies npm install in the template directory.
• Run serverless deploy
• Configure your Slack app (i.e. set Slash command/Bot endpoint URLs to appropriate URLs)

Example

Handler

Every Lambda function needs a handler, which takes arguments (event, context). In this case, it is necessary to respond instantly to the request with a 200 so the handler below calls the actual functionality asynchronously and then returns a 200 response.

Example Handler

import json
from six.moves.urllib.parse import parse_qs
import requests
import boto3
from catops import convert_dispatch


def respond(event, context):
    """Call handler.main asynchronously and then return instant response."""
    lambda_client = boto3.client('lambda')
    response = {'statusCode':'200'}
    # Call actual function asynchronously
    lambda_client.invoke(
        FunctionName='CatOps-dev-dispatcher',
        InvocationType='Event',
        Payload=json.dumps(event))
    return response


def main(event, context):
    """Main lamda function logic, to be called asynchronously."""
    params = parse_qs(event.get('body'))
    payload = convert_dispatch(params)
    username =  params.get('user_name', ['catops'])[0] 

    # Post to Slack channel
    response_url = params.get('response_url')
    if type(response_url) is list:
        response_url = response_url[0]
    r = requests.post(response_url, data=json.dumps(payload))
    if not r.ok:
        print(r)
        print(r.reason)
        print(r.text)
    return

Plugins

CatOps works around plugins.

• Plugins are python files stored in the 'plugins/' directory.
• CatOps scans this directory for valid functions to import.
• All files and/or functions starting with _ are ignored. (_ means they are private and will not be added to the CatOps dispatcher)
• Other functions are added to the CatOps dispatcher and can be called via /catops <functionname> [argv]
• All functions need to have the arguments (argv, params)
  • argv will contain the arguments passed to the function e.g. for /catops role --user t.user, argv will contain `['role', '--user', 't.user']
  • params will contain the parse Lambda event body, which contains all the information from Slack e.g. {"text": ... , "username": ..., "response_url": ...}.

Example plugin

"""example.py - example plugin for ChatOps."""
from catops import create_slack_payload

def ping(argv, params):
    """Check is working."""
    text = '@{} Meow!'.format(params.get('user_name', ['CatOps'])[0]),
    return create_slack_payload(text=text)

Serverless configuration

service: CatOps

package:
  include:
    - handler.py
    - plugins/**

custom:
  pythonRequirements:
    slim: true

provider:
  name: aws
  stage: ${opt:stage, 'dev'}
  runtime: python3.6
  profile: serverless
  # Permissions for the lambda function
  # If using boto3, ensure correct permissions
  # have been granted to the lambda function
  iamRoleStatements:
    - Effect: Allow
      Action:
        - lambda:InvokeFunction
        - lambda:InvokeAsync
      Resource: "*"

functions:
  dispatcher:
    handler: handler.main
  respond:
    handler: handler.respond
    events:
      - http:
          path: ping
          method: post

plugins:
  - serverless-python-requirements

Deploy and Test

serverless deploy
serverless invoke --function dispatcher --path /path/to/json/data --log

See examples and templates for more.

Useful functions

# auth.py
def verify_request(event, slack_secret, timeout=True):
    """Verify that Lambda event came from Slack"""

def get_user_slack(event, oauth, team_id=None, channel_id=None):
    """Check whether user exists and is in specified team and channel.

    Arguments:
        event       - AWS Lambda event
        oauth       - Slack OAUTH token
        team_id     - Slack team_id (workspace, i.e. BBOXX)
        channel_id  - Channel user must be in
    Returns:
        False, err_msg
        True, user_dict with id, name, team_id, channel_id, email
    """

# dispatcher.py
def dispatch(command, params=None, plugin_dir='plugins/'):
    """Create Dispatcher object and run parse_command on (command, params)"""

# helpers.py
def get_slack_colour(level):
    """Return Slack colour value based on log level."""

def get_text(params):
    """Return event_text from parse_qs event.get('body')."""

def create_slack_attachment(fallback=None,
                            color=None,
                            pretext=None,
                            author_name=None,
                            author_link=None,
                            author_icon=None,
                            title=None,
                            title_link=None,
                            text=None,
                            fields=None,
                            image_url=None,
                            thumb_url=None,
                            footer=None,
                            footer_icon=None,
                            ts=None
    ):
    """Create slack attachment payload
    See https://api.slack.com/docs/message-attachments for more info.

    Arguments:
        fallback - Required plain-text summary of the attachment
        [color] - Colour of the attachment
        [pretext] - Optional text that appears above the attachment block
        [author_name]
        [author_link]
        [author_icon] - URL to author icon
        [title] - Title of the attachment
        [title_link]
        [text] - Optional text that appears inside the attachment
        [fields] - Array of dicts containing more values
        [image_url] - URL to image attached
        [thumb_url] - URL to image thumbnail
        [footer] - Footer message
        [footer_icon] - URL to footer icon
        [ts] - timestamp
    """

def create_slack_payload(response_type='ephemeral', text="", attachments=None):
    """Create a Slack payload formatted correctly."""

def convert_dispatch(params, convert_function=None, plugin_dir='plugins/'):
    """Call dispatch and convert the output accordingly into a payload."""

# slack_handler.py

class SlackHandler(logging.Handler):
    """Logger slack_handler which posts json log body to lambda_url."""

# install.py

def install(argv=None):
    """Install catops serverless Slack template."""

# parser.py 
class ArgumentParserError(Exception):
    """Error raised by ArgumentParser"""
    pass


class CatParser(argparse.ArgumentParser):
    """Overrides error method to throw an error instead of exiting"""
    def error(self, message):
        raise ArgumentParserError(message)

Installation

sudo apt-get install npm
sudo npm install -g serverless
npm install serverless-python-requirements
pip install catops

Install serverless-python-requirements in the same dir as serverless.yml.

Limitations

• Passive rather than active; needs to be triggered (e.g. by Slack slash commands (could run it every command))
• Limitations of FaaS
  • Max size (256MB for AWS Lambda)
  • Execution time limit (5 minute for AWS Lambda)
  • No state (recommend using a cloud-based database for state e.g. DynamoDB for AWS)
• No autocomplete inside of Slack.