storyboard-serverless

storyboard-serverless

Storyboard's set of serverless functionality utilizing the Serverless Framework.

Services

• audio-pipeline: our audio processing pipeline that's triggered by new episode uploads

  • dev environment: us-east-2
  • prod environment: us-east-1
  • full details within the services/audio-pipeline README

Project Structure

The general folder structure is following the 'Microservices + Monorepo' architecture outlined here. Example below:

├── layers
│    └── ffmpeg-layer
│        ├── serverless.yml
│        └── layer
│            └── ffmpeg
│                └── ffmpeg (the actual static FFmpeg binary)
│                └── ffprobe (the actual static FFprobe binary)
│    └── sox-layer
│        ├── serverless.yml
│        └── layer
│            └── lib
│                └── libgomp.so.1 (a dependency the SoX binary depends on)
│            └── sox
│                └── sox (the actual static SoX binary)
│
├── services
│    └── audio-pipeline
│        ├── package.json
│        ├── quality-control.js
│        ├── sox.js
│        └── serverless.yml
│
│    └── service-2
│        ├── package.json
│        ├── some-handler.js
│        └── serverless.yml
│
│    └── service-3
│        ├── package.json
│        ├── etc-handler.js
│        └── serverless.yml
│
├── package.json
├── README.md
├── variables.yml     // repo-wide variables can be set here
├── .gitignore
├── .npmignore
└── .npmrc

Updates & Releases

Use chan to provide updates to CHANGELOG.md as they're made. Some ease-of-use shortcuts are provided in package.json.

Example:

• npm run pipeline:added -- 'added some cool new stuff'
• npm run pipeline:fixed -- 'fixed that weird bug'
• npm run version // marks the unreleased changes as a new release in your changelog
• npm run gh-release // creates a github release at the root of the repo

Use lerna to bump version numbers of individual services

• Example: lerna publish

Deploying Services

Install the Serverless CLI: npm install -g serverless

You'll first need to run serverless login and provide the appropriate credentials for [e-mail]. These are stored in the SSM Parameter Store under serverless-framework-login.

Make sure your AWS CLI is configured properly before deploying. If you don't have any AWS profiles set locally, type aws configure and provide the AWS Access Key ID and Secret Access Key for the ServerlessFrameworkAccount IAM user. These are stored in the SSM Parameter Store under serverless-framework-iam-account.

Once your CLI is configured properly and you're using the ServerlessFrameworkAccount's AWS credentials, you can deploy the infrastructure.

Run serverless deploy --aws-profile {the serverless profile name in ~/.aws/credentials} inside one of the services folders. You need to be at the same level as a serverless.yml file.

Step-by-step example:

• npm install -g serverless
• serverless login
• aws configure
  • provide Access Key ID and Secret Access Key for serverless-framework-iam-account
• cd services/audio-pipeline
• serverless deploy --aws-profile storyboard-serverless
  • the storyboard-serverless aws-profile name here should be whatever you call your serverless profile name in ~/.aws/credentials