The power of Kubernetes with the ease of Heroku!
Enable the Heroku Flow deployment model with Control Plane using the cpflow
gem.
If you need a free demo account for Control Plane (no CC required), you can contact Justin Gordon, CEO of ShakaCode.
Be sure to see the demo app
If you would like to see the simple YAML configuration and setup,
Also, check how the cpflow
gem (this project) is used in the Github actions.
Here is a brief video overview.
This playbook shows how to move "Heroku apps" to "Control Plane workloads" via an open-source cpflow
CLI on top of
Control Plane's cpln
CLI.
Heroku provides a UX and CLI that enables easy publishing of Ruby on Rails and other apps. This ease of use comes via
many "Heroku" abstractions and naming conventions.
Control Plane, on the other hand, gives you access to raw cloud computing power. However, you need to know precisely how
to use it.
To simplify migration to and usage of Control Plane for Heroku users, this repository provides a concept mapping and
a helper CLI based on templates to save lots of day-to-day typing (and human errors).
- Key Features
- Concept Mapping
- Installation
- Steps to Migrate
- Configuration Files
- Workflow
- Environment
- Database
- In-memory Databases
- Scheduled Jobs
- CLI Commands Reference
- Mapping of Heroku Commands to
cpflow
and cpln
- Examples
- Migrating Postgres Database from Heroku Infrastructure
- Migrating Redis Database from Heroku Infrastructure
- Tips
Key Features
- A
cpflow
command to complement the default Control Plane cpln
command with "Heroku style scripting." The Ruby source
can serve as inspiration for your own scripts. - Easy to understand Heroku to Control Plane conventions in setup and naming.
- Safe, production-ready equivalents of
heroku run
and heroku run:detached
for Control Plane. - Automatic sequential release tagging for Docker images.
- A project-aware CLI that enables working on multiple projects.
Concept Mapping
On Heroku, everything runs as an app, which means an entity that:
- runs code from a Git repository.
- runs several process types, as defined in the
Procfile
. - has dynos, which are Linux containers that run these process types.
- has add-ons, including the database and other services.
- has common environment variables.
On Control Plane, we can map a Heroku app to a GVC (Global Virtual Cloud). Such a cloud consists of workloads, which can
be anything that can run as a container.
Heroku | Control Plane |
---|
app | GVC (Global Virtual Cloud) |
dyno | workload |
add-on | either a workload or an external resource |
review app | GVC (app) in staging organization |
staging env | GVC (app) in staging organization |
production env | GVC (app) in production organization |
On Heroku, dyno types are specified in the Procfile
and configured via the CLI/UI; add-ons are configured only via the
CLI/UI.
On Control Plane, workloads are created either by templates (preferred way) or via the CLI/UI.
For the typical Rails app, this means:
Function | Examples | On Heroku | On Control Plane |
---|
web traffic | rails , sinatra | web dyno | workload with app image |
background jobs | sidekiq , resque | worker dyno | workload with app image |
db | postgres , mysql | add-on | external provider or can be set up for development/testing with Docker image (lacks persistence between restarts) |
in-memory db | redis , memcached | add-on | external provider or can be set up for development/testing with Docker image (lacks persistence between restarts) |
others | mailtrap | add-on | external provider or can be set up for development/testing with Docker image (lacks persistence between restarts) |
Installation
-
Ensure your Control Plane account is set up. Set up an organization
<your-org>
for testing in that account and modify the value for aliases.common.cpln_org
in .controlplane/controlplane.yml
, or you can also set it with the CPLN_ORG
environment variable. If you need an organization, please contact Shakacode.
-
Install Node.js (required for Control Plane CLI).
-
Install Ruby (required for these helpers).
-
Install Control Plane CLI, and configure access (docs here).
npm install -g @controlplane/cli
cpln login
npm update -g @controlplane/cli
-
Run cpln image docker-login --org <your-org>
to ensure that you have access to the Control Plane Docker registry.
-
Install Control Plane Flow cpflow
CLI as a Ruby gem: gem install cpflow
. If you want to use cpflow
from Rake tasks in a Rails project, use Bundler.with_unbundled_env {
cpflow help} or else you'll get an error that
cpflowcannot be found. While you can add
cpflow` to your Gemfile, it's not recommended because it might trigger conflicts with other gems.
-
You can use this Dockerfile as an example for your project. Ensure that you have Docker running.
Note: Do not confuse the cpflow
CLI with the cpln
CLI. The cpflow
CLI is the Control Plane Flow playbook CLI.
The cpln
CLI is the Control Plane CLI.
Steps to Migrate
Click here to see the steps to migrate.
Configuration Files
The cpflow
gem is based on several configuration files within a /.controlplane
top-level directory in your project.
.controlplane/
├─ templates/
│ ├─ app.yml
│ ├─ postgres.yml
│ ├─ rails.yml
├─ controlplane.yml
├─ Dockerfile
├─ entrypoint.sh
controlplane.yml
describes the overall application. Be sure to have <your-org>
as the value for aliases.common.cpln_org
, or set it with the CPLN_ORG
environment variable.Dockerfile
builds the production application. entrypoint.sh
is an example entrypoint script for the production application, referenced in your Dockerfile.templates
directory contains the templates for the various workloads, such as rails.yml
and postgres.yml
.templates/app.yml
defines your project's GVC (like a Heroku app). More importantly, it contains ENV values for the app.templates/rails.yml
defines your Rails workload. It may inherit ENV values from the parent GVC, which is populated from the templates/app.yml
. This file also configures scaling, sizing, firewalls, and other workload-specific values.- For other workloads (like lines in a Heroku
Procfile
), you create additional template files. For example, you can base a templates/sidekiq.yml
on the templates/rails.yml
file. - You can have other files in the
templates
directory, such as redis.yml
and postgres.yml
, which could setup Redis and Postgres for a testing application.
Here's a complete example of all supported config keys explained for the controlplane.yml
file:
controlplane.yml
allow_org_override_by_env: true
allow_app_override_by_env: true
aliases:
common: &common
cpln_org: my-org-staging
default_location: aws-us-east-2
setup_app_templates:
- app
- redis
- postgres
- memcached
- rails
- sidekiq
skip_secrets_setup: true
secrets_name: my-secrets
secrets_policy_name: my-secrets-policy
one_off_workload: rails
app_workloads:
- rails
- sidekiq
additional_workloads:
- redis
- postgres
- memcached
maintenance_workload: maintenance
fix_terminal_size: true
runner_job_default_cpu: "2"
runner_job_default_memory: "4Gi"
runner_job_timeout: 1000
stale_app_image_deployed_days: 5
image_retention_max_qty: 20
image_retention_days: 5
apps:
my-app-staging:
<<: *common
my-app-review:
<<: *common
match_if_app_name_starts_with: true
hooks:
post_creation: bundle exec rake db:prepare
pre_deletion: bundle exec rake db:drop
my-app-production:
<<: *common
allow_org_override_by_env: false
allow_app_override_by_env: false
cpln_org: my-org-production
upstream: my-app-staging
release_script: release_script
default_domain: domain.com
my-app-other:
<<: *common
dockerfile: ../some_other/Dockerfile
Workflow
For a live example, see the react-webpack-rails-tutorial repository.
This example should closely match the below example.
Suppose your app is called tutorial-app
. You can run the following commands.
Setup Commands
cpflow apply-template app postgres redis rails daily-task -a tutorial-app
cpflow build-image -a tutorial-app
cpflow deploy-image -a tutorial-app
cpflow logs -a tutorial-app
cpflow open -a tutorial-app
Promoting Code Updates
After committing code, you will update your deployment of tutorial-app
with the following commands:
cpflow build-image -a tutorial-app
cpflow run -a tutorial-app --image latest -- rails db:migrate
cpflow deploy-image -a tutorial-app
If you needed to push a new image with a specific commit SHA, you can run the following command:
cpflow build-image -a tutorial-app --commit ABCD
Real World
Most companies will configure their CI system to handle the above steps. Please contact Shakacode for examples of how to do this.
You can also join our Slack channel for ShakaCode open source projects.
Environment
There are two main places where we can set up environment variables in Control Plane:
-
In workload/container/env
- those are container-specific and must be set up individually for each container.
-
In gvc/env
- this is a "common" place to keep env vars which we can share among different workloads. Those
common variables are not visible by default, and we should explicitly enable them via the inheritEnv
property.
Generally, gvc/env
vars are useful for "app" types of workloads, e.g., rails
, sidekiq
, as they can easily share
common configs (the same way as on a Heroku app). They are not needed for non-app workloads, e.g., redis
, memcached
.
It is ok to keep most of the environment variables for non-production environments in the app templates as, in general,
they are not secret and can be committed to the repository.
It is also possible to set up a Secret store (of type Dictionary
), which we can reference as, e.g.,
cpln://secret/MY_SECRET_STORE_NAME/MY_SECRET_VAR_NAME
. In such a case, we must set up an app Identity and proper
Policy to access the secret.
In templates/app.yml
:
spec:
env:
- name: MY_GLOBAL_VAR
value: "value"
- name: MY_SECRET_GLOBAL_VAR
value: "cpln://secret/MY_SECRET_STORE_NAME/MY_SECRET_GLOBAL_VAR"
In templates/rails.yml
:
spec:
containers:
- name: rails
env:
- name: MY_LOCAL_VAR
value: "value"
- name: MY_SECRET_LOCAL_VAR
value: "cpln://secret/MY_SECRET_STORE_NAME/MY_SECRET_LOCAL_VAR"
inheritEnv: true
Database
There are several options for a database setup on Control Plane:
-
Heroku Postgres. It is the least recommended but simplest. We only need to provision the Postgres add-on on Heroku
and copy its XXXXXX_URL
connection string. This is good for quick testing but unsuitable for the long term.
-
Control Plane container. We can set it up as a workload using one of the default
Docker Hub images. However, such a setup lacks persistence between container restarts. We
can use this only for an example or test app where the database doesn't keep any serious data and where such data is
restorable.
-
Any other cloud provider for Postgres, e.g., Amazon's RDS can be a quick go-to. Here are
instructions for setting up a free tier of RDS.
Tip: If you are using RDS for development/testing purposes, you might consider running such a database publicly
accessible (Heroku actually does that for all of its Postgres databases unless they are within private spaces). Then we
can connect to such a database from everywhere with only the correct username/password.
By default, we have structured our templates to accomplish this with only a single free tier or low tier AWS RDS
instance that can serve all your development/testing needs for small/medium applications, e.g., as follows:
aws-rds-single-pg-instance
mydb-staging
mydb-review-111
mydb-review-222
mydb-review-333
Additionally, we provide a default postgres
template in this repository optimized for Control Plane and suitable for
development purposes.
In-memory Databases
E.g., Redis, Memcached.
For development purposes, it's useful to set those up as Control Plane workloads, as in most cases, they don't keep any
valuable data and can be safely restarted, which doesn't affect application performance.
For production purposes or where restarts are not an option, you should use external cloud services.
We provide default redis
and memcached
templates in this repository optimized for Control Plane and suitable for
development purposes.
Scheduled Jobs
Control Plane supports scheduled jobs via cron workloads.
Here's a partial example of a template for a cron workload, using the app image:
kind: workload
name: daily-task
spec:
type: cron
job:
schedule: "0 2 * * *"
restartPolicy: Never
containers:
- name: daily-task
args:
- bundle
- exec
- rails
- db:prepare
image: "/org/APP_ORG/image/APP_IMAGE"
A complete example can be found at templates/daily-task.yml, optimized for Control Plane and
suitable for development purposes.
You can create the cron workload by adding the template for it to the .controlplane/templates/
directory and running
cpflow apply-template my-template -a my-app
, where my-template
is the name of the template file (e.g., my-template.yml
).
Then to view the logs of the cron workload, you can run cpflow logs -a my-app -w my-template
.
CLI Commands Reference
Click here to see the commands.
You can also run the following command:
cpflow --help
Mapping of Heroku Commands to cpflow
and cpln
Examples
Resources