kube-platform

kube-platform

A tool for managing Invoca developer platforms.

kube-platform can be run from any location that has a properly configured kubectl (configuring kubectl is outside the scope of this README). The currently selected context inside $HOME/.kube/config will be used by default, but can be overriden with the --context flag.

See the #dev-qa-infrastructure slack channel for further discussion.

Installation

1. Follow the instructions for configuring kubectl on your local workstation: kubectl configuration instructions.
2. Install the Invoca tools Homebrew tap.

$ brew tap invoca/tools [e-mail]:Invoca/homebrew-tools.git

1. Install kube-platform. The initial install builds a dedicated Ruby, which adds 3-5 minutes to the install time.

$ brew install kube-platform

1. Configure Development AWS IAM account Credentials on your local workstation

Upgrading

At times, changes will require that kube-platform be upgraded. When necessary you'll receive a message like this:

This version of kube-platform is out-of-date: 1.1.0 is required, but running 1.0.0.
Please update kube-platform: brew update && brew upgrade kube-platform

To upgrade, run

brew update && brew upgrade kube-platform

Platform Creation

To create a platform, run:

$ kube-platform create <platform-name> [image tags] [options]

<platform-name> corresponds to the Kubernetes namespace that the platform will be started in. A full list of available options can be viewed by running kube-platform help create. The developer mode, and continuous testing options are explained below.

Image tags for various containers can be provided on the command line. If an image tag is not provided for a container, latest is assumed, with one exception: if a tag is provided for invocaops/web_app, it will be the default tag for all containers started from invocaops/web_app.

To list the available tags, run kube-platform help create.

Platform creation failure? See the troubleshooting runbook in confluence

Platform Update

To update the image tags of a running platform, run:

$ kube-platform update <platform-name> [image tags]

Any database migrations contained in the new web image will be applied before the containers are restarted.

Platform Destruction

To tear down your platform once you're done with it, run:

$ kube-platform destroy <platform-name>

Developer Mode

Applications within containers run as PID 1. When PID 1 exits, it signals to the container manager that the container has stopped, which results in it being torn down and restarted. When the container is restarted, any local changes made inside the container will be lost.

If there's a need to edit and restart the code inside a container, the application must not be running as PID 1. The --dev-mode flag runs the application inside a simple bash loop, which allows the process to be killed and restarted while maintaining the container's state.

Dev mode can be enabled during platform creation with the --dev-mode option.

$ kube-platform create <namespace> --dev-mode [image tags] [options]

After enabling dev mode, perform the following steps to restart a process:

Login to the container (eg. frontend)

$ kubectl --namespace=<namespace> exec -it frontend-rc-<id> -c unicorn bash

Verify the process is not running as PID 1. See the unicorn master PID in this example:

root@frontend-rc-jx467:/srv/app# ps auxw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  18096  1656 ?        Ss   23:40   0:00 /bin/bash /invoca_run.sh frontend
root        75 84.1  2.5 829512 385848 ?       Sl   23:42   0:41 unicorn master -c /srv/app/config/unicorn_docker.rb -E   staging
root       127  0.1  2.4 829512 380168 ?       Sl   23:43   0:00 unicorn worker[0] -c /srv/app/config/unicorn_docker.rb -E staging
root       130  0.1  2.4 829512 380160 ?       Sl   23:43   0:00 unicorn worker[1] -c /srv/app/config/unicorn_docker.rb -E staging
root       133  0.0  2.4 829512 380160 ?       Sl   23:43   0:00 unicorn worker[2] -c /srv/app/config/unicorn_docker.rb -E staging
root       136  0.0  2.4 829512 380160 ?       Sl   23:43   0:00 unicorn worker[3] -c /srv/app/config/unicorn_docker.rb -E staging
root       139  0.1  2.4 829512 380160 ?       Sl   23:43   0:00 unicorn worker[4] -c /srv/app/config/unicorn_docker.rb -E staging
root       142  0.1  2.4 829512 380164 ?       Sl   23:43   0:00 unicorn worker[5] -c /srv/app/config/unicorn_docker.rb -E staging

Make any necessary changes to the code, configuration files, etc.

Kill the PID of the application, unicorn master in this example.

root@frontend-rc-jx467:/srv/app# kill 75

Observe that the unicorn master and its worker processes have restarted after a few seconds and that the container remains up.

root@frontend-rc-jx467:/srv/app# ps auxw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  18096  1656 ?        Ss   23:40   0:00 /bin/bash /invoca_run.sh frontend
root       157  0.0  0.0  18232  2092 ?        Ss   23:43   0:00 bash
root       242 45.3  2.4 694728 384656 ?       Sl   23:50   0:39 unicorn master -c /srv/app/config/unicorn_docker.rb -E staging
root       291  0.0  2.4 694728 378716 ?       Sl   23:51   0:00 unicorn worker[0] -c /srv/app/config/unicorn_docker.rb -E staging
root       294  0.0  2.4 694728 378700 ?       Sl   23:51   0:00 unicorn worker[1] -c /srv/app/config/unicorn_docker.rb -E staging
root       297  0.0  2.4 694728 378700 ?       Sl   23:51   0:00 unicorn worker[2] -c /srv/app/config/unicorn_docker.rb -E staging
root       300  0.0  2.4 694728 378700 ?       Sl   23:51   0:00 unicorn worker[3] -c /srv/app/config/unicorn_docker.rb -E staging
root       303  0.0  2.4 694728 378700 ?       Sl   23:51   0:00 unicorn worker[4] -c /srv/app/config/unicorn_docker.rb -E staging
root       306  0.0  2.4 694728 378704 ?       Sl   23:51   0:00 unicorn worker[5] -c /srv/app/config/unicorn_docker.rb -E staging

Continuous Testing Mode

A normal developer platform requires access from the outside world. Frontends and PNAPI endpoints need to accessible, calls need to be routed into the platform, and so on. Most of the ingress points into a platform are provided by ELBs. This can cause problems during periods of heavy Autobahn usage; AWS rate limits the number of ELBs that can be created at once.

Since Autobahn runs inside the platform, the ingress points aren't necessary. Continuous testing mode eliminates the ELBs from the platform and routes all traffic to internal platform IP addresses. This mode will also create an autobahn pod within the platform. This option should only be used for Autobahn runs.

To create a platform in continuous testing mode, run:

$ kube-platform create <platform-name> --ct [image tags] [options]

Autobahn Development Mode

In this mode, an autobahn pod will be created within the platform, to be used for Autobahn development and testing. Normal platforms created without this option do not include an autobahn pod.

To create a platform in autobahn developer mode, run:

$ kube-platform create <platform-name> --autobahn-dev [image tags] [options]

FAKETIME

With --faketime option enabled, the date/time presented to select running application can be changed by editing /srv/app/FAKETIME file. The running application does not need to be restarted when /srv/app/FAKETIME is modified.

NOTE: --faketime option requires --dev_mode therefore dev_mode is automatically enabled when --faketime option is specified.

For example, by putting "-30d" (without quotes) into /srv/app/FAKETIME, then killing the master unicorn and allowing the unicorns to come back up, the application will see date as 30 days before today.

See https://github.com/wolfcw/libfaketime for more info.

Frontend (unicorn), job and task applications will honor changes to /srv/app/FAKETIME file when --faketime option is enabled.

To create a platform with FAKETIME support, include --faketime option during platform creation:

$ kube-platform create <platform-name> [image tags] --faketime [options]

To include scope to script/con:

LD_PRELOAD=/usr/local/lib/libfaketime.so.1 script/console + bundle exec rails console

CONTRIBUTING

The Rough Steps

• Update kube-platform itself
• Manually release the new version via Github
• Seed the new version into Homebrew
• Make an announcement
• Update tooling (if required)

The Detailed Steps

• Merge or push new kube-platform functionality to mainline.
• Tag the kube-platform release manually here, bumping the version according to semantic versioning.
• Clone invoca/homebrew-tools (Invoca’s custom Homebrew tap) and update the kube-platform Homebrew formula with the new version.
• Open PR and after review, merge to mainline.
  • After merging to mainline, running brew update will pull in the new formula, meaning that a subsequent brew upgrade will update kube-platform.
• Make an announcement/post in #dev-announcements about the new version
  • If you plan to continue below, make sure to call out that an upgrade will be required in some timeframe (perhaps give a day or two for engineers to brew upgrade).

The below steps are when you want to force all previous version of kube-platform to be upgraded to the new version.

• Clone invoca/invocaops_docker (which contains Invoca's Kubernetes resources) and change the version field in kube-platform/manifest.jsonnet to match the new version number. Doing so will cause any kube-platform commands ran from a previous version to fail with a prompt asking the user to upgrade.
• Open PR and after review, merge to mainline.
  • You do not have to tag this “as a release” since kube-platform pulls the manifest.yaml.erb from the repository with every command.
• Update Thanatos with the new release of kube-platform since Thanatos uses kube-platform as a gem to handle platform deletions.
  • Run bundle update —conservative kube-platform in Invoca/thanatos
  • Open a PR with the Gemfile.lock changes, and after review merge to mainline.
  • Note! Someone will need to redeploy Thanatos by deleting the Pod when the mainline build completes so the new SHA can come up.
    • kubectl delete pod --context .staging.us-east-1 --namespace=ops-tools -l app=thanatos