@siteone/gitlab-ci-tools

Deploy package

NOTE: This is a work in progress, please note that it's far from stable version. It's in usable state, but still in development. There are a good chance the configuration and API will change in the future.

Table of contents

// TODO

Introduction

This package main purpose is to provide basic tools to help you implement gitlab CI deployment process for projects. We use predefined (SiteOne standard) way to do this. If your project requires any changes to this standard process you can override or extend any part of the process.

It's created for deploying javascript frontend projects. In theory it can be used to deploy any project, but configuration is in javascript.

There are two main parts of this package:

1. CI jobs task runner - Run any scripts in predefined CI stages
2. Deploy tools - Provide tools to upload, start, stop and delete your build from server during this stages

No previous knowledge of .gitlab-ci.yml is required, but it really helps to understand the concept. For the deploy part we are using PM2 process manager.

NOTE: You will need gitlab project with CI/CD enabled and configured runners (it's beyond scope of this package).

SiteOne deploy process

SiteOne deploy process consists 2 stages (potentially 3 jobs)

1. Build
2. Deploy
3. Stop (deploy stage but different job (stop action), preview environment only)

and up to three environments

1. Production
2. Staging
3. Preview (note that there can be more than one preview running at the same time)

Deployment is initiated by creating new tag in gitlab matching certain regexes (more about this later).

Understanding "CI jobs task runner" part

This part helps with dealing with the CI part of your deployment process. It's just a tool to enable you to run any shell commands during your CI jobs. Now it supports only this predefined three jobs mentioned above.

We use three predefined jobs in two stages

1. Build

In this job build (default) folder should be created. This build folder is saved as artifact to gitlab and used later in deploy stage.

2. Deploy

This job should serve to upload previously created build artifact to your server start the build and make sure it's running. Potentially stop and delete old version of app if there is already one deployed.

3. Stop

This jobs is configured only for preview environment. It can be triggered manually from gitlab. It's purpose is to stop and delete preview (review) of an app that is no longer needed. We never stop or delete production or staging environments.

Process flow chart

graph TD
  subgraph create tag
  tag_created(Tag created) --> environment{environment}
  environment --> |"production || staging || preview"|build([build])
  environment --> |No env|E(end)
  build --> environment2{environment}
  environment2 --> |staging| deploy_staging([deploy_staging job])
  environment2 --> |preview| deploy_preview([deploy_preview job])

  end

  subgraph stop environment
  stop_environment("stop environment triggered <br> (manual in gitlab or autostop)") --> stop_preview([stop_preview job])
  end

  subgraph trigger deploy production
  trigger_production(Manually start <br> deploy production job in gitlab) --> deploy_production([deploy_production job])
  end

  environment2 -.-> |production| trigger_production

  deploy_preview -.-> stop_environment

Configuration (js)

1. ci.config.js

Configuration contains 3 main sections corresponding to gitlab CI jobs (build, deploy, stop). Each job is divided to three phases: before, main, after executed in this order. It really doesn't meter. You can execute all the script in main section and the result will be the same. It's just for easier organization of your tasks and it corresponds to gitlab CI job settings.

Each section corresponds to CiScripts type. The config it self is deep partial of the CiConfig. It's later merged with defaults and validated, so you don't need to fill all values.

type CiScripts {
  before: string[], // Array of shell commands
  main: string[], // Array of shell commands
  after: string[], // Array of shell commands
}

type CiConfig {
  build: CiScripts,
  deploy: CiScripts,
  stop: CiScripts,
}

So for example

const config: PartialCiConfig = {
  build: {
    main: [
      "yarn test",
      "yarn build",
      "yarn start",
      "yarn healthcheck:local"
    ],
  },
  deploy: {
    main: ['yarn ci:deploy']
    after: ['yarn notification:deploy:done']
  }
  stop: {
      main: ['yarn ci:stop'],
  }
};

NOTE: Implementing this npm scripts is developer responsibility. It's pretty much standard supported by all popular javascript frontend frameworks. This package will help you with the yarn ci:deploy and yarn ci:stop. Also yarn install is executed at start of every stage (this package wont work if it wasn't).

2. Gitlab .ci-config.yml

1st you need to create .ci-config.yml (gitlab will recognize it and consume if CI/CI is enabled it without any other setup). Than you can include our gitlab-ci config.

include: node_modules/@siteone/gitlab-ci-tools/ci-tools-gl-config.yml

Than, to ensure this file will be available to gitlab add it to your repository with command:

git add node_modules/@siteone/gitlab-ci-tools/ci-tools-gl-config.yml --force

Using --force will ensure it will be added even if you have node_modules folder in your .gitignore file.

This way you can make sure updating @siteone/gitlab-ci-tools to newer version will just work.

Alternatively, if you don't want to add file from node_modules to your project, you can copy this file in your project and update it manually, when needed.

3. setup gitlab ci/cd env vars

Next you will need to setup CI/CD environment variable in gitlab. First create CI_TOOLS_IMAGE variable. It's image gitlab runner should run your deploy scripts on. We are recommend alpine version and locking it to concrete version to ensure updates wont break your deploy process. It has to contain node and yarn installed. For example:

node:12.18.3-alpine

Triggering deploy process

To trigger deploy process just create new tag in gitlab. This tag must match certain regex to start the process. It differs by environment:

Production /.*-rc.*/
Staging /.*staging-.*/
Preview /.*preview-.*/

for example:

• v.1.0.9-rc.8
• v.1.1.0-rc.9
• staging-2020-09-14
• staging-2020-09-21
• preview-feature-739.v1
• preview-feature-739.v2
• preview-feature-972.v1

After creating this tag deployment process start using branch selected to create tag from.

Stop preview environment

Preview environment is configured to stop automatically after 1 month. You can stop them manually in your gitlab repository environments (operations -> environments) by clicking the stop button. You can also prevent this environment from-auto stopping using button in gitlab environments page.

This is basically all you need to setup your deploy process with gitlab CI, at least the CI part.

Understanding "Deploy tools" part

First of all. This part is independent of the previous "CI jobs task runner" part. This means you can use each of this parts separately and you don't have to use both. So you can use this deploy tools outside gitlab CI.

This part provide some tools to:

1. Create ssh connection to your severs and execute any command on this servers
2. Upload and delete files to and from you servers
3. Start, stop, delete processes from PM2
4. Perform basic health check on just deployed app

Again you can use all this tools separately, override or extend them.

How it works

Deploy script is started using CLI command gitlab-ci-tools deploy (more about this in CLI section) or you can create your custom script importing deploy function from this package. Than add this script to your package json scripts.

There are two possible actions (--action parameter in CLI script on 3rd parameter on deploy function) default and stop

Default Default action purpose is to copy your build folder to server, stop and remove old build (if present), start new build (in PM2) and make sure it's running. It's divided to 4 steps:

1. Deploy before
2. Deploy upload
3. Deploy start
4. Deploy after

Stop

Stop action is meant to stop and remove deployed build from server.

Deploy function flow chart

graph TD
  cli_deploy[run deploy function] --> setup_dynamic_args["deployConfig.setupDynamicArgs || environmentConfig.setupDynamicArgs || nill"]
  setup_dynamic_args --> action{action}
  action --> |default| deploy_before["environmentConfig.deployBefore || deployConfig.deployBefore || nill"]
  deploy_before --> deploy_upload["environmentConfig.deployUpload || deployConfig.deployUpload"]
  deploy_upload --> deploy_start["environmentConfig.deployStart || deployConfig.deployStart"]
  deploy_start --> deploy_after["environmentConfig.deployAfter || deployConfig.deployAfter || nill"]
  action --> |stop| on_stop["environmentConfig.onStop || deployConfig.onStop"]

Deploy configuration example

Again it's a deep partial of DeployConfig type. It's later merged with default's and validated. If you don't need any changes to standard process you just have to setup your environments.

For example:

const config: PartialDeployConfig = {
  environments: [
    // PRODUCTION ENVIRONMENT
    {
      deployEnvironment: "production",
      envVars: {
        APP_ENV: "production",
      }
      uploadConfig: {
        files: ["ecosystem.production.config.js"],
        folders: ["build"],
        uploadTo: "www",
      },
      servers: [
        // Deploy production to two servers
        {
          getPort: () => 8080,
          getAppName: () => "my-testapp-slave",
          sshConfig: {
            host: "my-production-slave-server.mysite.com",
            username: "deploy",
            privateKey: process.env.SSH_PRIVATE_KEY, // Setup this variable in gitlab ci/cd project settings
          },
        },
        {
          getPort: () => 8080,
          getAppName: () => "my-testapp-master",
          sshConfig: {
            host: "my-production-master-server.mysite.com",
            username: "deploy",
            privateKey: process.env.SSH_PRIVATE_KEY, // Setup this variable in gitlab ci/cd project settings
          },
        },
      ],
    },
    // STAGING ENVIRONMENT
    {
      deployEnvironment: "staging",
      envVars: {
        APP_ENV: "staging",
      },
      uploadConfig: {
        files: ["ecosystem.staging.config.js"],
        folders: ["build"],
        uploadTo: "www",
      },
      servers: [
        {
          getPort: () => 8081,
          getAppName: () => "my-testapp-staging",
          sshConfig: {
            host: "my-staging-server.mysite.com",
            username: "deploy",
            privateKey: process.env.SSH_PRIVATE_KEY, // Setup this variable in gitlab ci/cd
          },
        },
      ],
    },
    // PREVIEW ENVIRONMENT
    {
      deployEnvironment: "staging",
      envVars: {
        APP_ENV: "staging",
      },
      uploadConfig: {
        files: ["ecosystem.preview.config.js"],
        folders: ["build"],
        uploadTo: "www",
      },
      servers: [
        {
          getPort: () => {
            // Your get port logic
          },
          getAppName: () => {
            // There can be more preveiws so we need to have different app names for each preview
            return `${process.env.CI_COMMIT_TAG}`;
          },
          sshConfig: {
            host: "my-preview-server.mysite.com",
            username: "deploy",
            privateKey: process.env.SSH_PRIVATE_KEY, // Setup this variable in gitlab ci/cd
          },
        },
      ],
    },
  ],
};

NOTE: You can store any of this values in your gitlab CI/CD environment variables, so you can use the same configuration for more projects and change this variables in gitlab. Also you can access all predefined gitlab CI environment variables in your configuration. On top of that our default ci config exports CI_TOOLS_COMMIT_TIMESTAMP env variable. To use in your scripts.

CLI

// TODO

Configuration

// TODO in depth configuration (ci, preview, env vars)

Api documentation

// TODO in depth api docs

Debugging

If you need more debug information you can set (in gitlab ci/cd settings) DEBUG environment variable for more verbose output.

Available levels are (using loglevel library):

• TRACE
• DEBUG
• INFO
• WARN
• ERROR
• SILENT

Any other non falsy value will set loglevel to TRACE.

Examples

Basic example (production deploy only)

We will start with minimum configuration example. Please note that this is simplest possible example, most of this can be changed or overridden.

Pre requisites

• node/js project
  • build command (in package json scripts) that creates ./build folder and all externalities are bundled in, so your project don't need node_modules folder to run.
  • entry point (build/index.js) that starts your project's server
• gitlab with enabled ci/ci and configured runners.
• configured server(s) with node, pm2 installed where you wish to deploy your project.

Installation

yarn add @siteone/gitlab-ci-tools --dev

Configuration

package.json

In your package.json add ci script:

"scripts": {
    "ci": "gitlab-ci-tools",
    "ci:deploy": "gitlab-ci-tools deploy"
  }

deploy.config.js

Create file deploy.config.js and add following minimal configuration.

module.exports = {
  ciStages: {
    build: {
      ciScripts: {
        main: ["yarn build"],
      },
    },
  },
  deployConfig: {
    environments: [
      {
        deployEnvironment: "production",
        envVars: {
          APP_ENV: "production",
      },
        uploadConfig: {
          files: ["ecosystem.production.config.js"],
          folders: ["build"],
          uploadTo: "www",
        },
        servers: [
          {
            getPort: () => 8080,
            getAppName: () => "my-testapp",
            sshConfig: {
              host: "my.server.com",
              username: "deploy",
              privateKey: process.env.KEY_MASTER, // Setup this variable in gitlab ci/cd project settings
            },
          },
        ],
      },
    ],
  },
};

ecosyste.production.config.js

Create pm2 ecosystem config file:

// Extract arguments

var appName = process.argv[process.argv.length - 2];
var appPort = process.argv[process.argv.length - 1]; // the last argument

module.exports = {
  apps: [
    {
      name: appName,
      script: "./build/index.js",
      max_restarts: 50,
      instances: 6,
      exec_mode: "cluster",
      log_date_format: "YYYY-MM-DD HH:mm Z",
      source_map_support: true,
      // env related settings
      env: {
        NODE_ENV: "production",
        APP_PORT: appPort,
      },
    },
  ],
};

gitlab-ci.yml

Create (or update if you already have one) gitlab-ci.yml with following content.

include: node_modules/@siteone/gitlab-ci-tools/ci-tools-gl-config.yml

than, to ensure this file will be available to gitlab add it to your repository with command: git add node_modules/@siteone/gitlab-ci-tools/ci-tools-gl-config.yml --force

This way you can make sure updating @siteone/gitlab-ci-tools to newer version will just work.

Alternatively, if you don't want to add file from node_modules to your project, you can copy this file in your project and update it manually, when needed.

Environment variables

In your project add environment variables to your gitlab ci/cd settings. In this example we need KEY_MASTER, variable containing private key which gitlab runner can use to connect to your server over ssh.

Now if you create new rc tag in gitlab build stage will start automatically. For production environment you have to start deploy job manually.

// TODO add more examples

Development

This package is using microbundle to create production build.

yarn dev starts microbundle watch - complie files when they change yarn build runs microbundle - create build (dist) from source

More in microbundle package documentation