@siteone/gitlab-ci-tools

Deploy package

NOTE: This is a work in progress, please note that it's far from stable version. It's in usable state, but still in development. There are a good chance the configuration and API will change in the future. Also the docs are still work in progress.

Table of contents

// TODO

Introduction

This package main purpose is to provide basic tools to help you implement gitlab CI deployment process for projects. We use predefined (SiteOne standard) way to do this. If your project requires any changes to this standard process you can override or extend any part of the process.

It's created for deploying javascript frontend projects. In theory it can be used to deploy any project, but configuration is in javascript.

There are three main parts of this package:

1. CI jobs task runner - Run any scripts in predefined CI stages
2. Deploy tools - Provide tools to upload, start, stop and delete your build from server during this stages
3. CLI - to help you create your gitlab-ci.yml config and create tags

No previous knowledge of .gitlab-ci.yml is required, but it really helps to understand the concept. For the deploy part we are using PM2 process manager.

NOTE: You will need gitlab project with CI/CD enabled and configured runners (it's beyond scope of this package).

SiteOne deploy process

SiteOne deploy process consists 2 stages (potentially 3 jobs)

1. Build
2. Deploy
3. Stop (deploy stage but different job (stop action), when auto_stop_in is configured)

You can setup unlimited environments, with different configuration. Also there is a special isMultiEnv option, which allows deploying multiple instances of the same environment. More on that later.

Deployment is initiated by creating new tag in gitlab matching certain regExp, don't worry there is a CLI command that helps you create thees tags.

Understanding "CI jobs task runner" part

This part helps with dealing with the CI part of your deployment process. It's just a tool to enable you to run any shell commands during your CI jobs. Now it supports only this predefined three jobs mentioned above. You can run unlimited scripts and processes during these jobs, so it should fit most of your needs.

We use three predefined jobs in two stages

1. Build

In this job build (default) folder should be created. This build folder is saved as artifact to gitlab and used later in deploy stage. You can also run tests, and other code quality tools in this job, to make sure everything is ok and ready to deploy.

2. Deploy

This job should serve to upload previously created build artifact to your server start the build and make sure it's running. Potentially stop and delete old version of app if there is already one deployed.

3. Stop

This jobs is configured only for isMultiEnv=true environments. It can be triggered manually from gitlab, or you can set auto_stop_in option in you environment config. It's purpose is to stop and delete preview (review) of an app that is no longer needed.

NOTE: We are considering merging build and deploy job to one, to save time installing dependencies and setting up the runner.

Process flow chart

graph TD
  subgraph create tag
  tag_created(Tag created) --> environment{environment}
  environment --> |"match"|build([build])
  environment --> |no match|E(end)
  build --> deployManual{deployManual}
  deployManual --> |false| deploy([environnementName_deploy job])
  deployManual --> |true| trigger_deploy([manual deploy job])

  end

  subgraph stop environment
  stop_environment("stop environment triggered <br> (manual in gitlab or autostop)") --> stop_job([environnementName_stop job])
  end

  subgraph trigger deploy
  trigger_deploy(Manually start <br> deploy production job in gitlab) --> manual_deploy([environnementName_deploy job])
  end



  deploy -.-> autostop
  manual_deploy -.-> autostop

  autostop{auto_stop_in}

  autostop --> |false| E
  autostop --> |true| stop_environment

  stop_job --> E

Configuration (js)

1. ci.config.js

Configuration contains 3 main sections corresponding to gitlab CI jobs (build, deploy, stop). Each job is divided to three phases: before, main, after executed in this order. It really doesn't meter. You can execute all the script in main section and the result will be the same. It's just for easier organization of your tasks and it corresponds to gitlab CI job settings.

Each section corresponds to CiScripts type. The config it self is deep partial of the CiConfig. It's later merged with defaults and validated, so you don't need to fill all values.

type CiScripts {
  before: string[], // Array of shell commands
  main: string[], // Array of shell commands
  after: string[], // Array of shell commands
}

type CiConfig {
  build: CiScripts,
  deploy: CiScripts,
  stop: CiScripts,
}

So for example

const config: PartialCiConfig = {
  build: {
    main: [
      "yarn test",
      "yarn build",
      "yarn start",
      "yarn healthcheck:local"
    ],
  },
  deploy: {
    main: ['yarn ci:deploy']
    after: ['yarn notification:deploy:done']
  }
  stop: {
      main: ['yarn ci:stop'],
  }
};

NOTE: Implementing this npm scripts is developer responsibility. It's pretty much standard supported by all popular javascript frontend frameworks. This package will help you with the yarn ci:deploy and yarn ci:stop. Also yarn install is executed at start of every job in Gitlab runner (this package wont work if it wasn't).

2. setup gitlab ci/cd env vars

Next you will need to setup CI/CD environment variable in gitlab. First create CI_TOOLS_IMAGE variable. It's image gitlab runner should run your deploy scripts on. We are recommend alpine version and locking it to concrete version to ensure updates wont break your deploy process. It has to contain node and yarn installed. For example:

node:12.18.3-alpine

3. Gitlab .ci-config.yml

First you need to create .ci-config.yml file in root of your repository (gitlab will recognize it and consume if CI/CI is enabled without any other setup). Than you can include your gitlab-ci config.

include: ./ci-tools-gl-config.yml

This ci-tools-gl-config.yml config file is generated with our CLI script, based on your deploy.config.js. More on this later.

Triggering deploy process

To trigger deploy process just create new tag in gitlab. This tag must match certain regex to start the process. It differs by your environment settings:

// WIP // TODO describe default env tag regexes.

The best way to create your tag is use this package CLI command createEnv. More on this in CLI section.

Stop preview environment

If you configure auto_stop_in option your environment will auto stop in specified time. You can stop them manually in your gitlab repository environments (operations -> environments) by clicking the stop button. You can also prevent this environment from-auto stopping using button in gitlab environments page.

This is basically all you need to setup your deploy process with gitlab CI, at least the CI part.

Understanding "Deploy tools" part

This part provide some tools to:

1. Create ssh connection to your severs and execute any command on this servers
2. Upload and delete files to and from you servers
3. Start, stop, delete processes from PM2
4. Perform basic health check on just deployed app

You can use all this tools separately, override or extend them.

How it works

Deploy script is started using CLI command gitlab-ci-tools deploy (more about this in CLI section) or you can create your custom script importing deploy function from this package. Than add this script to your package json scripts.

There are two possible actions (--action parameter in CLI script on 3rd parameter on deploy function) default and stop

Default Default action purpose is to copy your build folder to server, stop and remove old build (if present), start new build (in PM2) and make sure it's running. It's divided to 4 steps:

1. Deploy before
2. Deploy upload
3. Deploy start
4. Deploy after

Stop

Stop action is meant to stop and remove deployed build from server.

Deploy function flow chart

graph TD
  cli_deploy[run deploy function] --> setup_dynamic_args["deployConfig.setupDynamicArgs || environmentConfig.setupDynamicArgs || nill"]
  setup_dynamic_args --> action{action}
  action --> |default| deploy_before["environmentConfig.deployBefore || deployConfig.deployBefore || nill"]
  deploy_before --> deploy_upload["environmentConfig.deployUpload || deployConfig.deployUpload"]
  deploy_upload --> deploy_start["environmentConfig.deployStart || deployConfig.deployStart"]
  deploy_start --> deploy_after["environmentConfig.deployAfter || deployConfig.deployAfter || nill"]
  action --> |stop| on_stop["environmentConfig.onStop || deployConfig.onStop"]

Configuration

Work in progress. Meanwhile please check DeployConfig in @siteone/gitlab-ci-tools/dist/configuration/interface

// TODO in depth configuration (ci, preview, env vars)

CLI

Work in progress. Meanwhile please check gitlab-ci-tools --help (node_modules/.bin/gitlab-ci-tools) for more information.

// WIP

Api documentation

// TODO in depth api docs

Debugging

If you need more debug information you can set (in gitlab ci/cd settings) CI_TOOLS_DEBUG environment variable for more verbose output.

Available levels are (using loglevel library):

• TRACE
• DEBUG
• INFO
• WARN
• ERROR
• SILENT

Any other non falsy value will set loglevel to TRACE.

Examples

Basic example (production deploy only)

We will start with minimum configuration example. Please note that this is simplest possible example, most of this can be changed or overridden.

Pre requisites

• node/js project
  • build command (in package json scripts) that creates ./build folder and all externalities are bundled in, so your project don't need node_modules folder to run.
  • entry point (build/index.js) that starts your project's server
• gitlab with enabled ci/ci and configured runners.
• configured server(s) with node, pm2 installed where you wish to deploy your project.

Installation

yarn add @siteone/gitlab-ci-tools --dev

Configuration

package.json

In your package.json add ci script:

"scripts": {
    "build": "webpack --mode=production",
    "ci": "gitlab-ci-tools",
    "ci:createEnv": "gitlab-ci-tools createEnv",
    "ci:deploy": "gitlab-ci-tools deploy",
    "ci:stop": "gitlab-ci-tools deploy --action=stop",
    "ci:createCiConfig": "gitlab-ci-tools createCiConfig"
  }

deploy.config.js

Create file deploy.config.js and add following minimal configuration.

const KEY_MASTER = process.env.KEY_MASTER || "UNSET OUTSIDE RUNNER"
module.exports = {
  environments: [
  {
    environmentName: "production",
    environmentTagCreate: createTag,
    environmentTagParseVersion: parseTag,
    environmentTagIsSemver: true,
    environmentTagIsVersioned: true,
    defaultEnvironmentBranch: 'master',
    isMultiEnv: false,
    deployManual: true,
    uploadConfig: {
      files: ["ecosystem.production.config.js"],
      folders: ["build"],
      uploadTo: "www",
    },
    servers: [
      {
        getPort: () => 8080,
        getAppName: () => "my-testapp",
        sshConfig: {
          host: "my.server.com",
          username: "deploy.user",
          privateKey: KEY_MASTER,
        },
      },
    ],
  },
};

ecosyste.production.config.js

Create pm2 ecosystem config file:

// Extract arguments
var appName = process.argv[process.argv.length - 2];
var appPort = process.argv[process.argv.length - 1];

module.exports = {
  apps: [
    {
      name: appName,
      script: "./build/index.js",
      max_restarts: 50,
      instances: 6,
      exec_mode: "cluster",
      log_date_format: "YYYY-MM-DD HH:mm Z",
      source_map_support: true,
      // env related settings
      env: {
        NODE_ENV: "production",
        APP_PORT: appPort,
      },
    },
  ],
};

ci.config.js

Create your ci.config.js. In this example default values is enough, but file still needs to exist. So just export empty object.

module.exports = {};

gitlab-ci.yml

Use CLI to generate your ci-tools-gl-config.yml.

Run ci:createCiConfig to generate your ci-tools-gl-config.yml file.

Create (or update if you already have one) gitlab-ci.yml with following content.

include: ./ci-tools-gl-config.yml

Environment variables

In your project add environment variables to your gitlab ci/cd settings. In this example we need KEY_MASTER, variable containing private key which gitlab runner can use to connect to your server over ssh.

You also need to setup CI_TOOLS_IMAGE which is image to create your gitlab CI runner from. Search docker hub to find image that suits your needs. Something like lts-alpine would be a good choice. It should be close to the server your project will run on.

Development

This package is using microbundle to create production build.

yarn dev starts microbundle watch - complie files when they change yarn build runs microbundle - create build (dist) from source

More in microbundle package documentation