corbel-composr

#Composr

Homepage

Overview

Composr is a nodeJS opinionated server for executing dinamically created endpoints, it's original purpose is to serve as a middleware for the Corbel - microservices generic backend - , but it's capabilities are growing.

It uses the composr-core API for executing random pieces of code that the developers pushes to the Composr API. This random pieces of code, called Phrases or Snippets are model definitions for endpoints and reusable utilities.

Why Composr?

When working with a microservices backend lot of requests have to be grouped together in a "business logic" middleware, Composr serves this purpose. It also adds a bit of coolness.

Features

• Enable/disable endpoints in runtime dynamically!
• Built in PM2 configuration for cluster mode
• Bunyan Access logs
• Winston Logs (debug, info, warn, error)
• Dynamic versioning support
• Autogenerated API documentation
• RabbitMQ communication for endpoints updates
• Status and Healthcheck endpoints
• Keymetrics and Newrelic integration
• Docker Ready!
• Configurable middlewares for each endpoint
  • Mock: It mocks the response of the endpoint with an autogenerated object
  • Validate: It validates the input object schema for POST/PUT requests
  • Auth Middlewares: Ensures that the endpoint receives a Corbel Auth Token
  • Cache Middlewares: Configure which endpoints will be cached and for how long.
  • ...
• Corbel ready, thanks to Corbel-js

QuickStart

• install

  npm install -g bq/corbel-composr
  

• run server

  corbel-composr
  

Configuration

One way of starting composr is using it as a command, corbel-composr.

This will search for the default configuration under a config folder in the current directory. Composr uses the NPM config module for the configuration. So it will basically search for a development.json file under the config folder.

If NODE_ENV is not set in the environment, a default value of development is used.

Default config file

{
    "serverName" : "CompoSR",
    "bodylimit" : "50mb",
    "port": 3000,
    
    "rabbitmq": {
        "host": "",
        "port": "",
        "username": "",
        "password": "",
        "reconntimeout": 10000,
        "event": "class com.bq.corbel.event.ResourceEvent",
        "forceconnect": false,
        "heartbeat" : 30
    },

    "bootstrap.retrytimeout": 10000,

    "services": {
        "timeout": 5000,
        "retries": 30,
        "time": 1000
    },

    "corbel": {
        "credentials": {
            "clientId": "",
            "clientSecret": "",
            "scopes": "composr:comp:base"
        },
        "options": {
            "urlBase": ""
        }
    },

    "redis": {
        "host": "localhost",
        "port": 6379,
        "user": "",
        "password": "",
        "db": ""
    },

    "bunyan": {
        "log" : true,
        "syslog" : true,
        "stdout": false,
        "streamServer": false
    },

    "composrLog": {
        "accessLog" : true,
        "accessLogFile" : "logs/access.log",
        "logLevel": "error",
        "logFile": "logs/composr.log",
        "syslog" : false
    },

    "newrelic": {
        "enabled": false,
        "name": "",
        "key": ""
    },

    "keymetrics": true,

    "execution": {
        "vm": false,
        "gc": false,
        "timeout": 40000,
        "local": false
    }
}

Almost all of the vales in the configuration file can be overwriten by environment variables, this can be useful if you use Docker, Travis or any other tool that could send environment variables to configure your server.

Environment variables

SERVER_NAME (Composr 2.0)
PORT (3000)
CREDENTIALS_CLIENT_ID
CREDENTIALS_CLIENT_SECRET
CREDENTIALS_SCOPES
URL_BASE
ACCESS_LOG => winston access log
ACCESS_LOG_FILE => winston access log file
LOG_LEVEL => winston log level
LOG_FILE => winston log file
BUNYAN_LOG(true) => Bunyan logs
BUNYAN_SYSLOG(true) => Send bunyan stream to syslog (127.0.0.1:514)
BUNYAN_STDOUT(false) => Bunyan output in terminal
BUNYAN_STREAM_SERVER (null) => Composr Stream Server endpoint
RABBITMQ_HOST
RABBITMQ_PORT
RABBITMQ_USERNAME
RABBITMQ_PASSWORD
RABBITMQ_FORCE_CONNECT => Only launch composr if rabbit is connected
RABBITMQ_HEARTBEAT => Heartbeat for the rabbitmq connection
REDIS_HOST
REDIS_PORT
REDIS_USER
REDIS_DB => Optional, specify the redis db to use, useful when multiple instances running vs the same redis
REDIS_PASSWORD
SERVICES_TIMEOUT
SERVIES_RETRIES
SERVICES_TIME 
KEYMETRICS (true) => Keymetrics active
NRACTIVE => New relic active
NRAPPNAME => New relic app name
NRAPIKEY => New relic api key
TIMEOUT=> Endpoint timeout
LOCAL_MODE=>If set to "true" it skips loading the endpoints from a remote server and uses the local files

Creating your endpoints:

What are Phrases or Snippets?

You can generate and publish your phrases and snippets by using the composr-cli.

npm install -g composr-cli

composr -g

Bootstraping a project:

composr init will generate a basic structure for your Phrases project.

Once you bootstrapped some phrases, just run corbel-composr in the current folder and the server will be ready at the port 3000.

Routing endpoints

Corbel-Composr has a similar routing mechanism than restify. You can define urls by following this conventions:

• :param : Url parameter
• user : Fixed path value

Some examples

• user/:userId
• user/status/:parameter
• thing/one

{
    "url": "paramsExample/:pathparam",
    "get": {
        "code": "res.status(200).send('path param: ' + req.params.pathparam + ',  query param: ' + req.query.queryparam);"
    },
    "post": {
       /*...*/
    },
    "put": {
       /*...*/
    }
}

Versioning your endpoints

Composr can take care of multiple endpoints and multiple versions for each endpoint. It uses the semantic versioning for executing different code for each endpoint.

For example, if you published the following phrases to Composr:

{
    "url": "user/:userId",
    "version": "3.1.0",
    "get": { ... }
}

{
    "url": "user/:userId",
    "version": "4.0.0",
    "get": { ... }
}

Then you could request executing the 3.x version by sending the Accept-Version header with a ~3 value, as seen in restify.

Documentation

Composr autogenerates documentation when navigating to http://localhost:3000/my:domain/doc. The documentation is generated by fulfilling the example documents that composr-cli creates when generating a phrase model.

See an example:

Phrase Middlewares

We offer a set of plugable add-ons that will automatically add functionality to your phrases. These middlewares or hooks are executed before or after the code on your phrase. You can specify them on the phrase's spec at a method level like this (order is important!):

{
  "url": "resource",
  "get": {
    "middlewares": [
      "auth",
      "validate", 
      "mock"
    ],
    "doc": {
...
}

Among the available middlewares you can find:

• 'corbel-auth-client': Automatic corbel authentication for clients
• 'corbel-auth-user': Automatic corbel authentication for users. A new attribute 'userId' will be available on the phrase for a succeeding user token.
• 'validate': Automatic validation of path params, query params and body based on documentations' schema
• 'mock': Mocked responses based on schema or example from documentation

Cache

Composr instances are connected to a redis machine that serves as a dynamic cache system.

In order to accomplish cache and cache invalidation, phrase models can define some rules:

{
  "url": "resource",
  "get": {
    "middlewares": [
      "cache"
    ],
    "cache": {
      "type" : "user",
      "duration": "5m"
    },
    "doc": { ... }
  }
  "post": {
    "middlewares": [
      "cache"
    ],
    "cache": {
      "invalidate": ["resource"]
    },
    "doc": {...}
  }
...
}

The phrase model uses the cache middleware in the verbs GET and POST. In the GET method the cache will store in Redis a key for each user with a time to live of five minutes.

When a request is made to the POST method, the cache middleware will hit in, too, but in this case it will invalidate the cache of the GET method for the resource url.

Cache can be used for:

• Client requests
• User requests

The cache for client request is the most common type of cache, client requests are made with a client token (which refers normally to a public resource), or without token at all.

The cache for user requests should be used carefully because it will create a Redis key-value pair for each user.

Each cached endpoint can have a duration (time to live), some examples of available values are:

• 1m : 1 minute
• 100ms: 100 miliseconds
• 2h 30mins: 2 hours 30minutes
• 1d: 1 day
• 1w: 1 week

See parse-duration for valid values

Logs

Composr is shipped with built-in bunyan and winston support.

Winston logs:

You can set logFile and logLevel in your config file.

Available log levels can be found at winston's npm page:

• debug
• info
• warn
• error

Bunyan Logs:

Bunyan logs are enabled by default. You can disable them by turning bunyan.log to false in your configuration.

Tests

npm test

Coverage

npm run coverage

Debug

Requires node-inspector

npm install -g node-inspector

• Server

  npm run debug --myphrase.get
  

• Tests

  npm run test:debug
  

Run in a docker container

• clone repo

• build image

  docker build -t <username>/corbel-composr .
  

• run container

  docker run -d -p 3000:3000 --name="corbel-composr"  <username>/corbel-composr
  

• start/stop container

  docker start/stop corbel-composr
  

Public hub

Reference

• composr-core
• corbel-js API
• RAML for phrase definition

Other requirements

A Redis instance, if you want to run it locally with docker:

sudo docker run --name some-redis -p 6379:6379 -d redis

TODO LIST

• Separate the hooks in different modules. Allow enabling o disabling hoooks via server configuration.