@linzjs/cdk-elastic-shipper

Elastic Log Shipper

Lambda function to ship logs from various AWS sources into a elastic search instance of your choosing.

Usage

This package exposes a AWS-CDK construct that can be imported into your CDK stack and can be used to configure logs to be automatically shipped into an elastic search of your choosing

import * as cdk from '@aws-cdk/aws-cdk'
import * as ssm from '@aws-cdk/aws-ssm'
import { LambdaLogShipperFunction } from '@linzjs/cdk-elastic-shipper'

export class YourStack extends cdk.Stack {

  public constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    const vpc = new ec2.Vpc(this, 'Vpc');

    const config = {
      name: 'default',
      accounts: [{
        id: '1234567890',
        elastic: '/es-shipper-config/elastic-default',
        name: 'linz',
        tags: ['hello'],
        prefix: 'account-prefix',
        logGroups: [{
          filter: '**',
          prefix: 'lg-prefix',
        }],
      }],
    };
    this.logShipper = new LambdaLogShipperFunction(this, 'LogShipper', { config, vpc });

    /** Register a listener on a bucket, so when files are added they are to submitted to the log shipper*/
    const logBucket = new s3.Bucket(this, 'LogBucket');
    this.logShipper.addS3Source(logBucket);
  }
}

The elastic connection strings need to be stored inside of SSM before running the deployment, there are three options for elasticsearch connections

• AWS

{ url: 'https://node-name.eu-west-1.es.amazonaws.com' }

• ElasticCloud

{
  id: 'cloud:abc123',
  username: 'foo',
  password: 'bar'
}

• Http

{
  url: 'https://fake.com'
  username: 'foo'
  password: 'bar'
}

Building

This repository requires NodeJs > 12 & Yarn

Use n to manage nodeJs versions

# Download the latest nodejs & yarn
n latest
npm install -g yarn

# Install node deps
yarn

# Build everything into /build
yarn run build

# Run the unit tests
yarn run test

Changelog

3.7.0 (2022-09-05)

Features

• allow transformation of logs with a callback function (6d4196c)
• remove CDK and let consumers build their own CDK (2b8f966)

Bug Fixes

• only log first 4KB of failed log lines (99963cb)
• substr is deprecated (acfca06)