aws-serverless-dataflow

aws-serverless-dataflow

Visualisation of AWS serverless (Lambda, API Gateway, SNS, SQS, etc.) components

aws-serverless-dataflow is a command line tool that can help you visualize the connections and dataflow among various AWS serverless components, such as AWS Lambda, API Gateway, SNS, SQS, etc. The tool surveys the serverless components in your AWS account, and then generates static website content that can be hosted on a website for viewing the diagrams.

The generated visual representation can help you to understand the architecture and data flow of their serverless application, and identify any issues or opportunities for optimization.

Quick start

aws-serverless-dataflow can be installed through Homebrew (brew install handy-common-utils/tap/aws-serverless-dataflow for Linux or MacOS), snap (snap install aws-serverless-dataflow for Linux except WSL), npm (npm i -g aws-serverless-dataflow for any system with Node.js installed), or manual download (https://github.com/james-hu/aws-serverless-dataflow/releases for Windows, Linux, and MacOS ). It can also be executed without installation through npx (npx aws-serverless-dataflow for any system with Node.js installed).

Before running it, you need to log into your AWS account through command line first.

Then, you can try something like this:

$ npx aws-serverless-dataflow -r ap-southeast-2 -i '*boi*' -i '*datahub*' -x '*jameshu*' -c -s
Surveyed 11/42 domains in API Gateway
Surveyed 72/224 queues in SQS
Surveyed 48/209 topics in SNS
Surveyed 65/250 subscriptions in SNS
Surveyed 100/1115 stacks in CloudFormation
Surveyed 120 APIs in API Gateway
(1/2) Surveying API Gateway, SQS, SNS and CloudFormation... done
Surveyed 85/410 buckets in S3
Surveyed 120/464 functions in Lambda
(2/2) Surveying S3 and Lambda... done
Finished survey in 72.672 seconds
Generating static website content in 'dataflow'... done
Local server started. Ctrl-C to stop. Access URL: http://localhost:8002/

The command line in the example above

1. searches for components in ap-southeast-2 region having boi or datahub but not jameshu in the ARN,
2. and finds out which CloudFormation stack each component belongs to,
3. stores the generated files in ./dataflow/ directory,
4. then launches a local web server at http://localhost:8002/ and opens the local website in the default browser.

This is what the website looks like:

There are a few options on the web page allowing further customisation of the diagram.

You can copy and host the generated static content files on your own website if you'd like to.

Command line option -r ap-southeast-2 specifies AWS region, -s tells the command line to start up a local web server and then open the local website in the default browser for viewing generated content.

If you don't want to include all the resources, you can use --include/-i and --exclude/-x options to specify which to include and which to exclude. Both of them can have multiple appearances in the command line. A resource would be included if any of the --include wild card patterns matches and none of the --exclude wild card patterns matches.

It may take a while for this tool to survey all relevant resources in your AWS account. To make it faster, you can try to increase parallelism by changing --parallelism/-l option which by default has a value of 4. If you see TooManyRequestsException: Rate exceeded error, you can try decreasing it.

-c or --cloud-formation would enable clustering resources by CloudFormation stacks. It is useful when you would like to have a high level view.

By passing -h or --help to the command line, you can see all supported arguments and options.

For more information on how to use aws-serverless-dataflow and the available options and arguments, visit https://github.com/james-hu/aws-serverless-dataflow or run the aws-serverless-dataflow --help command.

Usage

USAGE
  $ aws-serverless-dataflow  [PATH] [-v] [-h] [-r <value>] [-i
    <value>] [-x <value>] [-c] [-s] [-p <value>] [-l <value>] [-q] [-d]

ARGUMENTS
  PATH  [default: dataflow] path for putting generated website files

FLAGS
  -c, --cloud-formation      survey CloudFormation stack information (this takes
                             more time)
  -d, --debug                output debug messages
  -h, --help                 Show CLI help.
  -i, --include=<value>...   [default: *] wildcard patterns for domain names and
                             ARN of Lambda functions/SNS topics/SQS queues that
                             should be included
  -l, --parallelism=<value>  [default: 4] approximately how many AWS API calls
                             are allowed at the same time
  -p, --port=<value>         [default: 8002] port number of the local http
                             server for preview
  -q, --quiet                no console output
  -r, --region=<value>       AWS region (required if you don't have AWS_REGION
                             environment variable configured)
  -s, --server               start a local http server and open a browser for
                             pre-viewing generated website
  -v, --version              Show CLI version.
  -x, --exclude=<value>...   wildcard patterns for domain names and ARN of
                             Lambda functions/SNS topics/SQS queues that should
                             be excluded

DESCRIPTION
  Visualisation of AWS serverless (Lambda, API Gateway, SNS, SQS, etc.) dataflow

  This command line tool can visualise AWS serverless (Lambda, API Gateway, SNS,
  SQS, etc.) dataflow. It generates website files locally and can optionally
  launch a local server for you to preview.

  Before running this tool, you need to log into your AWS account (through
  command line like aws, saml2aws, okta-aws, etc.) first.

  This tool is free and open source:
  https://github.com/james-hu/aws-serverless-dataflow

EXAMPLES
  $ aws-serverless-dataflow -r ap-southeast-2 -s

  $ aws-serverless-dataflow -r ap-southeast-2 -s -i '*boi*' -i '*datahub*' \
        -x '*jameshu*' -c

  $ aws-serverless-dataflow -r ap-southeast-2 -s -i '*lr-*' \
        -i '*lead*' -x '*slack*' -x '*lead-prioritization*' \
        -x '*lead-scor*' -x '*LeadCapture*' -c

Contributing

Development:

• Run for test: ./bin/run ...
• Release: npm version patch -m "..." && npm publish

Please ignore main.go and go.mod files. They exist only because we are using goreleaser.

To build binaries for arm64 processors, you need a Linux machine, with binfmt and ldid installed:

• https://hub.docker.com/r/tonistiigi/binfmt
• https://stackoverflow.com/a/27769297

To debug goreleaser:

GITHUB_TOKEN=<the-token> goreleaser release --skip-validate --rm-dist --debug