cid-cmd

Cloud Intelligence Dashboards (CUDOS Framework)

Welcome to Cloud Intelligence Dashboards (CUDOS Framework) automation repository

This repository contains CloudFormation templates, Terraform modules, and a Command Line tool (cid-cmd) for managing various dashboards provided in AWS Well Architected LAB Cloud Intelligence Dashboards.

There are several ways we can manage dashboards:

1. CloudFormation Template (using cid-cmd tool in lambda)
2. Terraform module (wrapper around CloudFormation Template)
3. Using cid-cmd tool from command line

We recommend cid-cmd tool via AWS CloudShell.

Supported dashboards

---

Dashboard documentation	Demo URL	Prerequisites URL
CUDOS Dashboard	demo	link (Steps 1 and 2)
Cost Intelligence Dashboard	demo	link (Steps 1 and 2)
Trusted Advisor Organisation (TAO) Dashboard	demo	link
Trends Dashboard	demo	link (Steps 1 and 2)
KPI Dashboard	demo	link (Steps 1 and 2)
Compute Optimizer Dashboard	demo	link
Cost Anomaly Dashboard	demo	link
Data Transfer Cost Dashboard	demo	link (Steps 1 and 2)
AWS Budgets Dashboard	demo	link

See more dashboards on the workshop page.

Before you start

1. :heavy_exclamation_mark: Complete the prerequisites for respective dashboard (see above).
2. :heavy_exclamation_mark: Specifying a Query Result Location Using a Workgroup
3. :heavy_exclamation_mark: Make sure QuickSight Enterprise edition is activated.

Command line tool cid-cmd

Demo: Deployment of Dashboards with cid-cmd tool

Install

1. Launch AWS CloudShell or your local shell

   Automation requires Python 3

2. Make sure you have the latest pip package installed

   python3 -m ensurepip --upgrade
   

3. Install CID Python automation PyPI package

   pip3 install --upgrade cid-cmd
   

Dashboard Deployment

cid-cmd deploy

Update existing Dashboards

Update only Dashboard

cid-cmd update

Update dashboard and all dependencies (Datasets and Athena View). WARNING: this will override any customization of SQL files and Datasets.

cid-cmd update --force --recursive

Show Dashboard Status

Show dashboards status

cid-cmd status

Share QuickSight resources

cid-cmd share

Initialize Amazon QuickSight

One time action to initialize Amazon QuickSight Enterprise Edition.

cid-cmd init-qs

Initialize CUR

One time action to initialize Athena table and Crawler from s3 with CUR data. Currently only CUR1 supported.

cid-cmd init-cur

Create CUR proxy

There are 2 CUR formats that cid-cmd supports. CUR1 and CUR2 have slightly different fields structure. Each dashboard can be developed using CUR2 or CUR1, but on installation cid-cmd will deploy a special Athena View "CUR proxy". This View will play a role of compatibility layer between the CUR version used by Dashboard and available CUR. This View can be extended as new dashboard request CUR fields.

Please note that Cost Allocation Tags and Cost Categories are not present in CUR Proxy by default. You will need to add these fields by modifying CUR Proxy in Athena or using cid-cmd tool:

For CUR2 proxy (using CUR1 data as source)

cid-cmd create-cur-proxy -vv \
   --cur-version '2' \
   --cur-table-name 'mycur1' \
   --athena-workgroup 'primary' \
   --fields "resource_tags['user_cost_center'],resource_tags['user_owner']"

For CUR1 proxy (using CUR2 data as source)

cid-cmd create-cur-proxy -vv \
   --cur-version '1' \
   --cur-table-name 'mycur2' \
   --athena-workgroup 'primary' \
   --fields "resource_tags_user_cost_center,resource_tags_user_owner"

Delete Dashboard and all dependencies unused by other

Delete Dashboards and all dependencies unused by other CID-managed dashboards.(including QuickSight datasets, Athena views and tables)

cid-cmd delete

Delete Command Options:

 --dashboard-id TEXT QuickSight dashboard id
 --athena-database TEXT Athena database

Export

The command export lets you download or share a customized dashboard with another AWS Account. It takes the QuickSight Analysis as an input and generates all the assets needed to deploy your Analysis into another AWS Account. This command will generate a yaml file with a description of the Dashboard and all required Datasets. Also this command generates a QuickSight Template in the current AWS Account that can be used for Dashboard deployment in other accounts. The resource file can be used with all other cid commands. Both accounts must have relevant Athena Views and Tables.

Export from account A:

cid-cmd export

Deployment to account B:

cid-cmd deploy --resources ./mydashboard.yaml

See available commands and command line options

cid-cmd --help

CSV to Athena View

Generate and SQL file for Athena View for CSV file

cid-cmd csv2view --input my_mapping.csv --name my_mapping

This command generates a SQL file that you can execute. Please mind Athena Service Limit for Query Size.

Cloud Formation

CID is also provided in a form of CloudFormation templates. See detailed instructions in the Well Architected Labs site.

Terraform

CID offers a set of Terraform modules to deploy CUR replication and CID dashboards. Refer to these deployment instructions for details of how to deploy CID dashboards with these modules.

1. Create a bucket for consolidating CUR terraform-modules/cur-setup-destination/
2. Create a CUR in Payer Account(s) terraform-modules/cur-setup-source/
3. Create Dashboards (wrapper around CloudFormation) terraform-modules/cid-dashboards/

Rights Management

The ownership of CID is usually with the FinOps team, who do not have administrative access. However, they require specific privileges to install and operate CID dashboards. To assist the Admin team in granting the necessary privileges to the CID owners, a CFN template is provided. This template, located at CFN template, takes an IAM role name as a parameter and adds the required policies to the role.

Troubleshooting and Support

If you experience unexpected behaviour of the cid-cmd script please run cid-cmd in debug mode:

cid-cmd -vv [command]

This will produce a log file in the same directory that were at the tile of launch of cid-cmd.

:heavy_exclamation_mark:Inspect the produced debug log for any sensitive information and anonymize it.

We encourage you to open new issue with description of the problem and attached debug log file.