cloud-governance

Cloud Governance

What is it?

Cloud Governance tool provides a lightweight and flexible framework for deploying cloud management policies focusing on cost optimize and security. We have implemented several pruning policies.
When monitoring the resources, we found that most of the cost leakage is from available volumes, unused NAT gateways, and unattached Public IPv4 addresses (Starting from February 2024, public IPv4 addresses are chargeable whether they are used or not).

Providers	Disks	NatGateway	PublicIp	Snapshots	InstanceIdle	TagResources	EC2Stop	ocp_cleanup	ClusterRun	EmptyBucket	EmptyRoles
AWS	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Azure	✓	✓	✓	✓	✓	✗	✗	✗	✓	✗	✗

List of Policies:

AWS Polices!

• instance_idle
• instance_run
• unattached_volume
• zombie_cluster_resource
• ip_unattached
• zombie_snapshots
• unused_nat_gateway
• s3_inactive
• empty_roles
• tag_resources
• tag_iam_user
• cost_over_usage
• cluster_run

Azure Polices!

• instance_idle
• unattached_volume
• ip_unattached
• unused_nat_gateway

IBM Polices!

• tag_baremetal
• tag_vm
• tag_resources

Check out policy summary here!

Reference:

• Checkout blog: Optimizing cloud resource management with cloud governance
• The cloud-governance package is placed in PyPi
• The cloud-governance container image is placed in Quay.io
• The cloud-governance readthedocs link is ReadTheDocs

Table of Contents

• Installation
• Configuration
• Run Policies
• Run Policy Using Pod
• Pytest
• Post Installation

Installation

Download cloud-governance image from quay.io

podman pull quay.io/cloud-governance/cloud-governance

Environment variables configurations:

Key	Value	Description
AWS_ACCESS_KEY_ID	required	AWS access key
AWS_SECRET_ACCESS_KEY	required	AWS Secret key
AWS_DEFAULT_REGION	required	AWS Region, default set to us-east-2
BUCKET_NAME	optional	Cloud bucket Name, to store data
policy	required	check here for policies list
dry_run	optional	default set to "yes", supported only two: yes/ no
log_level	optional	default set to INFO
LDAP_HOST_NAME	optional	ldap hostnames
es_host	optional	Elasticsearch Host
es_port	optional	Elasticsearch Port
es_index	optional	Elasticsearch Index, to push the data. default to cloud-governance-es-index
GOOGLE_APPLICATION_CREDENTIALS	optional	GCP creds, to access google resources. i.e Sheets, Docs
AZURE_CLIENT_SECRET	required	Azure Client Secret
AZURE_TENANT_ID		Azure Tenant Id
AZURE_ACCOUNT_ID		Azure Account Id
AZURE_CLIENT_ID		Azure Client Id
GCP_DATABASE_NAME		GCP BigQuery database name, used to generate cost reports
GCP_DATABASE_TABLE_NAME		GCP BigQuery TableName, used to generate cost reports
IBM_API_USERNAME		IBM Account Username
IBM_API_KEY		IBM Account Classic Infrastructure key
IBM_CLOUD_API_KEY		IBM Cloud API Key
IBM_CUSTOM_TAGS_LIST		pass string with separated with comma. i.e: "cost-center: test, env: test"

AWS Configuration

Create IAM User with Read/Delete Permissions and create S3 bucket.

• Follow the instructions README.md.

IBM Configuration

• Create classic infrastructure API key
• Create IBM CLOUD API key to use tag_resources policy

Run Policies

AWS

• Passing environment variables

  podman run --rm --name cloud-governance \
  -e policy="zombie_cluster_resource" \
  -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
  -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
  -e AWS_DEFAULT_REGION="us-east-2" \
  -e dry_run="yes"  \
   "quay.io/cloud-governance/cloud-governance"

• Using involvement file config
• Create env.yaml file, and mount it to /tmp/env.yaml else mount to anypath and pass env DEFAULT_CONFIG_PATH where you mounted

AWS_ACCESS_KEY_ID: ""
AWS_SECRET_ACCESS_KEY: ""
AWS_DEFAULT_REGION: "us-east-2"
policy: "zombie_cluster_resource"
dry_run: "yes"
es_host: ""
es_port: ""
es_index: ""

  podman run --rm --name cloud-governance \
  -v "env.yaml":"/tmp/env.yaml" \
  --net="host" \
   "quay.io/cloud-governance/cloud-governance"

Run Policy Using Pod

Run as a pod job via OpenShift

Job Pod: cloud-governance.yaml

Configmaps: cloud_governance_configmap.yaml

Quay.io Secret: quayio_secret.sh

AWS Secret: cloud_governance_secret.yaml

* Need to convert secret key to base64 [run_base64.py](pod_yaml/run_base64.py)

Pytest

Cloud-governance integration tests using pytest

python3 -m venv governance
source governance/bin/activate
(governance) $ python -m pip install --upgrade pip
(governance) $ pip install coverage
(governance) $ pip install pytest
(governance) $ git clone https://github.com/redhat-performance/cloud-governance
(governance) $ cd cloud-governance
(governance) $ coverage run -m pytest
(governance) $ deactivate
rm -rf *governance*

Post Installation

Delete cloud-governance image

sudo podman rmi quay.io/cloud-governance/cloud-governance