github.com/kubernetes-sigs/multi-tenancy/benchmarks/kubectl-mtb

kubectl-mtb

Setup | Usage | Contributing|

kubectl plugin to validate multi-tenancy configuration for a Kubernetes cluster.

The mtb kubectl plugin provides behavioral and configuration checks to help validate if a cluster is properly configured for multi-tenant use.

Demo

Setup

Prerequisites : Make sure you have working Golang environment.

kubectl-mtb can be installed by cloning and building this repository:

git clone https://github.com/kubernetes-sigs/multi-tenancy
cd benchmarks/kubectl-mtb
make kubectl-mtb

The kubectl-mtb binary will be copied to your $GOPATH/bin directory.

Usage

List benchmarks:

kubectl-mtb get benchmarks

Run benchmarks:

kubectl-mtb run benchmarks -n "namespace" --as "user"

Run a specific benchmark by ID:

kubectl-mtb run benchmark MTB-PL2-CC-TI-1 -n "namespace" --as "user"

Complete example

Create a namespace

kubectl create ns "test"

Create a namespaced user role

You can use the following template to create a namespace admin role binding for a user (allie) in the namespace you want to test:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: allie
subjects:
- kind: User
  name: allie # "name" is case sensitive
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io

kubectl create -n "test" -f allie.yaml

Run benchmarks

kubectl-mtb run benchmarks -n "test" --as "allie"

Most of the benchmarks will fail, a few will pass as the user cannot access cluster resources:

Install Kyverno or Gatekeeper to secure the cluster

You can use a policy engine like Kyverno or Gatekeeper for conformance with the benchmarks. We are currently maintaining and policies for both here.

Kyverno

To install Kyverno, you can run the following command:

kubectl create -f https://github.com/nirmata/kyverno/raw/master/definitions/install.yaml

To apply all the Kyverno policies after installing, you can use the following command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/multi-tenancy/master/benchmarks/kubectl-mtb/test/policies/kyverno/all_policies.yaml

You can learn more about Kyverno here.

Gatekeeper

To install Gatekeeper, run following command:

 kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

You can find the policies of Gatekeeper here

You can refer here to know how to use Gatekeeper.

Configure Quotas and Limits

For conformance with benchmarks like Configure namespace resource quotas, the namespace will also need a ResourceQuota object. To create the quota, run the following command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/multi-tenancy/master/benchmarks/kubectl-mtb/test/quotas/ns_quota.yaml

After applying the policies and ResourceQuota object, run the benchmarks again. All benchmarks should pass.

List Policy Reports

You can output the benchmark results as a Policy Report

Install the Policy Report CR:

kubectl create -f https://github.com/kubernetes-sigs/wg-policy-prototypes/raw/master/policy-report/crd/policy.kubernetes.io_policyreports.yaml

Run the benchmarks with the -o policyreport flag:

kubectl-mtb run benchmarks -n "tenantnamespace" --as "user impersonation" -o policyreport

Contributing

You can use mtb-builder to add new benchmarks.

Clone the repo:

git clone https://github.com/kubernetes-sigs/multi-tenancy
cd benchmarks/kubectl-mtb

Run the following command to build mtb-builder:

make builder

The generated binary will create the relevant templates, needed to write the bechmark as well as associated unit test.

Example :

./mtb-builder create "block multitenant resources" -p 1

Here, block multitenant resources is name of the benchmark and -p flag is used here to mention the profile level. The above command will generate a directory named block_multitenant_resources under which following files would be present.

• config.yaml
• block_multitenant_resources_test.go
• block_multitenant_resources.go

Generate README

A README.md can be dynamically generated for the benchmarks from config.yaml (present inside each benchmark folder). You can add additional fields in the config.yaml, and they will be reflected in the README.md after running the following command from cloned repo.

make readme

Run unit tests

• The unit tests run on a separate kind cluster. To run all the unit test you can run the command make unit-tests this will create a new cluster if it cannot be found on your machine. By default, the cluster is named kubectl-mtb-suite, after the tests are done, the cluster will be deleted.

• If you want to run a particular unit test, you can checkout into the particular benchmark directory and run go test which will create a cluster named kubectl-mtb which will be deleted after the tests are completed.

If kind cannot be found on your system the target will try to install it using go get