Skip to content

Kubernetes scaling exploration

Some options to create "workloads" to exercise the kube cluster scale up/down in OpenShift.

Requirements

Install clients

  • openshift-install
  • oc

Optional:

  • kustomize
wget -O /tmp/kustomize.tgz  https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.3.0/kustomize_v5.3.0_linux_amd64.tar.gz
tar xvfz /tmp/kustomize.tgz  -C ~/bin/
  • k9s
wget -qO /tmp/k9s.tgz https://github.com/derailed/k9s/releases/download/v0.29.1/k9s_Linux_amd64.tar.gz
tar xvfz /tmp/k9s.tgz  -C /tmp/ && mv /tmp/k9s ~/bin/k9s

Install OpenShift cluster on AWS (default)

CLUSTER_NAME=aws-as-06
INSTALL_DIR=${PWD}/installdir-${CLUSTER_NAME}
mkdir $INSTALL_DIR

cat << EOF > $INSTALL_DIR/install-config.yaml
apiVersion: v1
metadata:
  name: $CLUSTER_NAME
publish: External
pullSecret: '$(cat $PULL_SECRET_FILE)'
sshKey: |
  $(cat ~/.ssh/id_rsa.pub)
baseDomain: devcluster.openshift.com
platform:
  aws:
    region: us-east-1
EOF
./openshift-install create cluster --dir $INSTALL_DIR

Setup OpenShift cluster auto scaling

Setup the cluster autoscaler used to run the tests:

Deploy monitor kube-ops-view

kube-ops-view utility helps to visualize (graphical web UI) the nodes and pod allocation within the cluster.

  • Deploy with in OpenShift cluster:
oc apply -k deploy-ops-view

Get the route and open in your browser:

oc get route -n ocp-ops-view

Example: https://proxy-ocp-ops-view.apps.aws-as-03.devcluster.openshift.com/

Deploy scaling monitor costs

TBD: How?

Run tests

Tests using the 'inflate app'

inflate app is merely pause pods with fixed size defined with kustomize to quickly test the nodes autoscaling.

  • Deploy the base app
oc apply -k deploy-inflate/

oc get all -n lab-scaling

OR deploy without tolerations:

oc apply -k deploy-inflate-no-taint/

oc get all -n lab-scaling
  • Warm up the scale (deploy single pod):
oc apply -k overlay/inflate-00-up-1

oc get pods -n lab-scaling

Check if the Cluster Autoscaler triggered the Machine creation through MachineAutoscaling/MachineSet.

  • Inflate the cluster to 12 pods (medium?)

12 CPU + 12 GiB Mem. 12 == (~4 regular nodes of 4CPU 16GiB [~3GiB] + cluster overhead + spare)

oc apply -k overlay/inflate-00-up-12/

oc get pods -n lab-scaling -o wide -w
  • Inflate the cluster to 25 pods (heavy)
oc apply -k overlay/inflate-00-up-25/

oc get pods -n lab-scaling -o wide -w

Check if there are nodes scaling:

oc get machines -n openshift-machine-api -w
oc get nodes
  • Scale down to single replica:
oc apply -k overlay/inflate-00-up-1
  • Scale down to 0 replica (expect to remove nodes):
oc apply -k overlay/inflate-00-up-0

Tests using the with kube-burner

kube-burner is a tool to stress kubernetes clusters.

  • Install
wget -qO /tmp/kube-burner.tgz  https://github.com/cloud-bulldozer/kube-burner/releases/download/v1.8.0/kube-burner-V1.8.0-linux-x86_64.tar.gz

tar xfz /tmp/kube-burner.tgz -C /tmp && mv /tmp/kube-burner ~/bin
  • Run cluster-density-v2
kube-burner ocp cluster-density-v2 --iterations=1 --churn-duration=5m0s  

kube-burner ocp cluster-density-v2 --iterations=50 --churn-duration=5m0s
  • Run cluster-density-v2
kube-burner ocp node-density --pods-per-node=150 --burst 50
  • (TODO?) Run the web-burner-cluster-density

References

Other future explorations: