DevOps has come a long way, and there is no doubt it will expand beyond product delivery to business value delivery and value stream delivery, enabling a broader digital transformation.

Platforms such as Docker and Kubernetes have helped companies ship their software faster than ever. With the ever-growing usage of containers to build and ship software, Kubernetes has gained colossal popularity among software enterprises as a de facto container orchestration tool.

Although out-of-the-box solutions can help you get a Kubernetes cluster up and running quickly, running a Kubernetes cluster optimized for production workloads is a challenge, especially for users with basic or intermediate knowledge. As your Kubernetes cluster grows, so does the complexity in managing it. To get the most out of your Kubernetes cluster and its features to support scaling, zero-downtime deployments, service discovery, automatic rollout, and rollback capabilities, prioritizing and implementing best practices in production are essential. Let’s look at some of the Kubernetes best practices in production.

Below are some of the best practices to have a Kubernetes cluster model that is scalable, secured, and highly optimized.

Using namespaces

Adopting Kubernetes in a large Enterprise with multiple teams accessing the same cluster requires a custom approach to resource usage. Improper provisioning often leads to conflicts among teams for resource usage.

To solve this, use multiple namespaces to achieve team-level isolation for teams trying to access the same cluster resources concurrently. In addition, the efficient use of Namespaces helps create multiple logical cluster partitions, thereby allocating distinct virtual resources among teams.

Resource requests & limits

Kubernetes’ concepts of resource limits and requests allow for efficient resource utilization when properly used. However, knowing what values to use when configuring resource limits and requests for your workloads can be challenging.

Setting your requests and limits too low on an application will cause problems. For example, if your memory limits are too low, Kubernetes is bound to kill your application for violating its limits. 

Meanwhile, setting your requests and limits too high results in resource over-allocation. So you waste resources and end up with a higher bill.  

Also, to avoid cases where a single team or application drains all the cluster resources, set requests and limits for cluster resources, specifically CPU and Memory. Doing so limits excessive resource usage by applications and services, thereby avoiding capacity downtime.

To set requests and limits on a container, you may use the following container spec as a reference:

containers:

– name: nginx

image: nginx

resources:

requests:

memory: “512Mi”

CPU: “256m”

limits:

memory: “1024Mi”

CPU: “512m”

The above container spec allocates resource requests to 512 MiB of memory and 256 mCPU, limiting a maximum of 512 mCPU and 1024 MiB to the nginx container.

ReadinessProbe & LivenessProbe

Kubernetes offers two mechanisms to track the lifecycle of your containers and Pods: liveness and readiness probes.

The readiness probe determines when a container can receive traffic.

Kubernetes checks if the application is ready to start serving traffic with a readiness probe for production apps before allowing traffic to a pod. This essentially indicates whether a pod is available to accept traffic and respond to requests.

The liveness probe determines when a container should be restarted.

Through a livenessProbe, Kubernetes performs a health check to ensure the application is responsive and running as intended. In the event the livenessProbe fails, the kubelet default policy restarts the container to bring it back up.

RBAC for security

Role-based Access Controls (RBAC) help administer access policies to define who can do what on the Kubernetes cluster. To set RBAC permissions on Kubernetes resources, Kubernetes provides these parameters:

The role for a namespaced resource

ClusterRole for a non-namespaced resource

Here is an example where ClusterRole is used to administer read access to services on all namespaces:

Copy

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

name: reader

rules:

– apiGroups: [“”]

resources: [“services”]

verbs: [“get”,”watch”,”list”]

Kubernetes also allows RoleBinding and ClusterRoleBinding by referencing roles already administered on a user, group, or service account. More details on Kubernetes RBAC policies can be found here.

Autoscaling

It is strongly recommended that you leverage benefits from Kubernetes’ autoscaling mechanisms to scale cluster services with a surge in resource consumption automatically.

With Horizontal Pod Autoscaler and Cluster Autoscaler, node and pod volumes get adjusted dynamically in real-time, thereby maintaining load to the optimum level and avoiding capacity downtimes.

Container image Size

One of the main concerns for container size increases is using the default base docker images. This makes containers significant and can potentially duplicate dependencies both in the container image and the application (If the application installs them locally).

Another reason for this is if we use the build tools within the container of the running application. This increases the size of the container and installs dependencies that might not be needed for the runtime application. If you are concerned about their size, some of these container images could be reduced by several factors.

• Start with a lean base image tailored to your application.
• Use multistage builds for languages that require runtime environments, such as Java. Eliminate the libraries and dependencies used during the build from the final deployed container image, and include only the artifacts and the environment needed to run them.
• Exploit layering to create base images shared among multiple images with lots of standard code.

Monitor control plane components

Monitoring the Kubernetes control plane. The control plane includes:

the Kubernetes API Service, Kubelet, controller-manager, etcd, kube-proxy, kube-dns

Monitoring the control plane helps identify issues/threats within the cluster and increases its latency. It is also recommended to use automated monitoring tools rather than manually managing the alerts.

Kubernetes introduces several abstractions on both the application and the infrastructure layer. These include infrastructure levels like namespaces, clusters, and nodes on the application layer like pods and containers.

Additionally, the trend towards micro-service architectures and that containers and pods are ephemeral entities introduce further monitoring challenges. A monitoring solution needs to consider all of these when choosing which metrics to monitor and the tools to be used.