Kubernetes - Best Practices & Lessons Learned

6 min readApr 24, 2019

Part 2 of 3

In Part 1 — Why use Kubernetes, I went over Monolithic vs Microservices Architecture and the benefits of Immutable Infrastructure. In this post, I want to share some of my best practices and lessons learned from using K8s.

The items I will cover are:

Resource Limits
Namespaces
ServiceTypes
Init Containers and Lifecycle Hooks
Secrets
Labels

Set Resource Limits

By default, a pod will spin up with no resource limits. Based on the amount of CPU and memory available on your K8s worker nodes, the pods will use as much resource as necessary from the running applications. For development and testing, NOT setting limits could be useful if you want to test the limits of your application. In a production environment however, setting limits is crucial because if you have an unexpected spike or bug in one container, it could take up all the resources on that worker node and thus affecting the ability for other containers and pods to use additional CPU and memory. The worker node won’t be able to spin up any additional pods either. There are other ways to prevent this from happening such as using Horizontal Pod Scaler (HPA) to automatically scale your pods or the Cluster Autoscaler to scale your worker nodes.

The best practice is to set the resource request and limit for all of your containers. In addition, there should be proper monitoring and alerting for your usage.

Request — the amount of resource guaranteed for each container
Limit — maximum amount the resource is able to use

The Downward API is a little-known functionality of Kubernetes that allows you to expose Pod and Container fields to a process running in a container. With this, we can create a container that inspects its own resource requests and automatically sets its heap size appropriately.

Extra note: if it’s a Java application, your JVM and heap settings will also need to be tuned as being in a container/pod may have some differences than running the application on an instance or physical server.

A useful resource on additional JVM tuning can be found here: https://very-serio.us/2017/12/05/running-jvms-in-kubernetes/

Use namespaces for better management, allocation, monitoring

You can think of a Namespace as a virtual cluster inside your Kubernetes cluster. You can have multiple namespaces inside a single Kubernetes cluster, and they are all logically isolated from each other. They can help your team with organization, security, and even performance. If your application is simple, then you can put everything into one namespace, but there are more benefits in having clear and organized namespaces. It is always better to plan for the future from the start.

Services in namespaces can talk to other namespaces. You can reference the service names using this: <Service Name>.<Namespace Name>.svc.cluster.local

ServiceTypes

Kubernetes defines Services as an abstraction layer for pod communication.

You can specify IP addresses, ports, protocols, or none of the above as a headless service.

Most importantly, you will probably want to expose your service externally with ServiceTypes.

ClusterIP: This is the default ServiceType where every service created gets assigned an internal cluster IP Address.
NodePort: The NodePort configuration works by opening a static port on each node’s external networking interface. Traffic to the external port will be routed automatically to the appropriate pods using an internal cluster IP service.
LoadBalancer: This is probably the most common use case as the cloud provider’s load balancer is automatically created with this service. Nodeport and ClusterIP services are also automatically created.
ExternalName: A CName record can also be used where the service is mapped to the ExternalName field.

The best and standard way for external access is to use LoadBalancer for services since its the easiest and most convenient. If you expose a lot of services however, you have to keep in mind that each LB created will incur costs from the cloud provider. Using Ingress may be more effective as an alternative but it is actually NOT a type of service. Instead, it sits in front of multiple services and act as a “smart router” or entrypoint into your cluster. This is the most powerful but most complicated. You can use the same IP address and expose multiple services. You can even use Ingress plugins for SSL cert management. Added complexity gives more options and customizations but also adds more to manage. In AWS, using ELBs are great because it is completely managed by AWS and adding SSL authentication is super easy. Automating this in K8s with LoadBalancer ServiceType is also super easy.

Init Containers and Lifecycle hooks

When planning your creation deployments and automation, there may be initiation steps required for your application and/or database. Additionally, your application may also require proper shutdown steps. Your existing application may have these steps built in already but after migrating to containers and pods, those steps may not work properly and might require some adjustments in order for it to work properly.

Init Containers

Init Containers always run before the application containers in a pod. These can have separate images from the application and are very similar to Jobs as they also run to completion. Init Containers are guaranteed to execute before the pod initializes or else the pod will restart.

Jobs — run to completion

There is also the concept of a Job type in Kubernetes. A Job can create one or more pods that runs to completion. It can even be run in parallel. You can run Init Containers in a Job but you can also run regular containers as well.

Lifecycle hooks

Lifecycle hooks may be more useful if you are looking for adjustments to your existing deployment and containers. There are two lifecycle hooks:

PostStart hook executes immediately after container is created.

PreStop hook is called just before the container is terminated. Prestop hooks are great for proper shutdown scripts for services.

There are two types of handlers for the lifecycle hooks. Exec handler is used for calling shell command/script while HTTP handler is used for calling HTTP endpoint.

Kubectl Drain is also a good way to schedule evicting running pods as it marks the pod unscheduleable or decommissioned. Adding a sleep 5 in the Lifecycle is recommended to allow for proper pod eviction so that it propagates before doing the graceful shutdown for the drain to complete.

Secrets management

The Kubernetes secrets feature is great for basic secret keys but it is only base64 encoded and not actually encrypted. One addon, I would recommend is to use Helm secrets — https://github.com/futuresimple/helm-secrets

It has the benefit of implementing a GIT workflow, allows for versioning, and you can easily rotate the keys. It works very similar to Vault as the containers pull the secrets using an encryption key to decrypt, encrypt, and edit. It uses Mozilla Sops as the backend as it encrypts the values the yaml or json file and not the actual keys.

Labels

I highly recommend using labels for Node objects to allow targeting pods to specific nodes or groups of nodes. Nodeselector/Node Affinity/Anti-Affinity provides a very simple way to constrain pods to nodes with particular labels.

Labels can also be great for versioning as you define it in the yaml file and object metadata. Labels are given as simple key-value pairs so each unit can have more than one label, but each unit can only have one entry for each key such as for environment, versioning, etc.

Hopefully these suggestions are helpful for anyone starting out in using Kubernetes. K8s definitely has a learning curve and taking advantage of all the available features will help significantly when planning for a proper environment.

You can read about some of the reasons why I feel like Kubernetes is the wave of the future in Part 1 — Why use Kubernetes, where I go over Monolithic vs Microservices Architecture and the benefits of Immutable Infrastructure.

In Part 3-Production Checklist, I go over my checklist that I think could be helpful in implementing K8s in production.

Resource References: