K8s Security : References from Kubecon2019

Extended NodeRestrictions for Pods: https://bit.ly/2XdeWOF
Bounding Self-Labeling Kubelets: https://bit.ly/351BaFN
Choose a minimal base image https://bit.ly/37eTPzT
Run as non root! https://bit.ly/2qpUNJ7 
Use resource limits https://bit.ly/37k48Tx 
Use least privilege authorization https://bit.ly/2CV1INd 
Restrict network access https://bit.ly/37cL9dv 
Node Authorizer: https://bit.ly/33XRIPb
Node Restriction: https://bit.ly/2QkRqhk
Kubelet Static Pods: https://bit.ly/2Qj0DGL
Extended NodeRestrictions for Pods: https://bit.ly/2XdeWOF
Bounding Self-Labeling Kubelets: https://bit.ly/351BaFN
ReplicaSet deletion logic: https://bit.ly/2NQTL1O
Run as non-root using security context https://bit.ly/2qpUNJ7
Minimal base images: https://bit.ly/37eTPzT
Resource limits: https://bit.ly/37k48Tx
Least privilege: https://bit.ly/2CV1INd
GKE hardening guide: g.co/gke/hardening
GKE sandboxes: g.co/gke/sandbox
Kata containers: katacontainers.io
State of Kubernetes Security https://bit.ly/2OdqgWC
“The Devil in the Details: Kubernetes’ First Security Assessment”
Walls Within Walls: What If Your Attacker Knows Parkour?”
“Binary Authorization in Kubernetes” https://bit.ly/32L2yqj
“Piloting Around the Rocks: Avoiding Threats in Kubernetes”
“Hello from the Other Side: Dispatches from a Kubernetes
Attacker” https://bit.ly/2NBpe7Y
“How Kubernetes Components Communicate Securely in Your
Cluster” https://bit.ly/2QrIzKP
“Sig-Auth Update” https://bit.ly/2Kk7kEQ
“Attacking and Defending Kubernetes Clusters: A Guided Tour”

kubectl productivity

Auto Complete

source <(kubectl completion bash) # setup autocomplete in bash into the current shell, bash-completion package should be installed first.

echo "source <(kubectl completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.


kubectx helps you switch between clusters back and forth:

kubens helps you switch between Kubernetes namespaces smoothly:


kubectl explain
command outputs the specification of the requested resource or field.


A script to generate hundreds of convenient kubectl aliases programmatically.

Syntax explanation
  • k=kubectl
    • sys=--namespace kube-system
  • commands:
    • g=get
    • d=describe
    • rm=delete
    • a:apply -f
    • exexec -i -t
    • lologs -f
  • resources:
    • po=pod, dep=deploymenting=ingresssvc=servicecm=configmapsec=secretns=namespaceno=node
  • flags:
    • output format: oyamlojsonowide
    • all--all or --all-namespaces depending on the command
    • sl--show-labels
    • w=-w/--watch
  • value flags (should be at the end):
    • n=-n/--namespace
    • f=-f/--filename
    • l=-l/--selector


eBPF, OPA, Blackbox exporter, ffwd, Heroic

eBPF can be used for 

1. Map application and HA architecture
2. Detect network issues
3. Identify misbehaving svc


Open Policy Agent https://github.com/open-policy-agent/opa can be used to validate CRD

The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP. https://github.com/prometheus/blackbox_exporter

ffwd is a flexible metric forwarding agent. It is intended to run locally on the system and receive metrics through a wide set of protocols and then forward them to your TSDB.  https://github.com/spotify/ffwd 

Heroic A scalable time series database based on Bigtable, Cassandra, and Elasticsearch. https://github.com/spotify/heroic


Easily Observing Operators

kube-state-metrics you can gather the following state about your cluster:
  • Counts of each object type
  • All of the Kubernetes labels and their values attached to each object
  • The creation time (as an epoch) of each object
  • Some generic, object specific “info”
  • Other states specific to the object in question

kube-state-metrics can be deployed like a classic Kubernetes service with only one replica.

List of metrics

Metrics about your CRD

Kustomize plugins

This is the third article out of three articles on Kubernetes tool : Kustomize. This article covers the plugins. 

kustomize plugins

Kustomize offers a plugin framework allowing people to write their own resource generators and transformers.


- gen_file.yaml
- trans_file.yaml

Let's focus on gen_file. Trans_file will be similar. 


apiVersion: "apiVersion"
kind: Gen_File
  name: "some name"

Now the file name "Gen_File" will be searched at path  
XDG_CONFIG_HOME = $HOME/.config = /home/manish.config
Possible value for apiVersion = someteam.example.com/v1

If failed then "Gen_File.so" will be searched at same path

This file will be invoked with gen_file.yaml

Reference : https://github.com/kubernetes-sigs/kustomize/tree/master/docs/plugins

Built-in plugins : https://github.com/kubernetes-sigs/kustomize/tree/master/plugin/builtin

Plugin Development