K8s Security : References from Kubecon2019

Extended NodeRestrictions for Pods: https://bit.ly/2XdeWOF
Bounding Self-Labeling Kubelets: https://bit.ly/351BaFN
Choose a minimal base image https://bit.ly/37eTPzT
Run as non root! https://bit.ly/2qpUNJ7 
Use resource limits https://bit.ly/37k48Tx 
Use least privilege authorization https://bit.ly/2CV1INd 
Restrict network access https://bit.ly/37cL9dv 
Node Authorizer: https://bit.ly/33XRIPb
Node Restriction: https://bit.ly/2QkRqhk
Kubelet Static Pods: https://bit.ly/2Qj0DGL
Extended NodeRestrictions for Pods: https://bit.ly/2XdeWOF
Bounding Self-Labeling Kubelets: https://bit.ly/351BaFN
ReplicaSet deletion logic: https://bit.ly/2NQTL1O
Run as non-root using security context https://bit.ly/2qpUNJ7
Minimal base images: https://bit.ly/37eTPzT
Resource limits: https://bit.ly/37k48Tx
Least privilege: https://bit.ly/2CV1INd
GKE hardening guide: g.co/gke/hardening
GKE sandboxes: g.co/gke/sandbox
Kata containers: katacontainers.io
State of Kubernetes Security https://bit.ly/2OdqgWC
“The Devil in the Details: Kubernetes’ First Security Assessment”
Walls Within Walls: What If Your Attacker Knows Parkour?”
“Binary Authorization in Kubernetes” https://bit.ly/32L2yqj
“Piloting Around the Rocks: Avoiding Threats in Kubernetes”
“Hello from the Other Side: Dispatches from a Kubernetes
Attacker” https://bit.ly/2NBpe7Y
“How Kubernetes Components Communicate Securely in Your
Cluster” https://bit.ly/2QrIzKP
“Sig-Auth Update” https://bit.ly/2Kk7kEQ
“Attacking and Defending Kubernetes Clusters: A Guided Tour”

kubectl productivity

Auto Complete

source <(kubectl completion bash) # setup autocomplete in bash into the current shell, bash-completion package should be installed first.

echo "source <(kubectl completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.


kubectx helps you switch between clusters back and forth:

kubens helps you switch between Kubernetes namespaces smoothly:


kubectl explain
command outputs the specification of the requested resource or field.


A script to generate hundreds of convenient kubectl aliases programmatically.

Syntax explanation
  • k=kubectl
    • sys=--namespace kube-system
  • commands:
    • g=get
    • d=describe
    • rm=delete
    • a:apply -f
    • exexec -i -t
    • lologs -f
  • resources:
    • po=pod, dep=deploymenting=ingresssvc=servicecm=configmapsec=secretns=namespaceno=node
  • flags:
    • output format: oyamlojsonowide
    • all--all or --all-namespaces depending on the command
    • sl--show-labels
    • w=-w/--watch
  • value flags (should be at the end):
    • n=-n/--namespace
    • f=-f/--filename
    • l=-l/--selector


eBPF, OPA, Blackbox exporter, ffwd, Heroic

eBPF can be used for 

1. Map application and HA architecture
2. Detect network issues
3. Identify misbehaving svc


Open Policy Agent https://github.com/open-policy-agent/opa can be used to validate CRD

The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP. https://github.com/prometheus/blackbox_exporter

ffwd is a flexible metric forwarding agent. It is intended to run locally on the system and receive metrics through a wide set of protocols and then forward them to your TSDB.  https://github.com/spotify/ffwd 

Heroic A scalable time series database based on Bigtable, Cassandra, and Elasticsearch. https://github.com/spotify/heroic


Easily Observing Operators

kube-state-metrics you can gather the following state about your cluster:
  • Counts of each object type
  • All of the Kubernetes labels and their values attached to each object
  • The creation time (as an epoch) of each object
  • Some generic, object specific “info”
  • Other states specific to the object in question

kube-state-metrics can be deployed like a classic Kubernetes service with only one replica.

List of metrics

Metrics about your CRD

Kustomize plugins

This is the third article out of three articles on Kubernetes tool : Kustomize. This article covers the plugins. 

kustomize plugins

Kustomize offers a plugin framework allowing people to write their own resource generators and transformers.


- gen_file.yaml
- trans_file.yaml

Let's focus on gen_file. Trans_file will be similar. 


apiVersion: "apiVersion"
kind: Gen_File
  name: "some name"

Now the file name "Gen_File" will be searched at path  
XDG_CONFIG_HOME = $HOME/.config = /home/manish.config
Possible value for apiVersion = someteam.example.com/v1

If failed then "Gen_File.so" will be searched at same path

This file will be invoked with gen_file.yaml

Reference : https://github.com/kubernetes-sigs/kustomize/tree/master/docs/plugins

Built-in plugins : https://github.com/kubernetes-sigs/kustomize/tree/master/plugin/builtin

Plugin Development

UseCases of Kustomize

This is the second article out of three articles on Kubernetes tool : Kustomize. This article covers the usecases. 

UseCase 1 : Config map generation and secreat generation 
- name: myregistrykey
 type: docker-registry
 - docker-username=DOCKER_USER
 - docker-password=DOCKER_PASSWORD
 - docker-email=DOCKER_EMAIL

This is same as: 

kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

Same as above configMapGenerator

- name: profile
  - hello.config
We can also merge two configmaps

- name: my-configmap
  behavior: merge
  - plumbing.properties
  - secret.properties

Configmap from literals

- name: my-configmap
  - foo=bar
  - baz=qux
UseCase 2 : Creating multiple variants using overlays. 
Edit attributes  as per specific file. Here localserv.yaml . Keyword is patchesStrategicMerge

kustomize edit add patch ocalserv.yaml

- ../../base
- localserv.yaml

Multi Variant Examples: 

we can use "kustomize diff base/variant1/variant2" command to see the difference. 

UseCase 3 : edit container image and tag

kustomize edit set image busybox=alpine:3.6

- name: busybox
  newName: alpine
  newTag: 3.6
UseCase 4 : Remote Target
kustomize build can be run on a URL.

The effect is the same as cloning the repo, checking out a particular ref (commit hash, branch name, release tag, etc.), then running kustomize build against the desired directory in the local copy.
UseCase 5 : applying a JSON patch. Replace and add

cat <$DEMO_HOME/ingress_patch.json
  {"op": "replace", "path": "/spec/rules/0/host", "value": "foo.bar.io"},
  {"op": "replace", "path": "/spec/rules/0/http/paths/0/backend/servicePort", "value": 8080}
You can also write the patch in YAML format. This example also shows the "add" operation:

cat <$DEMO_HOME/ingress_patch.yaml
- op: replace
  path: /spec/rules/0/host
  value: foo.bar.io

- op: add
  path: /spec/rules/0/http/paths/-
    path: '/test'
      serviceName: my-test
      servicePort: 8081

- target:
    group: apps
    version: v1
    kind: Deployment
    name: my-nginx
  path: patch.yaml
UseCase 6 : Patch on multiple objects
JSON patch and strategic merge patch can be applied to selected resources

- path: "PatchFile"
    group: "Group"
    version: "Version"
    kind: "Kind"
    name: "Name"
    namespace: "Namespace"
    labelSelector: "LabelSelector"
    annotationSelector: "AnnotationSelector"
UseCase 7 : Injecting k8s runtime data into containers



This is the first article out of three articles on Kubernetes tool : Kustomize. This article covers the basics. 


Kustomize is a CLI for managing K8s style objects with declarative ways

Edit name attribute by adding prefix/suffix  to it

namePrefix: prod-
nameSuffix: "-001"

Edit labels (k,v)  

  app: my-wordpress

Edit annotations (k,v)  

  description: this is test
Edit namespace

namespace: my-namespace
Edit attribute to Multiple Files. 
kustomize edit add resource deployment.yaml
kustomize edit add resource service.yaml

  app: my-wordpress
- deployment.yaml
- service.yaml
Edit attribute to Multiple directory. 

  app: my-wordpress
- ./wordpress
- ./mysql

$ kustomize build $DEV_HOME
Its output is concatenated YAML documents. They can be directed to file or to kubectl 

kubectl apply -k $DEV_HOME
kustomize build $DEV_HOME | kubectl apply -f -
kubectl kustomize $DEV_HOME

Useful Terms

base: A combination of a kustomization and resource(s). Bases can be referred to by other kustomizations.

kustomization: Refers to a kustomization.yaml file, or more generally to a directory containing the kustomization.yaml file and all the relative file paths that the YAML file references.

overlay: A combination of a kustomization that refers to a base, and a patch. An overlay may have multiple bases.

patch: General instructions to modify a resource.

resource: Any valid YAML file that defines an object with a kind and a metadata/name field.

target: The argument to kustomize build. For example, kustomize build $TARGET. A target must be a path or a URL to a kustomization. A target can be a base or an overlay.

variant: The outcome of applying an overlay to a base.

Read More: https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md


Other similar tools

Transcendence - 3

---------------------------------------------------------------------------------------Let me share some pearls of wisdom from a book. 
My Spiritual Experiences with Pramukh Swamiji
by A.P.J. Abdul Kalam
with Arun Tiwari. 

This is not a book review.
This article is just like
'key take away points' for me. 

The book is divided in 4 parts. This article covers 3rd part of the book. 

Part 4 Fusion of science and spirituality 


People who are unable to understand perfectly both scripture and science far outnumber those who do understand them perfectly. The former glancing superficially through the scriptures, could easily arrogate to themselves the authority to decide upon every question of physics on the strength of some word which they have misunderstood, and which was consciously employed by the sacred authors for some different purpose. And the smaller number of understanding men could not dam up the furious torrent of such people. These people would gain the most followers, simply because it is much more pleasant to gain a reputation for wisdom without effort or study, than to consume oneself tirelessly in the most laborious disciplines.


To sense that behind everything that can be experienced there is something that our minds cannot grasp, whose beauty and sublimity reaches us only indirectly: that is religiousness. In this sense… I am a devoutly religious man.

- Albert Einstein from conclusion of ‘What I Believe’ , 1930

A religion is a system of symbols, which acts to establish powerful, pervasive, and long-lasting moods in men by formulating conceptions of a general order of existence and clothing those conceptions with such and aura of factuality that the moods and motivations seem uniquely realistic.

-American anthropologist Clifford Geertz

We see what we want to see!

‘Faith’ is interpreted as being similar to ‘having working hypothesis’.

– Max Plank

It is a dangerous act of self-delusion if one attempts to get rid of an unpleasant moral obligation, by claiming the human action is the inevitable result of an inexorable law of nature.

Since we are totally connected with everything and every creature in this universe, it is most foolish for each of us to strive to better ourselves without regard to what happens to others.


The purpose of journey on this precious Earth is now to align our personalities with our souls. It is to create harmony, cooperation, sharing, and reverence for life. It is to grow spiritually. This is our new evolutionary pathway.

-   Gary Zukav (Spiritual author and teacher)

Genes do not control life. It is the environment, and more specifically, our perception of the environment that controls gene activity. In the end, it comes down to a simple case of mind over matter in controlling the fate of our lives.

-Bruce Harold Lipton (American developmental Biologist) from his book “The Biology of Belief”

There are very dark forces trying hard to manipulate us into negativity.

-Bruce Harold Lipton (American developmental Biologist) from his book “The Biology of Belief”

One can live and work in this world as an awakened professional.

This chapter 20 also sheds light on how status quo of our own inner lives harms us. We denounce wisdom of others when it does not fit into our emotional framework. We choose to agree only with opinions of others who share our personal values and feelings...In this state of ignorance, we become subservient, we are at the mercy of the popular status quo.


Chapter 21 is more about Baruch Spinoza a Dutch philosopher and Hindu Vedaanta philosophy.

Reality is perfection. If circumstances are seen as unfortunate, it is only because of our inadequate conception of reality. …. The world as it exists look imperfect only because of our limited perception.

- Baruch Spinoza a Dutch philosopher


The deep emotional conviction of the presence of a superior reasoning power, which is revealed in the incomprehensible universe, form my idea of God. 

- Albert Einstein

In daily life, we can reliably apply spiritual action at four levels - obedience, belief, understanding and knowledge. Each individual ... favour one specific level...based on his/her relative stage of spiritual practice. 

1. Obedience as a type of spiritual action, is the relationship between word and action. 

2. Beliefs are conclusions that we accept about our world. Our beliefs significantly comprises our sense of reality, ....how we perceive and feel about our world. ... They predispose us to expectations about outcome. When we imbibe beliefs in goodness... it gives us our trust and faith. 

3. Understanding ...the mental seeing that results from the spiritual process of understanding.....

4. Knowledge Spiritual Knowledge expands and evolves as we identify with our spiritual principles, understand them, embody them, and allow them to be infinite. 

Understanding and experience can work together to form knowledge. 


Our prime purpose in this life is to help others. And if you cannot help them, at least do not hurt them. 

- Dalai Lama

God sleeps in the minerals, awakens in plants, walks in animals and thinks in man. 
- Arthur Middleton Young (Designer of first helicopter and writer)

You cannot pluck a little flower
Without the shaking of star. 
- A poet


Where your talents and the needs of the world cross, there lies your purpose. 

- Aristotle
Pre-Christian Greek philosopher and scientist 

Transcendence - 1

---------------------------------------------------------------------------------------Let me share some pearls of wisdom from a book. 
My Spiritual Experiences with Pramukh Swamiji
by A.P.J. Abdul Kalam
with Arun Tiwari. 

This is not a book review.
This article is just like
'key take away points' for me. 

The book is divided in 4 parts. This article covers 1st part of the book. 


Part 1 Experiencing the presence 

Tolerance of others' view and opinions is essential in building teams and accomplishing tasks that are beyond the individuals' capacities. 
- Dr. Brahma Prakash


Srimad Bhagavad Gita defines renunciation in a unique way: "One must renounce not the performance of deeds, but renounce the desire for the fruits of those actions." 
- Pramukh Swamiji


When things go smoothly and fall into place, you have most likely made the right choices so that the right event could come into your life. When your decisions of efforts encounter resistance or roadblocks, seriously re-examine the choices you have made. 

"Who I really am".... can only be revealed when the mind is quiet and no longer telling me who I am. When all the preconceptions about myself are stilled, what remains is who I really am: consciousness, awareness, stillness, presence, peace, love, and the Divine. You are that which is nameless and yet has been given a thousand names.


Harmony makes small things grow, lack of it makes great things decay. 
- Sallust
1st century BC Roman senator

When you go to a place of worships, you pray for peace and prosperity, not only for yourself but also for everyone around you. 

Sulah meaning peace and peacemaking in Arabic. 
Sulah is also the root of the word Islah denoting development and improvement. 


It is easier to build strong children than to repair broken men
- Frederick Douglass
19th Centaury African-American social reformer

The empires of the future are the empires of the mind. Dream, dream, dream. Dreaming leads to thoughts. Thoughts lead to action. 

"How can you mix spirituality and social service?"
Pramukh Swamiji asks, "How can you separate the two?"


Success can only come to you by courageous devotion to the task in front of you... You will be remembered for creating the one page in the history of the nation - whether it is the page of invention, innovation, discovery or fighting injustice. 
A.P.J. Abdul Kalam (in book 'Indomitable Spirit') 


The wise discipline themselves, the unwise discipline others
- Pramukh Swamiji

As long as there is the pull of gravity whatever you throw up is always going to come down. But once a rocket is out of the attraction of the Earth's gravity, it will not fall back and will escape into space. Likewise, as long as we are attracted and attached to 
- the comforts of this body
- desires of the mind and
- the material world,
we are consigned to the cycle of births and deaths. There will be no escape. But as your wordly desires decreases, you transcend the pull of the world and eventually you connect to God. 
- Pramukh Swamiji

Transcendence - 2

---------------------------------------------------------------------------------------Let me share some pearls of wisdom from a book. 
My Spiritual Experiences with Pramukh Swamiji
by A.P.J. Abdul Kalam
with Arun Tiwari. 

This is not a book review.
This article is just like
'key take away points' for me. 

The book is divided in 4 parts. This article covers 2nd part of the book. 


Part 2 Spirituality in Action


Appearance is a glimpse of the unseen
- Anaxagora
Pre-Socratic Greek Philosopher

A temple is the physical  manifestation of the unseen
- Pramukh Swami Maharaj

Do not confuse excellence with perfection. Excellence man can reach, but perfection is God's work. 
- Pramukh Swami Maharaj


Courage is not absence of fear, but the triumph over it. The brave man is not be who does not feel afraid, but he who conquers the fear. 
-Nelson Mandela

To overcome fear by forgiving through faith. 
- Pramukh Swami Maharaj


If you surround yourself with good and righteous, they can only raise you up. 
If you surround yourself with ordinary, they will drag you down into the pessimism of mediocrity, and they will keep you there, but only as long as you permit it. 
- father of A.P.J. Abdul Kalam

Youth needs a wisdom. 
Youth needs a vision.
Youth needs credible guidance
and above youth needs an example. 
Great minds give all three. 

The vision of a prosperous and peaceful humanity. 
guidance through the idea of great institutions like BAPS
and the example of impeccable service 
are beacons of that guide souls from drifting and foundering 
however deep and dark the turbulent sea. 

Where there is faith, there is love
Where there is love, there is peace
Where there is peace, there is God
And where there is God, there is no need. 
- Leo Tolstoy 
in book "The Kingdom of God Is Within You"


If you are irritated by every run, how will your mirror be polished? 
- Rumi
13th Century Persian poet

Where there is righteousness in the heart, 
There is beauty in the character. 
When there is beauty in the character, 
There is harmony in the home.
When there is harmony in the home, 
There is order in the nation.
When there is order in the nation, 
There is peace in the world. 


We cannot teach people anything; we can only help them discover it within themselves. 
-Galileo Galilei
16th century Italian philosopher. 

Human brain
* 2% of body weight
* receives 15% of cardiac power
* 20 % of total body consumption
* 25 % of total body glucose utilization


If you talk to a man in a language he has learned in school, it goes to his head. If you talk to him in a lanugage he has heard from his mother, it goes to his heart
-Nelson Mandela

If I have a beautiful mind, I will have beautiful thoughts
If I have a beautiful thoughts, I will have beautiful life
If I have a beautiful life, I will become a great soul like Pramukh Swamiji

Confidence leads to creativity
Creativity leads to knowledge
Knowledge leads to thinking
Thinking makes on great. 

Every time we smile angel wins and every time we sulk Satan wins

Jaeger + Elastic Search: Handson


docker run --rm -it --name=elasticsearch -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:6.8.0

docker run --rm -it --link=elasticsearch --name=kibana -p 5601:5601 docker.elastic.co/kibana/kibana:6.8.0

docker run --rm -it --link=elasticsearch --name=jaeger -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://elasticsearch:9200 -e ES_TAGS_AS_FIELDS_ALL=true -p 16686:16686 jaegertracing/all-in-one:1.12

docker run --rm --link jaeger --env JAEGER_AGENT_HOST=jaeger --env JAEGER_AGENT_PORT=6831 -p8080-8083:8080-8083 jaegertracing/example-hotrod:latest all 

Create a file JSON.txt with below conntent

    "query" : {
        "match_all" : {}


curl -XGET "localhost:9200/jaeger-span-YYYY-MM-DD/_search" -H "Content-Type: application/json" -d @json.txt

curl -XGET "localhost:9200/jaeger-span-YYYY-MM-DD/_search?size=10000" -H "Content-Type: application/json" -d @json.txt



JSON Output: 

  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  "hits": {
    "total": 441,
    "max_score": 1,
    "hits": [
        "_index": "jaeger-span-2019-11-12",
        "_type": "span",
        "_id": "mQ2cXm4BVYoOLxBFNHpP",
        "_score": 1,
        "_source": {
          "traceID": "553c8316169ab970",
          "spanID": "2cd1c33438cb79f5",
          "flags": 1,
          "operationName": "HTTP GET /route",
          "references": [
              "refType": "CHILD_OF",
              "traceID": "553c8316169ab970",
              "spanID": "44fa89467cc873aa"
          "startTime": 1573545324720315,
          "startTimeMillis": 1573545324720,
          "duration": 53302,
          "tags": [],
          "tag": {
            "component": "net/http",
            "http@method": "GET",
            "http@status_code": 200,
            "http@url": "/route?dropoff=728%2C326&pickup=106%2C311",
            "internal@span@format": "proto",
            "span@kind": "server"
          "logs": [
              "timestamp": 1573545324720369,
              "fields": [
                  "key": "event",
                  "type": "string",
                  "value": "HTTP request received"
                  "key": "level",
                  "type": "string",
                  "value": "info"
                  "key": "method",
                  "type": "string",
                  "value": "GET"
                  "key": "url",
                  "type": "string",
                  "value": "/route?dropoff=728%2C326&pickup=106%2C311"
          "process": {
            "serviceName": "route",
            "tags": [],
            "tag": {
              "client-uuid": "761a24cbd892ada8",
              "hostname": "367b2536b5f9",
              "ip": "",
              "jaeger@version": "Go-2.20.0"

Trace JSON at Kibana APM

    "events": [
            "@timestamp": "2018-07-30T18:53:42.281Z",
            "agent": {
                "name": "elastic-node",
                "version": "3.14.0"
            "parent": {
                "id": "abcdef0123456789"
            "processor": {
                "event": "span",
                "name": "transaction"
            "service": {
                "environment": "staging",
                "name": "backendspans"
            "span": {
                "action": "query.custom",
                "duration": {
                    "us": 141581
                "id": "abcdef01234567",
                "name": "GET /api/types",
                "subtype": "postgresql",
                "type": "db"
            "timestamp": {
                "us": 1532976822281000
            "trace": {
                "id": "fdedef0123456789abcdef9876543210"
            "transaction": {
                "id": "01af25874dec69dd"