CKA 6: Cluster Maintenance
Drain Cordon Uncordon
If a worker node down, then K8s wait for "pod-eviction-timeout" period. Default value is 5 min
This value is passed to kube-control-manager as command line argument.
so if we can upgrade worker node within 5 min, it is ok. Else safer way is:
k drain "worker node name"
it also means the worker node is cordoned, i.e. no pod can be scheduled on it. So once it is back, we should uncorden it
k uncorden "worker node name"
K8s Version
It applies to: (1) kube-apiserver (2) controller-manager (3) kube-scheduler (4) kubelet (5) kube-proxy (6) kubectl
It does not apply to (1) etcd cluster (2) coreDNS
Cluster Upgrade
If Kube-apiserver has version X then
(1) controller-manager and (2) kube-scheduler can be with version [X - 1, X]
(1) kubelet and (2) kube-proxy can be with version [X - 2, X - 1, X]
kubectl can be with version [X - 1, X, X + 1]
* We should always upgrade one minor release at a time
kubeadm upgrade plan //Gives all the information about present version and latest available version
kubeadm upgrade apply "New version"
On master node
1. Upgrade kubeadm tool itself.
2. kubeadm upgrade plan "New version"hold hold is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
3. kubeadm upgrade apply "New version"
k get nodes command show version of kublet, not version of kube-apiserver
3. if kubelet is present at master node then upgrade it
4. systemctl restart kubelet
on worker node
1. k drain "worker node name"
2. upgrade kubeadm
3. upgrade kubelet
4. kubeadm upgrade node config --kubelet-version "new version"
5. systemctl restart kubelet
6. k uncorden "worker node name"
OR
1. k drain "worker node name"
2. kubeadm upgrade node
3. upgrade kubelet
4. k uncorden "worker node name"
If we run k drain command for master node, then also it applies only to user applications. The K8s applications running on master node will not evacuated.
"apt-mark hold" is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
Backup and Restore
1. YAML files
k get all -A -o yaml > all.yaml
OR use tools
ARK by HeptIO is now called Velero
instead of getting backup of YAML file, take back up etcd cluster.
2. etcd cluster
2.1 etcd started with "--data-dir" option. Its default value is /var/lib/etcd
If the snapshot is copied from the data directory, there is no integrity hash and it will only restore by using --skip-hash-check.
2.2 ETCDCTL_API=3 etcdctl snapshot save snapshot.db
To restore
A. service kube-apiserver stop
B. ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --name="" --data-dir "" --initial-cluster "" --initial-cluster-token "" --initial-advertise-peer-urls "" --data-dir=""
C. Preferably use different (1) initial-cluster-token and (2) data-dir. change etcd configuration with these 2 new values
D. service etcd restart
E. service kube-apiserver start
We must specify (1) endpoints (2) cacert (3) cert and (4) key
In managed K8s, we may not have access to etcd. So back up of YAML file is better approach.
3. If etcd is running on a storage volume that supports backup, such as Amazon Elastic Block Store, back up etcd data by taking a snapshot of the storage volume.
Reference
https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md
https://www.youtube.com/watch?v=qRPNuT080Hk
If a worker node down, then K8s wait for "pod-eviction-timeout" period. Default value is 5 min
This value is passed to kube-control-manager as command line argument.
so if we can upgrade worker node within 5 min, it is ok. Else safer way is:
k drain "worker node name"
it also means the worker node is cordoned, i.e. no pod can be scheduled on it. So once it is back, we should uncorden it
k uncorden "worker node name"
K8s Version
It applies to: (1) kube-apiserver (2) controller-manager (3) kube-scheduler (4) kubelet (5) kube-proxy (6) kubectl
It does not apply to (1) etcd cluster (2) coreDNS
Cluster Upgrade
If Kube-apiserver has version X then
(1) controller-manager and (2) kube-scheduler can be with version [X - 1, X]
(1) kubelet and (2) kube-proxy can be with version [X - 2, X - 1, X]
kubectl can be with version [X - 1, X, X + 1]
* We should always upgrade one minor release at a time
kubeadm upgrade plan //Gives all the information about present version and latest available version
kubeadm upgrade apply "New version"
On master node
1. Upgrade kubeadm tool itself.
2. kubeadm upgrade plan "New version"hold hold is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
3. kubeadm upgrade apply "New version"
k get nodes command show version of kublet, not version of kube-apiserver
3. if kubelet is present at master node then upgrade it
4. systemctl restart kubelet
on worker node
1. k drain "worker node name"
2. upgrade kubeadm
3. upgrade kubelet
4. kubeadm upgrade node config --kubelet-version "new version"
5. systemctl restart kubelet
6. k uncorden "worker node name"
OR
1. k drain "worker node name"
2. kubeadm upgrade node
3. upgrade kubelet
4. k uncorden "worker node name"
If we run k drain command for master node, then also it applies only to user applications. The K8s applications running on master node will not evacuated.
"apt-mark hold" is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
Backup and Restore
1. YAML files
k get all -A -o yaml > all.yaml
OR use tools
ARK by HeptIO is now called Velero
instead of getting backup of YAML file, take back up etcd cluster.
2. etcd cluster
2.1 etcd started with "--data-dir" option. Its default value is /var/lib/etcd
If the snapshot is copied from the data directory, there is no integrity hash and it will only restore by using --skip-hash-check.
2.2 ETCDCTL_API=3 etcdctl snapshot save snapshot.db
To restore
A. service kube-apiserver stop
B. ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --name="" --data-dir "" --initial-cluster "" --initial-cluster-token "" --initial-advertise-peer-urls "" --data-dir=""
C. Preferably use different (1) initial-cluster-token and (2) data-dir. change etcd configuration with these 2 new values
D. service etcd restart
E. service kube-apiserver start
We must specify (1) endpoints (2) cacert (3) cert and (4) key
In managed K8s, we may not have access to etcd. So back up of YAML file is better approach.
3. If etcd is running on a storage volume that supports backup, such as Amazon Elastic Block Store, back up etcd data by taking a snapshot of the storage volume.
Reference
https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md
https://www.youtube.com/watch?v=qRPNuT080Hk
CKA 5: Application Life Cycle Management
Init Containers
initContainers:
It is a list
Each init container run, one at a time in sequential order.
It is like job. Run to complete
If any one fails then pod will be restarted, unless pod has restartPolicy = Never
The init container does not support
- lifecycle,
- livenessProbe,
- readinessProbe,
- startupProbe
Pod-> activeDeadlineSeconds and pod->container->liveness probe , prevents init containers from restarting from ever.
Resources
Highest value among all init containers is considered for resources->request and resources->limit
For pod effective limit for resources->request and resources->limit = Highest ( Highest (init containers) , summation ( all normal containers) )
This effect limit is considered to schedule a pod
The resources reserved by init container(s) is not used once they are completed, by other containers.
If we change image of container then only that container is restarted. If we change image of init container then whole pod is restarted.
initContainers:
It is a list
Each init container run, one at a time in sequential order.
It is like job. Run to complete
If any one fails then pod will be restarted, unless pod has restartPolicy = Never
The init container does not support
- lifecycle,
- livenessProbe,
- readinessProbe,
- startupProbe
Pod-> activeDeadlineSeconds and pod->container->liveness probe , prevents init containers from restarting from ever.
Resources
Highest value among all init containers is considered for resources->request and resources->limit
For pod effective limit for resources->request and resources->limit = Highest ( Highest (init containers) , summation ( all normal containers) )
This effect limit is considered to schedule a pod
The resources reserved by init container(s) is not used once they are completed, by other containers.
If we change image of container then only that container is restarted. If we change image of init container then whole pod is restarted.
SPIFEE & SPIRE
SPIFEE Introduction
SPIFEE standard is all about
- How to encode SPIFEE ID into X.509 certificate
- Which field to use
- How to validate (1) X.509 certificate and (2) JWT token, when SPIFEE ID is inside.
SPIFEE is for universal ID. SPIFEE is for how different components can trust each other in distributed system. SPIFEE was launched in KubeCon 2017. SPIFEE Federation API was main focus during 2018, 2019.
SPIRE
SPIRE is open source project, that implements SPIFEE standards.
1. It expose workload API
2. It is framework to manage issue of ID.
3. It is for 'secure introduction' = 'credential zero' ='bootstrap credential' Workload authenticate itself with SPIRE agent and agent with server.
4. trust bootstrapping
SPIRE is stand alone. It has many custom plugins
SPIRE server
- Identity Mapping / Identity Registry. It exposes Registration API
- Node API using Node Attestation plugin + Node Resolver plugin
- Federation API
- SVID Issuance (SPIFFE Verifiable Identity Document)
- signing key
- registry of workloads
It can have plugins like
1. Upstream CA plugin
2. Node Attestor plugin to validate node. Both SIRE agent and SPIRE server
3. Node Resolver plugin
4. Datastore plugin MySQL, SQLite 3 (default), or PostgresSQL
5. Key Manager plugin. To store private key to sign SVIDs (X.509 and JWT both)
It can be deployed as stateful set. It can have PV. In production environment, it can use DB: Pstgres or MySQL
SPIRE Agent
SPIRE Agent assign SPIFEE ID to workload and generates CSR to SPIRE server. The SPIRE server returns SPIFEE ID and trust bundle (a set of certificates to verify X.509-SVID OR public key to verify JWT). They gets transfer from SPIRE server to Node agent to workload. The private key of workload, never leave node.
- Workload API
- Workload Attestation : Verify authenticity of caller. only SPIRE agent
It can have plugins like
1. Multiple Workload attestor plugins
1.a Unix attestor (OS attestor). It use out-of-band Linux kernel to verify selector mentioned in request are genuine or not.
1.b K8s attestor. It communicate with kubelet. Verify it is genuine K8s workload. then ns, sa, docker image id etc.
2. Node attestor plugin. It used bootstrap configuration. Server responds with SVID to agent. Also SPIFEE ID of node. It becomes parent ID for workload.
3. key manager plugin. Generate and use private keys for X.509-SVID
It can be deployed as daemon set
Valid Node ID
1. cloud platform e.g. AWS Instance Identification Document IID, Azure Managed Service Identities, GCE Instance Identity Tokens
2. Private key stored at TPM = Trusted Platform Module or HSM = Hardware Security Module
3. manual verification through a joint token
4. SA token
5. etc.
SVID (SPIFFE Verifiable Identity Document) has two format
1. X.509 certificate
2. JWT token
- it is susceptible to replay attacks
- Use it when L7 proxy of L7 LB is on path.
SPIRE supports a specific form of JWT that is specifically designed to encode SPIFFE IDs, the JWT-SVID.
(1) ns = namespace
(2) sa = service account
(3) docker image id
- Parent ID can be K8s cluster name
- SPIFEE ID: Format spiffe://trust domain/workload
- DNS Name: OR CN:
- TTL:
- Entry ID:
Usecases
1. DB Access
2. Access to cloud provider
3. identity translation,
4. OAuth client authentication,
5. mTLS "encryption everywhere" and
6. workload observability.
7. Square talks about how Square uses SPIFFE and SPIRE to secure communications across hybrid infrastructure services: https://youtu.be/H5IlmYmEDKk?t=2585
8. Uber talks about integrating SPIRE with workload schedulers: https://youtu.be/H5IlmYmEDKk?t=4703
9. Tigera demonstrates how Calico, Envoy and SPIRE are used to deliver unified Layer 4 and Layer 7 authorization policies: https://youtu.be/H5IlmYmEDKk?t=7812
10. Bloomberg talks about TPM node attestation with SPIRE: https://youtu.be/30S0sKRxzjM
11. NGINX/F5 on how NGINX service mesh leverages SPIFFE and SPIRE https://youtu.be/plRkDK5xFpM
Other tools
1. Secret Stores
1.1 Hashicorp Vault
1.2 Square Keywhiz
2. Identity Provider
2.1 ory.sh
2.2 VMWare Lightwave
2.3 WS02 Identity Serve
3. Authorization Policy Engines
3.1 Open Policy Agent
4. Service Mesh
In case of Istio: "Istio Node Agent" is "SPIRE Agent". The "SPIRE server" can have "Istio Node Attestor Plugin"
Reference:
https://www.youtube.com/watch?v=5m6kjzdysBI
https://www.youtube.com/watch?v=ikmxZdZRTio
https://www.youtube.com/watch?v=0LSaNrOabH4
https://www.thoughtworks.com/radar/platforms/spiffe
https://github.com/spiffe/spire/blob/master/ADOPTERS.md
https://www.youtube.com/watch?v=OHiPsqT1gcI
SPIFEE standard is all about
- How to encode SPIFEE ID into X.509 certificate
- Which field to use
- How to validate (1) X.509 certificate and (2) JWT token, when SPIFEE ID is inside.
SPIFEE is for universal ID. SPIFEE is for how different components can trust each other in distributed system. SPIFEE was launched in KubeCon 2017. SPIFEE Federation API was main focus during 2018, 2019.
SPIRE
SPIRE is open source project, that implements SPIFEE standards.
1. It expose workload API
2. It is framework to manage issue of ID.
3. It is for 'secure introduction' = 'credential zero' ='bootstrap credential' Workload authenticate itself with SPIRE agent and agent with server.
4. trust bootstrapping
SPIRE is stand alone. It has many custom plugins
SPIRE server
- Identity Mapping / Identity Registry. It exposes Registration API
- Node API using Node Attestation plugin + Node Resolver plugin
- Federation API
- SVID Issuance (SPIFFE Verifiable Identity Document)
- signing key
- registry of workloads
It can have plugins like
1. Upstream CA plugin
2. Node Attestor plugin to validate node. Both SIRE agent and SPIRE server
3. Node Resolver plugin
4. Datastore plugin MySQL, SQLite 3 (default), or PostgresSQL
5. Key Manager plugin. To store private key to sign SVIDs (X.509 and JWT both)
It can be deployed as stateful set. It can have PV. In production environment, it can use DB: Pstgres or MySQL
SPIRE Agent
SPIRE Agent assign SPIFEE ID to workload and generates CSR to SPIRE server. The SPIRE server returns SPIFEE ID and trust bundle (a set of certificates to verify X.509-SVID OR public key to verify JWT). They gets transfer from SPIRE server to Node agent to workload. The private key of workload, never leave node.
- Workload API
- Workload Attestation : Verify authenticity of caller. only SPIRE agent
It can have plugins like
1. Multiple Workload attestor plugins
1.a Unix attestor (OS attestor). It use out-of-band Linux kernel to verify selector mentioned in request are genuine or not.
1.b K8s attestor. It communicate with kubelet. Verify it is genuine K8s workload. then ns, sa, docker image id etc.
2. Node attestor plugin. It used bootstrap configuration. Server responds with SVID to agent. Also SPIFEE ID of node. It becomes parent ID for workload.
3. key manager plugin. Generate and use private keys for X.509-SVID
It can be deployed as daemon set
Valid Node ID
1. cloud platform e.g. AWS Instance Identification Document IID, Azure Managed Service Identities, GCE Instance Identity Tokens
2. Private key stored at TPM = Trusted Platform Module or HSM = Hardware Security Module
3. manual verification through a joint token
4. SA token
5. etc.
SVID (SPIFFE Verifiable Identity Document) has two format
1. X.509 certificate
2. JWT token
- it is susceptible to replay attacks
- Use it when L7 proxy of L7 LB is on path.
SPIRE supports a specific form of JWT that is specifically designed to encode SPIFFE IDs, the JWT-SVID.
Workload registry entry fields
- Properties are called selector (1) ns = namespace
(2) sa = service account
(3) docker image id
- Parent ID can be K8s cluster name
- SPIFEE ID: Format spiffe://trust domain/workload
- DNS Name: OR CN:
- TTL:
- Entry ID:
Usecases
1. DB Access
2. Access to cloud provider
3. identity translation,
4. OAuth client authentication,
5. mTLS "encryption everywhere" and
6. workload observability.
7. Square talks about how Square uses SPIFFE and SPIRE to secure communications across hybrid infrastructure services: https://youtu.be/H5IlmYmEDKk?t=2585
8. Uber talks about integrating SPIRE with workload schedulers: https://youtu.be/H5IlmYmEDKk?t=4703
9. Tigera demonstrates how Calico, Envoy and SPIRE are used to deliver unified Layer 4 and Layer 7 authorization policies: https://youtu.be/H5IlmYmEDKk?t=7812
10. Bloomberg talks about TPM node attestation with SPIRE: https://youtu.be/30S0sKRxzjM
11. NGINX/F5 on how NGINX service mesh leverages SPIFFE and SPIRE https://youtu.be/plRkDK5xFpM
Other tools
1. Secret Stores
1.1 Hashicorp Vault
1.2 Square Keywhiz
2. Identity Provider
2.1 ory.sh
2.2 VMWare Lightwave
2.3 WS02 Identity Serve
3. Authorization Policy Engines
3.1 Open Policy Agent
4. Service Mesh
In case of Istio: "Istio Node Agent" is "SPIRE Agent". The "SPIRE server" can have "Istio Node Attestor Plugin"
Reference:
https://www.youtube.com/watch?v=5m6kjzdysBI
https://www.youtube.com/watch?v=ikmxZdZRTio
https://www.youtube.com/watch?v=0LSaNrOabH4
https://www.thoughtworks.com/radar/platforms/spiffe
https://github.com/spiffe/spire/blob/master/ADOPTERS.md
https://www.youtube.com/watch?v=OHiPsqT1gcI
Replication Controller
Replication Controller
replicationcontrollerMetadata attributes:
replicationcontrollerSpec attributes:
By default, deployment will add a pod-template-hash to the name of RS that it creates.
replicationcontrollerMetadata attributes:
- clusterName: The name of the cluster which the object belongs to.
- clusterName->namespace->name
- deletionGracePeriodSeconds: Seconds allowed for gracefully terminate before forcefully removed from the system.
- deletionTimestamp: RFC 3339 date and time at which this resource will be deleted. Is not directly settable by a client.
- finalizers: Must be empty before the object is deleted from the registry.
- generateName: Optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided.
- generation: A sequence number representing a specific generation of the desired state. Populated by the system. Read-only.
- initializers: An initializer is a controller which enforces some system invariant at object creation time.
- labels: Map of string keys and values that can be used to organize and categorize (scope and select) objects.
- ownerReferences: List of objects depended by this object.
replicationcontrollerSpec attributes:
- minReadySeconds: Minimum number of seconds for which a newly created pod should be ready without any of its container crashing, for it to be considered available. Defaults to 0
By default, deployment will add a pod-template-hash to the name of RS that it creates.
CKA: 4. Logging and Monitoring
Open Source: Metrics Server, Prometheus, Elastic Stack
Proprietary: Datadog, dynatrace
Original project was: Heapster
Now it is Metrics Server (in-memory)
kubelet has cAdvisor = Container Advisor
minikube addons enable metrics-server
k top node
k top pode
Proprietary: Datadog, dynatrace
Original project was: Heapster
Now it is Metrics Server (in-memory)
kubelet has cAdvisor = Container Advisor
minikube addons enable metrics-server
k top node
k top pode
