CKAD : 5. Deployment Configuration
K8s volume share lifetime of pod, not lifetime of containers within. If container restart, then data available to new container. Volume is available to multiple pods. Volume's life can be longer than life of pod. If multiple pods have write access mode, then data corruption is possible unless there is locking mechanism.
Pod has Volume. Single volume can be available to multiple container within pod using VolumeMount (directory or folder path). (If VolumeMount has different value, then within each container, the path will be different.) So it can be used for intra pod communication.
28+ volume types available
- rbd for Ceph, GlusterFS: block storage.
- NFS, iSCSI : for multiple reading
- gcePersistentDisk
- Others: azureDisk, azureFile, csi, downardAPI, fc (fiber channel) it has raw block volume, Flocker, gitRepo, Local, projectd, portworxVolume, quobyte, scaleIO, secret, StorageOS, vsphereVolume, persistentVolumeClaim, awsElasticStore, CepthFS, Cinder, FlexVolume, GCEPersistentDisk, Quobyte,
PV K8s PV has longer lifetime than pod. Pod can claim PV as PVC.
PV is cluster scoped.
Volume plugin (as per CSI), are in-tree. i.e. they compiled along with k8s binaries
Out-of-tree object allow storage vendors to develop single driver and plugin can be containerized. it needs elevated access to host node so it is security risk.
Pod can claim PV as PVC. Multiple pods can share PVC. Other pod cannot use data inside volume.
Pod:Voulme:PVC
Then it is mounted in container.
Access Mode (accessModes) :
1. RWO (Read Write Once)
2. ROX (Read Only Many)
3. RWx (Read Write Many)
for node
1. cluster group all volumes with same access mode.
2. then sort volume by size. smallest to largest.
PVC request with parameter (1) access mode, (2) volume size (3) storage class is optional parameter.
PVC is namespaced scope
phase
1. Provisioning : PV
PV can be of type (1) empty or (2) pre-populated volume.
1.1. emptyDir
1.2. hostPath : mounts resource from host node file system
- directory
- file
- socket
- character device
- block device
path:
* All types have its own configuration settings.
The resource must exist on host node.
- DirecotryOrCreate
- FileOrCreate
When PV is created using NFS storage, we must mention, server and path both
nfs:
server: nfs01
# Exported path of your NFS server
path: "/mysql"
2. Binding: PVC
k get pv
Here CLAIM column indicate name of PVC
Other columns are : Capacity, Access Mode, Reclaim policy, status = Bound | Available, Storageclass. StorageClass = manual if PV is creating using hostPath.
3. use
If we create a file in path, which is mounted in container pointing to PVC of pod. Then we delete all pods, who have this PVC, then create again, the file is as it is. Because PVC is not deleted.
4. Release: when PVC is deleted
5. reclaim as per persistentVolumeReclaimPolicy
- Retain
- Delete
- Recycle
With retain, the PV is not available for any other PVC, even after old PVC is deleted.
StorageClass is used for dynamic provisioning. We no need to define PV. PV is created automatically. At PVC we can use storageClassName to link PVC with storage class.
Secret
k create secret generic "name of secret" --from-literal=key=value
To encrypt secret, use EncryptionConfiguration object with (1) key and (2) provider identity. The kube-apiserver should be started with --encryption-provider-config flag. Example provider: aescbc, ksm. Other tools Helm Secrets, HashiCorp Vault. Then we need to recreate all existing secrets.
Max size of secret = 1 MB
We can have many secret. No limit
They will stored in tempfs of host node.
It can be mounted in container as environment variable or file. It is stored as plain text inside container.
Inside pod it is referred as: secretKeyRef
Secret has two maps
1. data
2. stringData: It is a write-only convenience field. Not display in output of command kubectl get secret mysecret -o yaml
If any key is specified at both data and stringData, then value specified at stringData is used and value specified at data will be ignored
When we mount secret, we can mount individual key
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
items:
- key: username
path: my-group/my-username
OR all keys of secret.
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
If we specify file permission in JSON, it should be decimal. In YAML it should be octal.
In K8s 1.18 Alpha feature cm and secret both can be immutable, if they specify with "immutable: true"
Secret resources reside in a namespace. Secrets can only be referenced by Pods in that same namespace.
secret->data->.secret-file It will create hidden file, when mounted.
The ability to watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.
Configmap:
1. Key value pair
2. Plain config file in any format
Data come from
1. singe file
2. collection of files in a single directory.
3. literal value (use --from-literal with command k create cm 'cm name' --from-liternal)
Inside pod
1. env var
2. volume
3. a file in volume (a filename is key, file content is value)
4. in pod command
5. set file name and its access mode in volume using ConfigMaps
ConfigMaps can be used by system components and controllers.
Inside pod it is referred as: configMapKeyRef
If ConfigMap is defined as K,V pairs. Pod - > volume is linked with that ConfigMap. The volume is mounted in container at specific path. Inside that directory, there are multiple files as per K in config map and the content of file is V
If we want to mount env variable inside container then no need to modify anything in pod section. Suppose we have cm with name cmone and its content KONE=vone. Another cm with name cmtwo and its content is KA=va and KB=vb
Now to load only one env variable from cmone, we shall use:
envFrom:
- configMapRef:
name: cmone
All K,v pairs from cmone will be loaded to the container as env variables. env name = k and env value = v
If we want to load individual k,v pair use below syntax
- name: KAA
valueFrom:
configMapKeyRef:
name: cmtwo
key: KA
- name: KBB
valueFrom:
configMapKeyRef:
name: cmtwo
key: KB
Here we can change env variable name also.
Same thing applies for secret
envFrom:
- secretRef:
name: sone
and
- name: KAA
valueFrom:
secretKeyRef:
name: stwo
key: KA
- name: KBB
valueFrom:
secretKeyRef:
name: stwo
key: KB
We can also define a readiness probe with command ls /path/for/volumeMount. So when the configmap is loaded to container then only the container becomes ready.
Deployment Status
k get deploy grafana -o yaml
A deployment has strategy with value “Recreate” or “RollingUpdate”.
We can see status
k rollout status deploy "name of deployment"
We can see all events about new pod creation and old pod termination with command
k describe deploy "name of deployment"
Pod has Volume. Single volume can be available to multiple container within pod using VolumeMount (directory or folder path). (If VolumeMount has different value, then within each container, the path will be different.) So it can be used for intra pod communication.
28+ volume types available
- rbd for Ceph, GlusterFS: block storage.
- NFS, iSCSI : for multiple reading
- gcePersistentDisk
- Others: azureDisk, azureFile, csi, downardAPI, fc (fiber channel) it has raw block volume, Flocker, gitRepo, Local, projectd, portworxVolume, quobyte, scaleIO, secret, StorageOS, vsphereVolume, persistentVolumeClaim, awsElasticStore, CepthFS, Cinder, FlexVolume, GCEPersistentDisk, Quobyte,
PV K8s PV has longer lifetime than pod. Pod can claim PV as PVC.
PV is cluster scoped.
Volume plugin (as per CSI), are in-tree. i.e. they compiled along with k8s binaries
Out-of-tree object allow storage vendors to develop single driver and plugin can be containerized. it needs elevated access to host node so it is security risk.
Pod can claim PV as PVC. Multiple pods can share PVC. Other pod cannot use data inside volume.
Pod:Voulme:PVC
Then it is mounted in container.
Access Mode (accessModes) :
1. RWO (Read Write Once)
2. ROX (Read Only Many)
3. RWx (Read Write Many)
for node
1. cluster group all volumes with same access mode.
2. then sort volume by size. smallest to largest.
PVC request with parameter (1) access mode, (2) volume size (3) storage class is optional parameter.
PVC is namespaced scope
phase
1. Provisioning : PV
PV can be of type (1) empty or (2) pre-populated volume.
1.1. emptyDir
1.2. hostPath : mounts resource from host node file system
- directory
- file
- socket
- character device
- block device
path:
* All types have its own configuration settings.
The resource must exist on host node.
- DirecotryOrCreate
- FileOrCreate
When PV is created using NFS storage, we must mention, server and path both
nfs:
server: nfs01
# Exported path of your NFS server
path: "/mysql"
k get pv
Here CLAIM column indicate name of PVC
Other columns are : Capacity, Access Mode, Reclaim policy, status = Bound | Available, Storageclass. StorageClass = manual if PV is creating using hostPath.
3. use
If we create a file in path, which is mounted in container pointing to PVC of pod. Then we delete all pods, who have this PVC, then create again, the file is as it is. Because PVC is not deleted.
4. Release: when PVC is deleted
5. reclaim as per persistentVolumeReclaimPolicy
- Retain
- Delete
- Recycle
With retain, the PV is not available for any other PVC, even after old PVC is deleted.
StorageClass is used for dynamic provisioning. We no need to define PV. PV is created automatically. At PVC we can use storageClassName to link PVC with storage class.
Secret
k create secret generic "name of secret" --from-literal=key=value
To encrypt secret, use EncryptionConfiguration object with (1) key and (2) provider identity. The kube-apiserver should be started with --encryption-provider-config flag. Example provider: aescbc, ksm. Other tools Helm Secrets, HashiCorp Vault. Then we need to recreate all existing secrets.
Max size of secret = 1 MB
We can have many secret. No limit
They will stored in tempfs of host node.
It can be mounted in container as environment variable or file. It is stored as plain text inside container.
Inside pod it is referred as: secretKeyRef
Secret has two maps
1. data
2. stringData: It is a write-only convenience field. Not display in output of command kubectl get secret mysecret -o yaml
If any key is specified at both data and stringData, then value specified at stringData is used and value specified at data will be ignored
When we mount secret, we can mount individual key
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
items:
- key: username
path: my-group/my-username
OR all keys of secret.
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
If we specify file permission in JSON, it should be decimal. In YAML it should be octal.
In K8s 1.18 Alpha feature cm and secret both can be immutable, if they specify with "immutable: true"
Secret resources reside in a namespace. Secrets can only be referenced by Pods in that same namespace.
secret->data->.secret-file It will create hidden file, when mounted.
The ability to watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.
Configmap:
1. Key value pair
2. Plain config file in any format
Data come from
1. singe file
2. collection of files in a single directory.
3. literal value (use --from-literal with command k create cm 'cm name' --from-liternal)
Inside pod
1. env var
2. volume
3. a file in volume (a filename is key, file content is value)
4. in pod command
5. set file name and its access mode in volume using ConfigMaps
ConfigMaps can be used by system components and controllers.
Inside pod it is referred as: configMapKeyRef
If ConfigMap is defined as K,V pairs. Pod - > volume is linked with that ConfigMap. The volume is mounted in container at specific path. Inside that directory, there are multiple files as per K in config map and the content of file is V
If we want to mount env variable inside container then no need to modify anything in pod section. Suppose we have cm with name cmone and its content KONE=vone. Another cm with name cmtwo and its content is KA=va and KB=vb
Now to load only one env variable from cmone, we shall use:
envFrom:
- configMapRef:
name: cmone
All K,v pairs from cmone will be loaded to the container as env variables. env name = k and env value = v
If we want to load individual k,v pair use below syntax
- name: KAA
valueFrom:
configMapKeyRef:
name: cmtwo
key: KA
- name: KBB
valueFrom:
configMapKeyRef:
name: cmtwo
key: KB
Here we can change env variable name also.
Same thing applies for secret
envFrom:
- secretRef:
name: sone
and
- name: KAA
valueFrom:
secretKeyRef:
name: stwo
key: KA
- name: KBB
valueFrom:
secretKeyRef:
name: stwo
key: KB
We can also define a readiness probe with command ls /path/for/volumeMount. So when the configmap is loaded to container then only the container becomes ready.
Deployment Status
k get deploy grafana -o yaml
At the end Status is present.
1. availableReplicas
2. observedGeneration : For rollout and rollback situation. This parameter specifies current revision number.
Rolling update by
1. changing replica
k scale deploy "name of deployment" --replicas="new count for replica Count"
2. edit deployment and change container image with other version
A deployment has strategy with value “Recreate” or “RollingUpdate”.
We can use --record option to set annotation in deployment. then with command we can rollback. The --record option put annotation in CHANGE-CAUSE column. Once we use --record, then same annotation is continued to future upgrade also, if we do not use --record in future.
The rollback is possible with "rollout undo' command as below. Suppose current version is 5 and if you rollback to 2. Then new version 6 is created. version 2 is removed. record of version 2 is automatically added to 6. if you are at version 5, and you rollback with 'rollout undo' command then we are going back to revision 4. new revision is created as revision 6. So 4 is renamed as 6. You can see revisions 1, 2, 3, 5, 6
The rollback is possible with "rollout undo' command as below. Suppose current version is 5 and if you rollback to 2. Then new version 6 is created. version 2 is removed. record of version 2 is automatically added to 6. if you are at version 5, and you rollback with 'rollout undo' command then we are going back to revision 4. new revision is created as revision 6. So 4 is renamed as 6. You can see revisions 1, 2, 3, 5, 6
k rollout undo deploy "name of deployment"
rollback to specific version with option --to-revision='number'
k rollout undo deploy "name of deployment" --to-revision=1
When we create deployment, replicaset A is automatically created.
When we rollout, new replica set B is created. B's replica count increase from 0 to rpelica count of old A replicaset. A's replicacount will decrease from original value to 0. If we rollback (rollout undo) then reverse will happen.
if we use 'k rolling-update' command, then update will be stopped if client is closed.
When we create deployment, replicaset A is automatically created.
When we rollout, new replica set B is created. B's replica count increase from 0 to rpelica count of old A replicaset. A's replicacount will decrease from original value to 0. If we rollback (rollout undo) then reverse will happen.
if we use 'k rolling-update' command, then update will be stopped if client is closed.
We can see status
k rollout status deploy "name of deployment"
We can see all events about new pod creation and old pod termination with command
k describe deploy "name of deployment"
we can pause and resume deployment.
k rollout pause deploy "name of deployment"
k rollout resume deploy "name of deployment"
k rollout history deploy "name of deployment"
provides all the revisions.
We can see specific revision
k rollout history deploy "name of deployment" --revision=1
We can see diff
k rollout history deploy "name of deployment" --revision=1 > one.out
k rollout history deploy "name of deployment" --revision=1 > two.out
We can see specific revision
k rollout history deploy "name of deployment" --revision=1
We can see diff
k rollout history deploy "name of deployment" --revision=1 > one.out
k rollout history deploy "name of deployment" --revision=1 > two.out
diff one.out two.out
The changes are:
1. line 1 has revision number
2. pod-template-hash
3. Image
If we change replica count, then new revision is not created.
We can trigger rollout with command
k set image deploy "name of deployment" "name of container"=nginx:1.9.1
The changes are:
1. line 1 has revision number
2. pod-template-hash
3. Image
If we change replica count, then new revision is not created.
We can trigger rollout with command
k set image deploy "name of deployment" "name of container"=nginx:1.9.1
Real-Time Kubernetes Debugging, Monitoring and Alerting with BotKube
Ref: https://www.meetup.com/k8s-cloudnative-online/events/269706847
My take away points
Real-Time Kubernetes Debugging
1 Describe
* describe pod and see event
* describe service and see endpoint
2 Events
* kubectl get events
gives events in that namespace
* kubectl get events --all-namespace
3 exec with -it (interactive terminal) for pod. Use -c to run inside specific container.
4. kubectl debug is K8s 1.18 alpha feature. First enable alpha feature. We can dynamically insert container in pod, for debug purpose.
5. Logs
* kubectl logs,
Container shall expose its logs using stdout
* kubectl logs --previous
provides log for when it run last time. It is useful if any pod is keep restating.
Grafana Dashboard: https://grafana.com/grafana/dashboards?orderBy=name&direction=asc
For K8s + Prometheus setup, relevant dashboards are : https://grafana.com/grafana/dashboards?dataSource=prometheus&direction=asc&orderBy=name&search=kubernetes
Select one dashboard. import in you Grafana with its ID. OR one can download JSON file and import the JSON file
BotKube
BotKube is for debugging and monitoring
it can run kubectl also
kubectl get clusterroles
give all who have cluster role
BotKube can work with multiple cluster
We can add custom filter in GoLang to BotKube: https://www.botkube.io/filters/
BotKube next release
- It will be integrated with Microsoft Team
- It will have json path based status:field monitoring
Reference:
https://github.com/infracloudio/botkube
https://www.botkube.io
From Live Chat
* if service has two label and pod has only 1 out of 2 labels then also that pod is part of service due to OR condition.
* deployments are not associated with any node, so one cannot get node IP address from deployment
* Better go with Prometheus Operator Model that is a part of Kube-Prometheus project. It'll install node-exporter that is a daemonset. It'll act as a agent for nodes
* Prometheus can work with docker container also, without K8s
* splunk and kibana both can be combined.
* https://blogs.halodoc.io/production-grade-eks-logging-with-efk-via-sidecar/
* how scaling is designed for botkube ? once is a while, there can be many events from K8s cluster . Most of the time, there are no events from k8s cluster.
Scaling is using K8s HPA only. Even we have so many events, each event processing needs little CPU.
Meetup Recording : https://www.youtube.com/watch?v=bGnQep5bY6c
My take away points
Real-Time Kubernetes Debugging
1 Describe
* describe pod and see event
* describe service and see endpoint
2 Events
* kubectl get events
gives events in that namespace
* kubectl get events --all-namespace
3 exec with -it (interactive terminal) for pod. Use -c to run inside specific container.
4. kubectl debug is K8s 1.18 alpha feature. First enable alpha feature. We can dynamically insert container in pod, for debug purpose.
5. Logs
* kubectl logs,
Container shall expose its logs using stdout
* kubectl logs --previous
provides log for when it run last time. It is useful if any pod is keep restating.
Grafana Dashboard: https://grafana.com/grafana/dashboards?orderBy=name&direction=asc
For K8s + Prometheus setup, relevant dashboards are : https://grafana.com/grafana/dashboards?dataSource=prometheus&direction=asc&orderBy=name&search=kubernetes
Select one dashboard. import in you Grafana with its ID. OR one can download JSON file and import the JSON file
BotKube
BotKube is for debugging and monitoring
it can run kubectl also
kubectl get clusterroles
give all who have cluster role
BotKube can work with multiple cluster
We can add custom filter in GoLang to BotKube: https://www.botkube.io/filters/
BotKube next release
- It will be integrated with Microsoft Team
- It will have json path based status:field monitoring
Reference:
https://github.com/infracloudio/botkube
https://www.botkube.io
From Live Chat
* if service has two label and pod has only 1 out of 2 labels then also that pod is part of service due to OR condition.
* deployments are not associated with any node, so one cannot get node IP address from deployment
* Better go with Prometheus Operator Model that is a part of Kube-Prometheus project. It'll install node-exporter that is a daemonset. It'll act as a agent for nodes
* Prometheus can work with docker container also, without K8s
* splunk and kibana both can be combined.
* https://blogs.halodoc.io/production-grade-eks-logging-with-efk-via-sidecar/
* how scaling is designed for botkube ? once is a while, there can be many events from K8s cluster . Most of the time, there are no events from k8s cluster.
Scaling is using K8s HPA only. Even we have so many events, each event processing needs little CPU.
Meetup Recording : https://www.youtube.com/watch?v=bGnQep5bY6c
K8s Dashboard
https://k8slens.dev/ and https://github.com/lensapp/lens
https://github.com/kubernetes/dashboard
https://github.com/kubevious/kubevious
https://github.com/holandes22/kube-admin
http://kube-admin-demo.s3-website-us-east-1.amazonaws.com/
https://github.com/supernova106/kubestorm
https://github.com/containerum/containerum
https://github.com/kuberneto/kuberneto
https://kuberneto.com/
https://github.com/vmware-tanzu/octant
https://github.com/derailed/k9s
https://github.com/kubernetes/dashboard
https://github.com/kubevious/kubevious
https://github.com/holandes22/kube-admin
http://kube-admin-demo.s3-website-us-east-1.amazonaws.com/
https://github.com/supernova106/kubestorm
https://github.com/containerum/containerum
https://github.com/kuberneto/kuberneto
https://kuberneto.com/
https://github.com/vmware-tanzu/octant
https://github.com/derailed/k9s
CKAD : 4.Design
Each component should be decoupled from resources.
All resources have transient relationship with others.
K8s orchestration works with series of agents = controllers = watch loops.
PodSpec determines the best node for deployment
One CPU =
1 AWS vCPU or
1 GCP Core
1 Azure vCore
1 Hyperthread in bare metal Intel CPU with Hyperthreading
Label and Selector
Label can be
- production / development
- department name
- team name
- primary application developer
Selectors are namespace scoped, unless --all-namespace is used
--selector key=value
OR
-l key=value
If we use command
kubectl create deployment design2 --image=nginx
Then a pod is created with label app=design2
Here we can not specify replicas in same command.
We need to run one more
if we edit label of that pod, then design2 deployment will create another pod with label app=design2
if we delete deployment design2 then all pod whose label is app=design2, those will only get deleted.
To show all labels use --show-labels options with kubectl get command.
Job
Job is for scenarios when you don’t want the process to keep running indefinitely.
It does support parallel processing of a set of independent but related work items. These might be
- emails to be sent,
- frames to be rendered,
- files to be transcoded,
- ranges of keys in a NoSQL database to scan, and so on.
A Replication Controller manages Pods which are not expected to terminate (e.g. web servers), and a Job manages Pods that are expected to terminate (e.g. batch tasks).
Jobs are part of batch API group
It has following parameters
1. activeDeadlineSeconds it can remain alive for that many seconds only.
2. completions : How many instance ? default 1. We can edit it using
k edit job 'job name'
3. parallelism : how many should run at a time ? default 1. with value 0, the job is paused.
4. restartPolicy : {Never | OnFailure } Default is Always. Default is not suitable for Job
5. It is restarted for backoffLimit times default 6
6. ttlSecondsAfterFinished default 'never'
7. backoffLimit: Total how many job shall be created.
* If parallelism > completions then parallelism = completions
So logic create 'parallelism' number of pod in one shot. Check how many are successful. It is equal or more than completions then stop. Else create another set of 'parallelism' number of pod. Continue untill total pod creation count is less than backoffLimit.
While debugging set restartpolicy = never. This policy applies to pod not to job
The job status is Failed if
- restarted more than backoffLimit times OR
- it run more than activeDeadlineSeconds
else status is Complete
To create a job with imperative command:
k run 'name of job' --image='container image' --restart=OnFailure
Delete Job
With cascade = false, only job get deleted, not pods. Default is true.
kubectl delete jobs job_name cascade=false
CronJob
Linux style cronjob syntax
MM(minute) HH DD MM(month) WW
It can be list with comma separated value: 1,2
It can be range with hyphen: 1-5
It can be * to indicate all
It can be */ and number to indicate periodic: */2
* and ? has same meaning.
CronJob creates multiple jobs as per schedule. The CronJob is only responsible for creating Jobs that match its schedule, and the Job in turn is responsible for the management of the Pods it represents.
It has following parameters
1. If a cronJob has sleep for 30 seconds and activeDeadlineSeconds is 10 then none of the job created by cronJob get completed state.
2. startingDeadlineSeconds If cornjob cannot scheduled within this time then it is considered as failed. Failure can also because of forbid policy. After 100 such failure, no more job will get scheduled.
Note: If startingDeadlineSeconds is set then failure count is considered in last startingDeadlineSeconds . It should be less then 100.
3. concurrencyPolicy
Allow:
Forbid: If second job is scheduled, before earlier job finished, then it is not allowed
Replace
4. suspend
all subsequent job will not be scheduled.
5. successfulJobsHistoryLimit and failedJobsHistoryLimit
How many job shall be kept
To create CronJob (i.e. cj) using imperative commands
k run 'name of job' --image='container image' --restart=OnFailure --schedule="* * * * *"
Terms for multi container pod
1. Ambassador : Communicate with outside resources / outside cluster. E.g. Envoy Proxy
- Proxy local connection
- Reverse Proxy
- Limits HTTP request
- Re-route to outside world
2. Adapter : Modify data that primary container generates
3. sidecar : helps to provide service that is not found in primary container. E.g. logging
Flexibility : one application per pod
granular scalability : one application per pod
best inter container performance : Multiple application per pod
Containerizing an application
- It should be stateless
- It should be transient
- Remove the environment configuration. It should be via ConfigMap and Secrets
- it is like converting city bus to scooters
After containerization of application, just ask
Q1 : Is my application as decoupled as it could possible be?
Q2 : Are all components design considering other components are transient. Will it work with Chaos Monkey?
Q3 : Can I scale any particular component?
Q4 : Have I used stable and open standard to meet my need?
Managing Resource Usage
If pod ask more CPU then defined, then
- Nothing
If pod ask more memory then defined, then behavior is undefined
- restart pod OR
- evicted node
If pod ask memory more than node has, then
- evicted node
If pod ask more storage then defined, then
- evicted node
Resource Limits
1. CPU: cpu
2. Memory: memory
3. Huge Pages: hugepages-2Mi
4. Ephemeral Storage: ephemeral-storage
they apply at container level.
* By default (if resources->requests are not mentioned), K8s assume container request for 0.5 CPU and 256 Mi memory.
It is set by default LimitRange K8s object
apiVersion: v1
kind: LimitRange
metadata:
name: cpu-limit-range
spec:
limits:
- default:
cpu: 1
defaultRequest:
cpu: 0.5
type: Container
* By default (if resources->limits are not mentioned), K8s assume container limit for 1 vCPU and 512 Mi memory.
It is set by default LimitRange K8s object
apiVersion: v1
kind: LimitRange
metadata:
name: mem-limit-range
spec:
limits:
- default:
memory: 512Mi
defaultRequest:
memory: 256Mi
type: Container
Pod level value is summation of all container's values
The resources can be specify at project quota level also.
limits:
cpu: "1"
memory: "1Gi"
requests:
cpu: "0.5"
memory: "500Mi"
k describe node "Node Name"
We can specify LimitRange object with default values. It is applicable within namespace. It is applicable if admission controller LimitRanger is enabled.
apiVersion: v1
kind: LimitRange
metadata:
name: limit-mem-cpu-per-container
spec:
limits:
- max:
cpu: "800m"
memory: "1Gi"
min:
cpu: "100m"
memory: "99Mi"
default:
cpu: "700m"
memory: "900Mi"
defaultRequest:
cpu: "110m"
memory: "111Mi"
type: Container
- While creating container, if memory request and limit both are not specified
then default range applies.
- While creating container, if memory request is not specified and limit is specified. Then request value is same as limit
- While creating container, if memory request is specified and limit is not specified. Then limit value is double then request value
CNI
* Some CNI plugins supports Network Policies. E.g. Calico, Canal, Kube Router, Romana, Weave Net
* Some CNI plugins supports encryption of UDP and TCP traffic. E.g. Calico, Kopeio, Weave Net
* Some CNI plugins allows vxlan. E.g. Canal, Flannel, Kopeio-networking, Weave Net
* CNI plugins are layer 2 or layer 3
Layer 2: Canal, Flannel, Kopeio-networking, Weave Net
Layer 3: Calico, Romana, Kube Router
* kubenet is basic CNI. It relis on cloud provider for routing and cross node networking
All resources have transient relationship with others.
K8s orchestration works with series of agents = controllers = watch loops.
PodSpec determines the best node for deployment
One CPU =
1 AWS vCPU or
1 GCP Core
1 Azure vCore
1 Hyperthread in bare metal Intel CPU with Hyperthreading
Label and Selector
Label can be
- production / development
- department name
- team name
- primary application developer
Selectors are namespace scoped, unless --all-namespace is used
--selector key=value
OR
-l key=value
If we use command
kubectl create deployment design2 --image=nginx
Then a pod is created with label app=design2
Here we can not specify replicas in same command.
We need to run one more
if we edit label of that pod, then design2 deployment will create another pod with label app=design2
if we delete deployment design2 then all pod whose label is app=design2, those will only get deleted.
To show all labels use --show-labels options with kubectl get command.
Job
Job is for scenarios when you don’t want the process to keep running indefinitely.
It does support parallel processing of a set of independent but related work items. These might be
- emails to be sent,
- frames to be rendered,
- files to be transcoded,
- ranges of keys in a NoSQL database to scan, and so on.
A Replication Controller manages Pods which are not expected to terminate (e.g. web servers), and a Job manages Pods that are expected to terminate (e.g. batch tasks).
Jobs are part of batch API group
It has following parameters
1. activeDeadlineSeconds it can remain alive for that many seconds only.
2. completions : How many instance ? default 1. We can edit it using
k edit job 'job name'
3. parallelism : how many should run at a time ? default 1. with value 0, the job is paused.
4. restartPolicy : {Never | OnFailure } Default is Always. Default is not suitable for Job
5. It is restarted for backoffLimit times default 6
6. ttlSecondsAfterFinished default 'never'
7. backoffLimit: Total how many job shall be created.
* If parallelism > completions then parallelism = completions
So logic create 'parallelism' number of pod in one shot. Check how many are successful. It is equal or more than completions then stop. Else create another set of 'parallelism' number of pod. Continue untill total pod creation count is less than backoffLimit.
While debugging set restartpolicy = never. This policy applies to pod not to job
The job status is Failed if
- restarted more than backoffLimit times OR
- it run more than activeDeadlineSeconds
else status is Complete
To create a job with imperative command:
k run 'name of job' --image='container image' --restart=OnFailure
Delete Job
With cascade = false, only job get deleted, not pods. Default is true.
kubectl delete jobs job_name cascade=false
CronJob
Linux style cronjob syntax
MM(minute) HH DD MM(month) WW
It can be list with comma separated value: 1,2
It can be range with hyphen: 1-5
It can be * to indicate all
It can be */ and number to indicate periodic: */2
* and ? has same meaning.
CronJob creates multiple jobs as per schedule. The CronJob is only responsible for creating Jobs that match its schedule, and the Job in turn is responsible for the management of the Pods it represents.
It has following parameters
1. If a cronJob has sleep for 30 seconds and activeDeadlineSeconds is 10 then none of the job created by cronJob get completed state.
2. startingDeadlineSeconds If cornjob cannot scheduled within this time then it is considered as failed. Failure can also because of forbid policy. After 100 such failure, no more job will get scheduled.
Note: If startingDeadlineSeconds is set then failure count is considered in last startingDeadlineSeconds . It should be less then 100.
3. concurrencyPolicy
Allow:
Forbid: If second job is scheduled, before earlier job finished, then it is not allowed
Replace
4. suspend
all subsequent job will not be scheduled.
5. successfulJobsHistoryLimit and failedJobsHistoryLimit
How many job shall be kept
To create CronJob (i.e. cj) using imperative commands
k run 'name of job' --image='container image' --restart=OnFailure --schedule="* * * * *"
Terms for multi container pod
1. Ambassador : Communicate with outside resources / outside cluster. E.g. Envoy Proxy
- Proxy local connection
- Reverse Proxy
- Limits HTTP request
- Re-route to outside world
2. Adapter : Modify data that primary container generates
3. sidecar : helps to provide service that is not found in primary container. E.g. logging
Flexibility : one application per pod
granular scalability : one application per pod
best inter container performance : Multiple application per pod
Containerizing an application
- It should be stateless
- It should be transient
- Remove the environment configuration. It should be via ConfigMap and Secrets
- it is like converting city bus to scooters
After containerization of application, just ask
Q1 : Is my application as decoupled as it could possible be?
Q2 : Are all components design considering other components are transient. Will it work with Chaos Monkey?
Q3 : Can I scale any particular component?
Q4 : Have I used stable and open standard to meet my need?
Managing Resource Usage
If pod ask more CPU then defined, then
- Nothing
If pod ask more memory then defined, then behavior is undefined
- restart pod OR
- evicted node
If pod ask memory more than node has, then
- evicted node
If pod ask more storage then defined, then
- evicted node
Resource Limits
1. CPU: cpu
2. Memory: memory
3. Huge Pages: hugepages-2Mi
4. Ephemeral Storage: ephemeral-storage
they apply at container level.
* By default (if resources->requests are not mentioned), K8s assume container request for 0.5 CPU and 256 Mi memory.
It is set by default LimitRange K8s object
apiVersion: v1
kind: LimitRange
metadata:
name: cpu-limit-range
spec:
limits:
- default:
cpu: 1
defaultRequest:
cpu: 0.5
type: Container
* By default (if resources->limits are not mentioned), K8s assume container limit for 1 vCPU and 512 Mi memory.
It is set by default LimitRange K8s object
apiVersion: v1
kind: LimitRange
metadata:
name: mem-limit-range
spec:
limits:
- default:
memory: 512Mi
defaultRequest:
memory: 256Mi
type: Container
The resources can be specify at project quota level also.
limits:
cpu: "1"
memory: "1Gi"
requests:
cpu: "0.5"
memory: "500Mi"
k describe node "Node Name"
We can specify LimitRange object with default values. It is applicable within namespace. It is applicable if admission controller LimitRanger is enabled.
apiVersion: v1
kind: LimitRange
metadata:
name: limit-mem-cpu-per-container
spec:
limits:
- max:
cpu: "800m"
memory: "1Gi"
min:
cpu: "100m"
memory: "99Mi"
default:
cpu: "700m"
memory: "900Mi"
defaultRequest:
cpu: "110m"
memory: "111Mi"
type: Container
- While creating container, if memory request and limit both are not specified
then default range applies.
- While creating container, if memory request is not specified and limit is specified. Then request value is same as limit
- While creating container, if memory request is specified and limit is not specified. Then limit value is double then request value
CNI
* Some CNI plugins supports Network Policies. E.g. Calico, Canal, Kube Router, Romana, Weave Net
* Some CNI plugins supports encryption of UDP and TCP traffic. E.g. Calico, Kopeio, Weave Net
* Some CNI plugins allows vxlan. E.g. Canal, Flannel, Kopeio-networking, Weave Net
* CNI plugins are layer 2 or layer 3
Layer 2: Canal, Flannel, Kopeio-networking, Weave Net
Layer 3: Calico, Romana, Kube Router
* kubenet is basic CNI. It relis on cloud provider for routing and cross node networking
./enter_pod.sh "pod name"
#!/bin/sh
containerId=`kubectl get pods $1 -o jsonpath='{.status.containerStatuses[1].containerID}' | sed -e "s/^docker:\/\///"`
pid=`docker inspect --format {{.State.Pid}} $containerId`
echo $pid
sudo nsenter --target $pid --mount --uts --ipc --net --pid sh
containerId=`kubectl get pods $1 -o jsonpath='{.status.containerStatuses[1].containerID}' | sed -e "s/^docker:\/\///"`
pid=`docker inspect --format {{.State.Pid}} $containerId`
echo $pid
sudo nsenter --target $pid --mount --uts --ipc --net --pid sh
