K8s API
The K8s APIs are grouped as follows:
The below table indicates, composition relationships among all K8s objects, as per K8s 1.17
Reference:
https://kubernetes.io/docs/reference/
https://kubernetes.io/docs/reference/kubectl/overview/
https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md
https://www.cncf.io/blog/2020/03/17/how-to-setup-role-based-access-to-kubernetes-cluster/
https://github.com/kubernetes/kubernetes/issues/7856
https://kubernetes.io/docs/concepts/overview/kubernetes-api/
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md
- api
- apis
- healthz
- logs
- metrics
- version
Out of them, first two are related to cluster functionalities. "api" is also called "core api" and "apis" are called "named api". They are defined as "" in RBAC. The named api (apis) are further classified as following resource groups:
- admission
- admissionregistration.k8s.io
- apiextensions.k8s.io
- apiregistration.k8s.io
- apps
- auditregistration
- authentication.k8s.io
- authorization.k8s.io
- autoscaling
- batch
- certificates.k8s.io
- coordination.k8s.io
- core
- discovery
- events.k8s.io
- extensions
- flowcontrol
- imagepolicy
- monitoring.coreos.com
- networking.k8s.io
- node.k8s.io
- policy
- rbac.authorization.k8s.io
- scheduling.k8s.io
- settings
- storage.k8s.io
- testdata
One can perform following actions (verbs) on these resource groups, if RBAC policy allows.
- list
- get
- create
- delete
- update
- watch
As per K8s 1.17, the K8s resources are grouped as follows:
The below table indicates, composition relationships among all K8s objects, as per K8s 1.17
Reference:
https://kubernetes.io/docs/reference/
https://kubernetes.io/docs/reference/kubectl/overview/
https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md
https://www.cncf.io/blog/2020/03/17/how-to-setup-role-based-access-to-kubernetes-cluster/
https://github.com/kubernetes/kubernetes/issues/7856
https://kubernetes.io/docs/concepts/overview/kubernetes-api/
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md
CKA 7: Security
For all hosts (master nodes and worker nodes)
- Root access should be disabled.
- Password based authentication should be disabled.
- Only SSH key based authentication should be enabled.
Control access to the kube-apiserver
1. who can access? Authentication
-- Username and password :
* Static password file user-details.csv
* It has 4 columns (1) password (2) user name (3) user id (4) group id is optional
* command like option --basic-auth-file to kube-api-server. If kubeadm is used then modify pod YAML for kube-apiserver
* curl command pass: -u "username:password"
* not recommended
-- Username and token : Static token file
* It has 4 columns (1) user-token (2) user name (3) user id (4) group id is optional
* command like option --token-auth-file to kube-api-server. If kubeadm is used then modify pod YAML for kube-apiserver
* curl command pass: --header "Authentication: Bearer TOKEN"
* not recommended
-- Certificates
* curl command pass: --key USER.key --cert USER.crt --cacert ca.crt
-- External authentication provider. E.g. LDAP : Identity service
-- Service accounts
2. what they can do? Authorization
-- RBAC
-- ABAC
-- Node Authorization
-- Webhook mode
TLS
If data is encrypted with public key, then decrypt with private key
If data is encrypted with private key, then decrypt with public key
ssh-keygen command generates id_rsa and id_rsa.pub key pairs
openssl genrsa -out filename.key 1024
openssl rsa -in filename.key -pubout > certificate.pem
if any certificate have .crt OR .pem extension then it has public key
Example: server.crt, server.pem, client.crt, client.pem
If a file has private key then it has extension as .key OR file name has -key
Example: server.key, server-pem.key, client.key, client-key.pem
TLS in K8s
Server: Client
1. kube-apiserver: /CN=kube-apiserver
(1) Other K8s components
(i) kube-scheduler /CN=system:kube-scheduler
(ii) Kube-controller-manager /CN=system:kube-controller-manager
(iii) kube-proxy /CN=system:kube-proxy
(2) external users kubectl /CN=admin/O=system:masters
All of the above use kubeconfig file
2. etcd server : kube-apiserver only /CN=etcd-server
3. kubelet services: Kube-apiserver user name = system:node:NODENAME group name = systems:nodes
K8s cluster may have dedicated CA for 'etcd' as server and 'kube-api server' as client.
Tools to generate certificates
easyrsa
openssl
cfssl
1. Create private key
openssl genrsa -out FILE.key 1024
2. CSR using openssl
openssl req -new -key FILE.key -subj "/CN=NAME" -out FILE.csr
openssl req -new -key FILE.key -subj "/CN=NAME/O=GROUP" -out FILE.csr
openssl req -new -key FILE.key -subj "/CN=NAME/O=GROUP" -out FILE.csr -config openssl.conf
Here openssl.conf will have multiple names that includes DNS and IP
Once the CSR file is generated, it can be view with command
openssl req -in FILE.csr -text --noout
3. Sign the request
openssl x509 -req -in FILE.csr -signkey FILE.key -out FILE.crt
Here, when FILE=CA then it is self-signed using root certificate.
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt
Next for signed by CA, we need to pass both private key of CA ca.key and CA's self-signed certificate, that contains its public key.
openssl x509 -req -in FILE.csr -CA ca.crt -CAkey ca.key -out FILE.crt
View Certificates
openssl x509 -in CERT_FILE_WITH_PATH -text -noout
1. journalctl -u SERVICE_NAME -l
2. k logs POD_NAME
3. If kubctl is not working
docker ps -a
to find out container ID
then
docker logs CONTAINER_ID
Certificates API
1. Create private key
openssl genrsa -out FILE.key 1024
2. CSR using openssl
openssl req -new -key FILE.key -subj "/CN=NAME" -out FILE.csr
3. Create k8s object with YAML file. Kind = CertificateSigningRequest
4. cat FILE.CSR | base64
Use this output in "request:" field at YAML file.
5. k get csr
6. k certificate approve FILE
OR
k certificate deny FILE
7. k get csr FILE -o yaml
8. Extract content of "status->certificate"
9. echo CONTENT | base64 -decode
At master node, Controller-Manager performs all certificate related tasks. It has two controllers (1) CSR-Approving (2) CSR-Signing . The kube-controller-manager is passed with command line arguments "--cluster-signing-cert-file" it points to ca.crt and "--cluster-signing-key-file" it points to ca.key
Kubeconfig
k COMMAND --server K8S_MASTER_NODE:PORT --client-key admin.key --client-certificate admin.crt --certificate-authority ca.crt
k COMMAND --kubeconfig config
config file shall contain
--server K8S_MASTER_NODE:PORT
--client-key admin.key
--client-certificate admin.crt
--certificate-authority ca.crt
If we use curl command and invoke REST API
curl https://URL:6443/api/v1/pods --key admin.key --cert admin.cert --cacert ca.cert
If we use command
k proxy
then we can use curl command as per config file, at port 8001
- Root access should be disabled.
- Password based authentication should be disabled.
- Only SSH key based authentication should be enabled.
Control access to the kube-apiserver
1. who can access? Authentication
-- Username and password :
* Static password file user-details.csv
* It has 4 columns (1) password (2) user name (3) user id (4) group id is optional
* command like option --basic-auth-file to kube-api-server. If kubeadm is used then modify pod YAML for kube-apiserver
* curl command pass: -u "username:password"
* not recommended
-- Username and token : Static token file
* It has 4 columns (1) user-token (2) user name (3) user id (4) group id is optional
* command like option --token-auth-file to kube-api-server. If kubeadm is used then modify pod YAML for kube-apiserver
* curl command pass: --header "Authentication: Bearer TOKEN"
* not recommended
-- Certificates
* curl command pass: --key USER.key --cert USER.crt --cacert ca.crt
-- External authentication provider. E.g. LDAP : Identity service
-- Service accounts
2. what they can do? Authorization
-- RBAC
-- ABAC
-- Node Authorization
-- Webhook mode
TLS
If data is encrypted with public key, then decrypt with private key
If data is encrypted with private key, then decrypt with public key
ssh-keygen command generates id_rsa and id_rsa.pub key pairs
openssl genrsa -out filename.key 1024
openssl rsa -in filename.key -pubout > certificate.pem
if any certificate have .crt OR .pem extension then it has public key
Example: server.crt, server.pem, client.crt, client.pem
If a file has private key then it has extension as .key OR file name has -key
Example: server.key, server-pem.key, client.key, client-key.pem
TLS in K8s
Server: Client
1. kube-apiserver: /CN=kube-apiserver
(1) Other K8s components
(i) kube-scheduler /CN=system:kube-scheduler
(ii) Kube-controller-manager /CN=system:kube-controller-manager
(iii) kube-proxy /CN=system:kube-proxy
(2) external users kubectl /CN=admin/O=system:masters
All of the above use kubeconfig file
2. etcd server : kube-apiserver only /CN=etcd-server
3. kubelet services: Kube-apiserver user name = system:node:NODENAME group name = systems:nodes
K8s cluster may have dedicated CA for 'etcd' as server and 'kube-api server' as client.
Tools to generate certificates
easyrsa
openssl
cfssl
1. Create private key
openssl genrsa -out FILE.key 1024
2. CSR using openssl
openssl req -new -key FILE.key -subj "/CN=NAME" -out FILE.csr
openssl req -new -key FILE.key -subj "/CN=NAME/O=GROUP" -out FILE.csr
openssl req -new -key FILE.key -subj "/CN=NAME/O=GROUP" -out FILE.csr -config openssl.conf
Here openssl.conf will have multiple names that includes DNS and IP
Once the CSR file is generated, it can be view with command
openssl req -in FILE.csr -text --noout
3. Sign the request
openssl x509 -req -in FILE.csr -signkey FILE.key -out FILE.crt
Here, when FILE=CA then it is self-signed using root certificate.
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt
Next for signed by CA, we need to pass both private key of CA ca.key and CA's self-signed certificate, that contains its public key.
openssl x509 -req -in FILE.csr -CA ca.crt -CAkey ca.key -out FILE.crt
View Certificates
openssl x509 -in CERT_FILE_WITH_PATH -text -noout
1. journalctl -u SERVICE_NAME -l
2. k logs POD_NAME
3. If kubctl is not working
docker ps -a
to find out container ID
then
docker logs CONTAINER_ID
Certificates API
1. Create private key
openssl genrsa -out FILE.key 1024
2. CSR using openssl
openssl req -new -key FILE.key -subj "/CN=NAME" -out FILE.csr
3. Create k8s object with YAML file. Kind = CertificateSigningRequest
4. cat FILE.CSR | base64
Use this output in "request:" field at YAML file.
5. k get csr
6. k certificate approve FILE
OR
k certificate deny FILE
7. k get csr FILE -o yaml
8. Extract content of "status->certificate"
9. echo CONTENT | base64 -decode
At master node, Controller-Manager performs all certificate related tasks. It has two controllers (1) CSR-Approving (2) CSR-Signing . The kube-controller-manager is passed with command line arguments "--cluster-signing-cert-file" it points to ca.crt and "--cluster-signing-key-file" it points to ca.key
Kubeconfig
k COMMAND --server K8S_MASTER_NODE:PORT --client-key admin.key --client-certificate admin.crt --certificate-authority ca.crt
k COMMAND --kubeconfig config
config file shall contain
--server K8S_MASTER_NODE:PORT
--client-key admin.key
--client-certificate admin.crt
--certificate-authority ca.crt
Default path for config file is ~/.kube
If config file is located at this path, then no need to specify --kubeconfig config
If we use curl command and invoke REST API
curl https://URL:6443/api/v1/pods --key admin.key --cert admin.cert --cacert ca.cert
If we use command
k proxy
then we can use curl command as per config file, at port 8001
config file has 3 sections
1. clusters
k config set-cluster CLUSTER_NAME
certificate-authority OR certificate-authority-data. content for certificate-authority-data is set using cat ca.crt | base64
k config set-cluster CLUSTER_NAME
certificate-authority OR certificate-authority-data. content for certificate-authority-data is set using cat ca.crt | base64
2. users
k config set-credentials USER
k config set-credentials USER
3. contexts = user@cluster + ns
k config set-context CONTEXT_NAME
4. current-context
k config use-context USER1@CLUSTER2 //it will change current-context in file.
Linke with this command
k COMMAND --server K8S_MASTER_NODE:PORT --client-key admin.key --client-certificate admin.crt --certificate-authority ca.crt
Here
--server K8S_MASTER_NODE:PORT //clusters section
--client-key admin.key //users section
--client-certificate admin.crt //users section
--certificate-authority ca.crt //clusters section
Kubectl commands
k config view // to view config file at default path ~/.kube
k config view --kubeconfig=CUSTOM_CONFIG_FILE_WITH_PATH
RBAC
k auth can-i VERB_ACTION RESOURCE_OBJECT
k auth can-i VERB_ACTION RESOURCE_OBJECT --as USER
k auth can-i VERB_ACTION RESOURCE_OBJECT --as USER --namespace NS
Image Security
If image name is xyz then it is xyz/xyz = user account/image name
Here default registry = docker.io
so xyz is docker.io/xyz/xyz
Google registry = gcr.io
K8s has default secret type with name docker-registry. it has keys
- doceker-server
- docker-username
- docker-password
- docker-email
in YAML we shall specify
imagePullSecrets:
- name: docker-registry
Also refer: http://layers7.blogspot.com/2020/04/ckad-6security.html
K8s Secret
at EncryptionConfiguration Kind we have providers
- identity : default. encryption = none
other options
- aescbc // secret is prefixed with k8s:enc:aescbc:v1:
- secretbox
- aesgcm
- kms //Key Management Service
32 byte key is generated with
head -c 32 /dev/urandom | base64
Set the --encryption-provider-config flag on the kube-apiserver to point to the location of the config file: YAML with kind = EncryptionConfiguration
to update all existing secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
k config set-context CONTEXT_NAME
4. current-context
k config use-context USER1@CLUSTER2 //it will change current-context in file.
Linke with this command
k COMMAND --server K8S_MASTER_NODE:PORT --client-key admin.key --client-certificate admin.crt --certificate-authority ca.crt
Here
--server K8S_MASTER_NODE:PORT //clusters section
--client-key admin.key //users section
--client-certificate admin.crt //users section
--certificate-authority ca.crt //clusters section
Kubectl commands
k config view // to view config file at default path ~/.kube
k config view --kubeconfig=CUSTOM_CONFIG_FILE_WITH_PATH
RBAC
k auth can-i VERB_ACTION RESOURCE_OBJECT
k auth can-i VERB_ACTION RESOURCE_OBJECT --as USER
k auth can-i VERB_ACTION RESOURCE_OBJECT --as USER --namespace NS
Image Security
If image name is xyz then it is xyz/xyz = user account/image name
Here default registry = docker.io
so xyz is docker.io/xyz/xyz
Google registry = gcr.io
K8s has default secret type with name docker-registry. it has keys
- doceker-server
- docker-username
- docker-password
- docker-email
in YAML we shall specify
imagePullSecrets:
- name: docker-registry
Also refer: http://layers7.blogspot.com/2020/04/ckad-6security.html
K8s Secret
at EncryptionConfiguration Kind we have providers
- identity : default. encryption = none
other options
- aescbc // secret is prefixed with k8s:enc:aescbc:v1:
- secretbox
- aesgcm
- kms //Key Management Service
32 byte key is generated with
head -c 32 /dev/urandom | base64
Set the --encryption-provider-config flag on the kube-apiserver to point to the location of the config file: YAML with kind = EncryptionConfiguration
to update all existing secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
GitOps
What is GitOps?
* GitOps is code-based infrastructure and operational procedures that rely on Git as a source control system
* For Kubernetes GitOps means using git push instead of kubectl create/apply or helm install/upgrade.
* It’s an evolution of Infrastructure as Code (IaC) and a DevOps best practice that leverages Git as the single source of truth, and control mechanism for creating, updating, and deleting system architecture.
* Git is a single source of truth for
- CD
- automated deployment
- monitoring
- management
- entire state of the system
* GitOps extends pipelines with a feedback loop for observing and controlling the system .
* A set of practices to use "Git pull requests" (1) to manage infrastructure and (2) to manage application configurations (3) automatically deploy system infrastructure modifications. All git procedures are applies like: review, pull request, push requests, tagging, versioning etc.
* GitOps practice
- Declarative description of system : stored in Git
- Pull requests modify the state of the Git repository.
- Once approved and merged, the pull requests will automatically reconfigure and sync/reconcile the live infrastructure to the state of the repository. This is done by controller software. It self-heal the system or notify the drift.
- This live syncing pull request workflow is the core essence of GitOps.
* Declarative Description (YAML) of desired production infrastructure in git + automate to match production environment state with desired state at git
* Continuous Deployment for cloud native applications
* GitOps is an incredibly powerful workflow pattern for managing modern cloud infrastructure
* GitOps is an extension of IaC and declarative configuration
* git push + CI/CD toolchain + UI/UX
* Tools : Git , CD tools (tools for declarative infrastructure as code)
* Developer tools (read git) to drive operations.
* GitOps provide more stability and reliability over typical CI/CD pipeline
The GitOps idea was first hatched and shared by WeaveWorks, an enterprise Kubernetes management firm
GitOps operator is a mechanism that sits between the pipeline and the orchestration system (read K8s) . A pull request starts the pipeline that then triggers the operator. The operator examines the state of the repository and the start of the orchestration and syncs them.
Principles
1. The entire system is described declaratively.
2. The canonical desired system state versioned in Git
3. Approved changes that can be automatically applied to the system.
4. Software agents to ensure correctness and alert on divergence
Tools:
* GitOps is code-based infrastructure and operational procedures that rely on Git as a source control system
* For Kubernetes GitOps means using git push instead of kubectl create/apply or helm install/upgrade.
* It’s an evolution of Infrastructure as Code (IaC) and a DevOps best practice that leverages Git as the single source of truth, and control mechanism for creating, updating, and deleting system architecture.
* Git is a single source of truth for
- CD
- automated deployment
- monitoring
- management
- entire state of the system
* GitOps extends pipelines with a feedback loop for observing and controlling the system .
* A set of practices to use "Git pull requests" (1) to manage infrastructure and (2) to manage application configurations (3) automatically deploy system infrastructure modifications. All git procedures are applies like: review, pull request, push requests, tagging, versioning etc.
* GitOps practice
- Declarative description of system : stored in Git
- Pull requests modify the state of the Git repository.
- Once approved and merged, the pull requests will automatically reconfigure and sync/reconcile the live infrastructure to the state of the repository. This is done by controller software. It self-heal the system or notify the drift.
- This live syncing pull request workflow is the core essence of GitOps.
* Declarative Description (YAML) of desired production infrastructure in git + automate to match production environment state with desired state at git
* Continuous Deployment for cloud native applications
* GitOps is an incredibly powerful workflow pattern for managing modern cloud infrastructure
* GitOps is an extension of IaC and declarative configuration
* git push + CI/CD toolchain + UI/UX
* Tools : Git , CD tools (tools for declarative infrastructure as code)
* Developer tools (read git) to drive operations.
* GitOps provide more stability and reliability over typical CI/CD pipeline
The GitOps idea was first hatched and shared by WeaveWorks, an enterprise Kubernetes management firm
GitOps operator is a mechanism that sits between the pipeline and the orchestration system (read K8s) . A pull request starts the pipeline that then triggers the operator. The operator examines the state of the repository and the start of the orchestration and syncs them.
Principles
1. The entire system is described declaratively.
2. The canonical desired system state versioned in Git
3. Approved changes that can be automatically applied to the system.
4. Software agents to ensure correctness and alert on divergence
Tools:
- ArgoCD: A GitOps operator for Kubernetes with a web interface. It supports multiple clusters and multiple Git repositories.
- Flux: The GitOps Kubernetes operator by the creators of GitOps — Weaveworks No central Management, No central UI.
- Gitkube: A tool for building and deploying docker images on Kubernetes using
git push - kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.
- BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner.
- JenkinsX: Continuous Delivery on Kubernetes with built-in GitOps
- Tkton Pipelines project provides k8s-style resources for declaring CI/CD-style pipelines.
- Terragrunt: A wrapper for Terraform for keeping configurations DRY, and managing remote state
- WKSctl: A tool for Kubernetes cluster configuration management based on GitOps principles
- Helm Operator: An operator for using GitOps on K8s with Helm
- Non K8s tools such as Terraform
- harness.io
- Skaffold.dev handles the workflow for building, pushing and deploying application
- kubediff
- sealed-secrets by Bitnami OR Vault by Hashicorp
Gitops promoted by
* Weaveworks
* Cloudbees
* Bitnami (encrypt K8s secret) https://github.com/bitnami/sealed-secrets
* OpenFaaS
* Hasura
* Ocado
* Financial Times
Benefits
1. Cluster update are sequence of atomic transaction, that can fail or success. GitOps makes it easy so productivity increase.
2. Git provides transaction logs for audit, rollback and teamwork
3. config repo and image repo acts as firewall. so even if CI pipeline is hacked, production environment is not hacked.
Blog Posts and Social Media
CKA 6: Cluster Maintenance
Drain Cordon Uncordon
If a worker node down, then K8s wait for "pod-eviction-timeout" period. Default value is 5 min
This value is passed to kube-control-manager as command line argument.
so if we can upgrade worker node within 5 min, it is ok. Else safer way is:
k drain "worker node name"
it also means the worker node is cordoned, i.e. no pod can be scheduled on it. So once it is back, we should uncorden it
k uncorden "worker node name"
K8s Version
It applies to: (1) kube-apiserver (2) controller-manager (3) kube-scheduler (4) kubelet (5) kube-proxy (6) kubectl
It does not apply to (1) etcd cluster (2) coreDNS
Cluster Upgrade
If Kube-apiserver has version X then
(1) controller-manager and (2) kube-scheduler can be with version [X - 1, X]
(1) kubelet and (2) kube-proxy can be with version [X - 2, X - 1, X]
kubectl can be with version [X - 1, X, X + 1]
* We should always upgrade one minor release at a time
kubeadm upgrade plan //Gives all the information about present version and latest available version
kubeadm upgrade apply "New version"
On master node
1. Upgrade kubeadm tool itself.
2. kubeadm upgrade plan "New version"hold hold is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
3. kubeadm upgrade apply "New version"
k get nodes command show version of kublet, not version of kube-apiserver
3. if kubelet is present at master node then upgrade it
4. systemctl restart kubelet
on worker node
1. k drain "worker node name"
2. upgrade kubeadm
3. upgrade kubelet
4. kubeadm upgrade node config --kubelet-version "new version"
5. systemctl restart kubelet
6. k uncorden "worker node name"
OR
1. k drain "worker node name"
2. kubeadm upgrade node
3. upgrade kubelet
4. k uncorden "worker node name"
If we run k drain command for master node, then also it applies only to user applications. The K8s applications running on master node will not evacuated.
"apt-mark hold" is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
Backup and Restore
1. YAML files
k get all -A -o yaml > all.yaml
OR use tools
ARK by HeptIO is now called Velero
instead of getting backup of YAML file, take back up etcd cluster.
2. etcd cluster
2.1 etcd started with "--data-dir" option. Its default value is /var/lib/etcd
If the snapshot is copied from the data directory, there is no integrity hash and it will only restore by using --skip-hash-check.
2.2 ETCDCTL_API=3 etcdctl snapshot save snapshot.db
To restore
A. service kube-apiserver stop
B. ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --name="" --data-dir "" --initial-cluster "" --initial-cluster-token "" --initial-advertise-peer-urls "" --data-dir=""
C. Preferably use different (1) initial-cluster-token and (2) data-dir. change etcd configuration with these 2 new values
D. service etcd restart
E. service kube-apiserver start
We must specify (1) endpoints (2) cacert (3) cert and (4) key
In managed K8s, we may not have access to etcd. So back up of YAML file is better approach.
3. If etcd is running on a storage volume that supports backup, such as Amazon Elastic Block Store, back up etcd data by taking a snapshot of the storage volume.
Reference
https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md
https://www.youtube.com/watch?v=qRPNuT080Hk
If a worker node down, then K8s wait for "pod-eviction-timeout" period. Default value is 5 min
This value is passed to kube-control-manager as command line argument.
so if we can upgrade worker node within 5 min, it is ok. Else safer way is:
k drain "worker node name"
it also means the worker node is cordoned, i.e. no pod can be scheduled on it. So once it is back, we should uncorden it
k uncorden "worker node name"
K8s Version
It applies to: (1) kube-apiserver (2) controller-manager (3) kube-scheduler (4) kubelet (5) kube-proxy (6) kubectl
It does not apply to (1) etcd cluster (2) coreDNS
Cluster Upgrade
If Kube-apiserver has version X then
(1) controller-manager and (2) kube-scheduler can be with version [X - 1, X]
(1) kubelet and (2) kube-proxy can be with version [X - 2, X - 1, X]
kubectl can be with version [X - 1, X, X + 1]
* We should always upgrade one minor release at a time
kubeadm upgrade plan //Gives all the information about present version and latest available version
kubeadm upgrade apply "New version"
On master node
1. Upgrade kubeadm tool itself.
2. kubeadm upgrade plan "New version"hold hold is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
3. kubeadm upgrade apply "New version"
k get nodes command show version of kublet, not version of kube-apiserver
3. if kubelet is present at master node then upgrade it
4. systemctl restart kubelet
on worker node
1. k drain "worker node name"
2. upgrade kubeadm
3. upgrade kubelet
4. kubeadm upgrade node config --kubelet-version "new version"
5. systemctl restart kubelet
6. k uncorden "worker node name"
OR
1. k drain "worker node name"
2. kubeadm upgrade node
3. upgrade kubelet
4. k uncorden "worker node name"
If we run k drain command for master node, then also it applies only to user applications. The K8s applications running on master node will not evacuated.
"apt-mark hold" is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
Backup and Restore
1. YAML files
k get all -A -o yaml > all.yaml
OR use tools
ARK by HeptIO is now called Velero
instead of getting backup of YAML file, take back up etcd cluster.
2. etcd cluster
2.1 etcd started with "--data-dir" option. Its default value is /var/lib/etcd
If the snapshot is copied from the data directory, there is no integrity hash and it will only restore by using --skip-hash-check.
2.2 ETCDCTL_API=3 etcdctl snapshot save snapshot.db
To restore
A. service kube-apiserver stop
B. ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --name="" --data-dir "" --initial-cluster "" --initial-cluster-token "" --initial-advertise-peer-urls "" --data-dir=""
C. Preferably use different (1) initial-cluster-token and (2) data-dir. change etcd configuration with these 2 new values
D. service etcd restart
E. service kube-apiserver start
We must specify (1) endpoints (2) cacert (3) cert and (4) key
In managed K8s, we may not have access to etcd. So back up of YAML file is better approach.
3. If etcd is running on a storage volume that supports backup, such as Amazon Elastic Block Store, back up etcd data by taking a snapshot of the storage volume.
Reference
https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md
https://www.youtube.com/watch?v=qRPNuT080Hk
CKA 5: Application Life Cycle Management
Init Containers
initContainers:
It is a list
Each init container run, one at a time in sequential order.
It is like job. Run to complete
If any one fails then pod will be restarted, unless pod has restartPolicy = Never
The init container does not support
- lifecycle,
- livenessProbe,
- readinessProbe,
- startupProbe
Pod-> activeDeadlineSeconds and pod->container->liveness probe , prevents init containers from restarting from ever.
Resources
Highest value among all init containers is considered for resources->request and resources->limit
For pod effective limit for resources->request and resources->limit = Highest ( Highest (init containers) , summation ( all normal containers) )
This effect limit is considered to schedule a pod
The resources reserved by init container(s) is not used once they are completed, by other containers.
If we change image of container then only that container is restarted. If we change image of init container then whole pod is restarted.
initContainers:
It is a list
Each init container run, one at a time in sequential order.
It is like job. Run to complete
If any one fails then pod will be restarted, unless pod has restartPolicy = Never
The init container does not support
- lifecycle,
- livenessProbe,
- readinessProbe,
- startupProbe
Pod-> activeDeadlineSeconds and pod->container->liveness probe , prevents init containers from restarting from ever.
Resources
Highest value among all init containers is considered for resources->request and resources->limit
For pod effective limit for resources->request and resources->limit = Highest ( Highest (init containers) , summation ( all normal containers) )
This effect limit is considered to schedule a pod
The resources reserved by init container(s) is not used once they are completed, by other containers.
If we change image of container then only that container is restarted. If we change image of init container then whole pod is restarted.
