CKS Tips
1. Shortcut
export do="--dry-run=client -o yaml" # k get pod x $do
export now="--force --grace-period 0" # k delete pod x $now
export now="--force --grace-period 0" # k delete pod x $now
alias kn="k -n "namespace name""
2. VIM related
2.1 edit ~/.vimrc
set ts=2
set et
set sw=2
set nu
2.2 To search in vi "/var/lib"
We should issue comman
/\/var\/lib
Here \ is used as special character
2.3 To change last word of line
use "$" to go end of line
then move one word back using "b"
then "cw" to change word.
If needed use "c$" to remove and replace till end of line.
3. How to create pod, without YAML
kubectl run nginx --image=nginx --dry-run=client -o yaml > pod.yaml
4.If we have file with name a.txt and its sha512. then steps:
1. first create and open file a.txt.sha512
2. enter value of sha512
3. add two times space
4. add file name. here it is a.txt
5. close a.txt.sha512 file
6. run command: sha512sum -c a.txt.sha512
5. OPA and gatekeeper. We can list constraint with
k get constraint
NAME AGE
blacklistimages.constraints.gatekeeper.sh/pod-trusted-images 10m
requiredlabels.constraints.gatekeeper.sh/namespace-mandatory-labels 10m
Now, Here first part is constraint template and second part is constraint. So we can edit / view constraint
k get blacklistimages pod-trusted-images -o yaml
k get requiredlabels namespace-mandatory-labels -o yaml
To get template
k get constrainttemplate
Instead of typing such long spelling, easy way
1. k get crd
Then copy paste
2. k get constrainttemplates.templates.gatekeeper.sh
6. To find image
k get po -o=custom-columns=Image:"spec.containers[*].image"
k get po -o=custom-columns=Name:"metadata.name",Image:"spec.containers[0].image"
7. Useful command to set namespace
kubectl config set-context --current --namespace="namespace"
8. To install appArmor profile, we shall use
sudo apparmor_parser -q
sudo apparmot_status
will provide list of all profiles: loaded, complain mode, enforce mode
9. Mount secret as volume
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
10. securityContext.capabilities is only for container, not for pod
11. securityContext.readOnlyRootFilesystem is only for container, not for pod. Here we should use word "Root" and s is small in system.
12. Instead of applying (1) label to node and (2) then use nodeSelector, we can use nodeName in pod spec.
spec:
nodeName: cluster1-worker2 # add
13. To run command inside pod and take its output to outside pod. here the final command to be run inside the pod should be at the end.
k -n team-purple exec gvisor-test > /opt/course/10/gvisor-test-dmesg -- dmesg
14. To run etcdctl
14.1. start with
ETCDCTL_API=3
ETCDCTL_API=3
14.2. Check input parameter of api server at master node
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
14.3. Now do mapping. parameter value in kube-apiserver.yaml to input argument for etcd
--etcd-cafile mapped to --cacert
--etcd-certfile mapped to --cert
--etcd-keyfile mapped to --key
--etcd-certfile mapped to --cert
--etcd-keyfile mapped to --key
14.4. then add below to command
get /registry/"k8s resource type"/"namespace name"/"k8s resource name"
So the complete command will be:
ETCDCTL_API=3 etcdctl --cacert "Path as per --etcd-certfile" --cert "path as per --etcd-certfile" --key "path as per --etcd-keyfile" get /registry/"K8s resource typ"/"name of the namespace"/"name of k8s resource"
There is no "=" for option/argument and value.
For details on Minikube please refer: http://layers7.blogspot.com/2020/06/minikube-etcd.html
15. For any pod, if automountServiceAccountToken is true OR not false OR absent, then we can log in to that pod. Whatever that pod has access, as per its SA, same we can get as follows.
15.1. Its SA token is located/mounted inside pod
/run/secrets/kubernetes.io/serviceaccount/token
15.2. We can use this token as Bearer HTTP header.
15.3. We can form URI = https://kubernetes.default/api/v1/namespaces/"name of namespace/"k8s resource type"/"k8s resource name"
Here namespaces is plural.
15.4. we should add -k option to curl command
So complete command will be
curl https://kubernetes.default/api/v1/namespaces/"name of namespace/"k8s resource type"/"k8s resource name" -H "Authorization
: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -k
See the difference
etcdctl: /registry/"k8s resource type"/"namespace name"/"k8s resource name"
curl URL: /api/v1/namespaces/"name of namespace/"k8s resource type"/"k8s resource name"
In short:
etcdctl : first K8s resource then NS
curl URL: first NS then K8s resource. we need additional "namespaces"
16. If secret is mounted as volume or as ENV variable, then also we can access it using exec inside pod.
17. TLS type secret
kubectl create secret tls my-tls-secret \
--cert=path/to/cert/file \
--key=path/to/key/file
* This TLS type secret we can use in ingress resource
tls:
- hosts:
- sslexample.foo.com
secretName: testsecret-tls
Please note: Here both k8s resources (1) ingress and (2) TLS type secret, should belong to same namespace
18. Few useful podman command. Same as docker command
podman build -t "tag or registryFQDN/user/image:version" .
Do not forget last "dot" It indicates PWD. Dockerfile from PWD
One can also mention other path of directory. not path of Dockerfile
podman push "registryFQDN/user/image:version"
19. This command will tell uid, gid and groups (fsgroup)
19.1 id
19.2 cat /etc/passwd
20. Note : all "resources" are always in plural in all k8s api server audit policy file. e.g. cronjobs, podss etc.
21. We can specify readOnlyRootFileSystem as true inside container securityContext. Then to allow specific path as writeable, mount it as emptyDir
volumeMounts: # add
- mountPath: /tmp # add
name: temp-vol # add
volumes: # add
- name: temp-vol # add
emptyDir: {} # add
22. To upgrade k8s version
* k get node
It give kubelet version
22.1 k corden master --ignore-daemonsets
Note DaemonSet is plural
22.2 now log in to master
22.2.1 first update kubeadm
apt-get install kubeadm=1.x.y-00
Note: there is no V here
Before installing anything, we shall perform
apt-get update
22.2.2 kubeadm upgrade plan
It will suggest next command
22.2.3 kubeadm upgrade apply v1.x.y
Note: Here V is preesent
Now, optionally one can verify the upgrade is successful or not by running same command again
kubeadm upgrade plan
22.2.4 Now upgrade kubelet version
apt-get install kubelet=1.22.1-00
Note: No V here
22.2.5 Now optionally restart kubelet
systemctl daemon-reload && systemctl restart kubelet
22.2.6 exit from master node
22.3 k uncorden master
22.4 Now log in to worker
Here instead of
22.2.2 kubeadm upgrade plan
22.2.3 kubeadm upgrade apply v1.x.y
We have to use
22.4.1 kubeadm upgrade node
now run same as 22.2.4 to 22.2.6 and update kubelet
22.5 same as 22.3 now uncordon worker
k uncordon worker
Basically
Master
cordon
Install new kuebadm, kubelet
kubeadm upgrade plan
kubeadm upgrade apply "newVersion"
uncordon
Worker
Same steps, except 2 commands for kubeadm, here only one command
kubeadm upgrade node
23. We should never echo ENV, which is loaded from secret
24. Never put password in plain text in YAML file
25. in DockerFile, if you copy a secret token one command using COPY or ADD. and in second line if you use RUN command and remove using "rm" command, then also it remains in previous layer. Avoid it.
26. See various command start with
kubectl config -h
27. To get pod from container ID
27.1 crictl ps --id "container ID"
It will give pod id
27.2 crictl pods --id "pod id"
Note: It is pods , not pod
28. Annotation for apparmor profile.
- for pod it is : metadata.annotations
- for deployment it is: spec.template.metadata.annotations
We can annotate pod using kubectl imperative command. Not possible for deployment
k annotate pod "pod name" k1=v1 k2=v2 k3=v3
29. In network policy. podSelector applies to target pod and also in rule. namespaceSelector is only for rule. Both selector needs matchLabels. We cannot provide list of pods, by mentioning name of the pods. We can use matchExressions. However matchEpression also works on label k.v pair only. No other attributes like name, image name, container name etc.
30. Use command c$ to replace from cursor till end of file. VI editor short cut. useful while editing YAML file copied from k8s docs and you wish to change some values. cw is used to change single word. If existing value has - or / then c$ is more useful.
31. For K8s audit, specify log file path, which already exist. E.g. /var/log/audit.log
32. for k8s audit policy
We need to mention group of resource. Do not include version. To find API group, run the command
k api-resources
33 Specify image
33.1 k run for creating pod
kubectl run nginx --image=nginx -n mynamespace
33.2 k create to create deployment, job, cronjob
kubectl create deployment nginx --image=nginx
33.3 k set image, to upgrade. Only here we use container name. "www" in below example
kubectl set image deployment/frontend www=image:v2
34. CIS
34.1 report's sections
You can get by runnning command
grep "INFO" "report file name"
====================================
1. Control plane
1.1 All config files /* all file/folder permission and ownership */
1.2 API server
1.3 Controller-Manager
1.4 Scheduler
2. etcd
3. Control plane config
4. Worker node
4.1 worker node config files /* all file/folder permission and ownership */
4.2 kubelet
5. Policies
5.1 RBAC and SA
5.2 PSP
5.3 netpol and CNI
5.4 Secret mgmt
5.5 extensible admission control
5.7 General
====================================
34.2 section wise failure
You can get by runnning command
grep "check FAIL" "report file name"
35. To remove duplicate
cat "file name" | uniq
Note: It is uniq, not unique
36. All the Kind always start with capital letter. Useful while searching audit log
37. RBAC
37.1 Note:
system:serviceaccount:(singular) is the prefix for service account usernames.system:serviceaccounts:(plural) is the prefix for service account groups.
37.2 Grant a role to all service accounts in a namespace
kubectl create rolebinding serviceaccounts-view \
--clusterrole=view \
--group=system:serviceaccounts:my-namespace \
--namespace=my-namespace
37.3 Grant a limited role to all service accounts cluster-wide (discouraged)
kubectl create clusterrolebinding serviceaccounts-view \
--clusterrole=view \
--group=system:serviceaccounts
39 Important paths
39.1. SA token inside pod
/var/run/secrets/kubernetes.io/serviceaccount/token
39.2. Kube-api manifest file at master node.
/etc/kubernetes/manifest/kube-apiserver.yaml
39.3. seccomp path at worker | master node
/var/lib/kubelet/seccomp/profiles/name.json
In YAML file of pod/deployment we mention
localhostProfile: profiles/name.json
39.4 All apparmor profile file (not profile) are located at worker node
/etc/apparmod.d/
39.5 For runtime class configuration
39.5.1 with containerd
/etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.${HANDLER_NAME}]
39.5.2 with crio
/etc/crio/crio.conf
=============
38. module_request in SELinux
nice to read article: https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/
0 comments:
Post a Comment