Express YourSelf !: December 2021

minikube kubeconfig

from .kube/config file, extract cluster.certificate-authority-data| base64 -d > ca.crt

openssl x509 -in ca.crt -text -noout > ca.crt.decode

From ca.crt.decode:

Issuer: CN = minikubeCA

Subject: CN = minikubeCA

from .kube/config file, extract user.user. client-certificate-data | base64 -d > client.crt

openssl x509 -in client.crt -text -noout > client.crt.decode

From client.crt.decode

Issuer: CN = minikubeCA

Subject: O = system:masters, CN = kubernetes-admin

CKS Tips

1. Shortcut

export do="--dry-run=client -o yaml" # k get pod x $do
export now="--force --grace-period 0" # k delete pod x $now

alias kn="k -n "namespace name""

2. VIM related

2.1 edit ~/.vimrc

set ts=2

set et

set sw=2

set nu

2.2 To search in vi "/var/lib"

We should issue comman

/\/var\/lib

Here \ is used as special character

2.3 To change last word of line

use "$" to go end of line

then move one word back using "b"

then "cw" to change word.

If needed use "c$" to remove and replace till end of line.

3. How to create pod, without YAML

kubectl run nginx --image=nginx  --dry-run=client -o yaml > pod.yaml

4.If we have file with name a.txt and its sha512. then steps:

1. first create and open file a.txt.sha512

2. enter value of sha512

3. add two times space

4. add file name. here it is a.txt

5. close a.txt.sha512 file

6. run command: sha512sum -c a.txt.sha512

5. OPA and gatekeeper. We can list constraint with

k get constraint

NAME AGE

blacklistimages.constraints.gatekeeper.sh/pod-trusted-images 10m

requiredlabels.constraints.gatekeeper.sh/namespace-mandatory-labels 10m

Now, Here first part is constraint template and second part is constraint. So we can edit / view constraint

k get blacklistimages pod-trusted-images -o yaml

k get requiredlabels namespace-mandatory-labels -o yaml

To get template

k get constrainttemplate

Instead of typing such long spelling, easy way

1. k get crd

Then copy paste

2. k get constrainttemplates.templates.gatekeeper.sh

6. To find image

k get po -o=custom-columns=Image:"spec.containers[*].image"

k get po -o=custom-columns=Name:"metadata.name",Image:"spec.containers[0].image"

7. Useful command to set namespace

kubectl config set-context --current --namespace="namespace"

8. To install appArmor profile, we shall use

sudo apparmor_parser -q

sudo apparmot_status

will provide list of all profiles: loaded, complain mode, enforce mode

9. Mount secret as volume

apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret

10. securityContext.capabilities is only for container, not for pod

11. securityContext.readOnlyRootFilesystem is only for container, not for pod. Here we should use word "Root" and s is small in system.

12. Instead of applying (1) label to node and (2) then use nodeSelector, we can use nodeName in pod spec.

spec:

nodeName: cluster1-worker2 # add

13. To run command inside pod and take its output to outside pod. here the final command to be run inside the pod should be at the end.

k -n team-purple exec gvisor-test > /opt/course/10/gvisor-test-dmesg -- dmesg

14. To run etcdctl

14.1. start with
ETCDCTL_API=3

14.2. Check input parameter of api server at master node
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd

14.3. Now do mapping. parameter value in kube-apiserver.yaml to input argument for etcd

--etcd-cafile mapped to --cacert
--etcd-certfile mapped to --cert
--etcd-keyfile mapped to --key

14.4. then add below to command

get /registry/"k8s resource type"/"namespace name"/"k8s resource name"

So the complete command will be:

ETCDCTL_API=3 etcdctl --cacert "Path as per --etcd-certfile" --cert "path as per --etcd-certfile" --key "path as per --etcd-keyfile" get /registry/"K8s resource typ"/"name of the namespace"/"name of k8s resource"

There is no "=" for option/argument and value.

For details on Minikube please refer: http://layers7.blogspot.com/2020/06/minikube-etcd.html

15. For any pod, if automountServiceAccountToken is true OR not false OR absent, then we can log in to that pod. Whatever that pod has access, as per its SA, same we can get as follows.

15.1. Its SA token is located/mounted inside pod

/run/secrets/kubernetes.io/serviceaccount/token

15.2. We can use this token as Bearer HTTP header.

15.3. We can form URI = https://kubernetes.default/api/v1/namespaces/"name of namespace/"k8s resource type"/"k8s resource name"

Here namespaces is plural.

15.4. we should add -k option to curl command

So complete command will be

curl https://kubernetes.default/api/v1/namespaces/"name of namespace/"k8s resource type"/"k8s resource name" -H "Authorization

: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -k

See the difference

etcdctl: /registry/"k8s resource type"/"namespace name"/"k8s resource name"

curl URL: /api/v1/namespaces/"name of namespace/"k8s resource type"/"k8s resource name"

In short:

etcdctl : first K8s resource then NS

curl URL: first NS then K8s resource. we need additional "namespaces"

16. If secret is mounted as volume or as ENV variable, then also we can access it using exec inside pod.

17. TLS type secret

kubectl create secret tls my-tls-secret \

  --cert=path/to/cert/file \

  --key=path/to/key/file

* This TLS type secret we can use in ingress resource

  tls:

  - hosts:

      - sslexample.foo.com

    secretName: testsecret-tls

Please note: Here both k8s resources (1) ingress and (2) TLS type secret, should belong to same namespace

18. Few useful podman command. Same as docker command

podman build -t "tag or registryFQDN/user/image:version" .

Do not forget last "dot" It indicates PWD. Dockerfile from PWD

One can also mention other path of directory. not path of Dockerfile

podman push "registryFQDN/user/image:version"

19. This command will tell uid, gid and groups (fsgroup)

19.1 id

19.2 cat /etc/passwd

20. Note : all "resources" are always in plural in all k8s api server audit policy file. e.g. cronjobs, podss etc.

21. We can specify readOnlyRootFileSystem as true inside container securityContext. Then to allow specific path as writeable, mount it as emptyDir

volumeMounts: # add

- mountPath: /tmp # add

name: temp-vol # add

volumes: # add

- name: temp-vol # add

emptyDir: {} # add

22. To upgrade k8s version

* k get node

It give kubelet version

22.1 k corden master --ignore-daemonsets

Note DaemonSet is plural

22.2 now log in to master

22.2.1 first update kubeadm

apt-get install kubeadm=1.x.y-00

Note: there is no V here

Before installing anything, we shall perform

apt-get update

22.2.2 kubeadm upgrade plan

It will suggest next command

22.2.3 kubeadm upgrade apply v1.x.y

Note: Here V is preesent

Now, optionally one can verify the upgrade is successful or not by running same command again

kubeadm upgrade plan

22.2.4 Now upgrade kubelet version

apt-get install kubelet=1.22.1-00

Note: No V here

22.2.5 Now optionally restart kubelet

systemctl daemon-reload && systemctl restart kubelet

22.2.6 exit from master node

22.3 k uncorden master

22.4 Now log in to worker

Here instead of

22.2.2 kubeadm upgrade plan

22.2.3 kubeadm upgrade apply v1.x.y

We have to use

22.4.1 kubeadm upgrade node

now run same as 22.2.4 to 22.2.6 and update kubelet

22.5 same as 22.3 now uncordon worker

k uncordon worker

Also refer: http://layers7.blogspot.com/2020/08/cka-6-cluster-maintenance.html

Basically

Master

cordon

Install new kuebadm, kubelet

kubeadm upgrade plan

kubeadm upgrade apply "newVersion"

uncordon

Worker

Same steps, except 2 commands for kubeadm, here only one command

kubeadm upgrade node

23. We should never echo ENV, which is loaded from secret

24. Never put password in plain text in YAML file

25. in DockerFile, if you copy a secret token one command using COPY or ADD. and in second line if you use RUN command and remove using "rm" command, then also it remains in previous layer. Avoid it.

26. See various command start with

kubectl config -h

27. To get pod from container ID

27.1 crictl ps --id "container ID"

It will give pod id

27.2 crictl pods --id "pod id"

Note: It is pods , not pod

28. Annotation for apparmor profile.

- for pod it is : metadata.annotations

- for deployment it is: spec.template.metadata.annotations

We can annotate pod using kubectl imperative command. Not possible for deployment

k annotate pod "pod name" k1=v1 k2=v2 k3=v3

29. In network policy. podSelector applies to target pod and also in rule. namespaceSelector is only for rule. Both selector needs matchLabels. We cannot provide list of pods, by mentioning name of the pods. We can use matchExressions. However matchEpression also works on label k.v pair only. No other attributes like name, image name, container name etc.

30. Use command c$ to replace from cursor till end of file. VI editor short cut. useful while editing YAML file copied from k8s docs and you wish to change some values. cw is used to change single word. If existing value has - or / then c$ is more useful.

31. For K8s audit, specify log file path, which already exist. E.g. /var/log/audit.log

32. for k8s audit policy

We need to mention group of resource. Do not include version. To find API group, run the command

k api-resources

33 Specify image

33.1 k run for creating pod

kubectl run nginx --image=nginx -n mynamespace

33.2 k create to create deployment, job, cronjob

kubectl create deployment nginx --image=nginx

33.3 k set image, to upgrade. Only here we use container name. "www" in below example

kubectl set image deployment/frontend www=image:v2

34. CIS

34.1 report's sections

You can get by runnning command

grep "INFO" "report file name"

====================================

1. Control plane

1.1 All config files /* all file/folder permission and ownership */

1.2 API server

1.3 Controller-Manager

1.4 Scheduler

2. etcd

3. Control plane config

4. Worker node

4.1 worker node config files /* all file/folder permission and ownership */

4.2 kubelet

5. Policies

5.1 RBAC and SA

5.2 PSP

5.3 netpol and CNI

5.4 Secret mgmt

5.5 extensible admission control

5.7 General

====================================

34.2 section wise failure

You can get by runnning command

grep "check FAIL" "report file name"

35. To remove duplicate

cat "file name" | uniq

Note: It is uniq, not unique

36. All the Kind always start with capital letter. Useful while searching audit log

37. RBAC

37.1 Note:

system:serviceaccount: (singular) is the prefix for service account usernames.
system:serviceaccounts: (plural) is the prefix for service account groups.

37.2 Grant a role to all service accounts in a namespace

kubectl create rolebinding serviceaccounts-view \

--clusterrole=view \

--group=system:serviceaccounts:my-namespace \

--namespace=my-namespace

37.3 Grant a limited role to all service accounts cluster-wide (discouraged)

kubectl create clusterrolebinding serviceaccounts-view \

  --clusterrole=view \

 --group=system:serviceaccounts

39 Important paths

39.1. SA token inside pod

/var/run/secrets/kubernetes.io/serviceaccount/token

39.2. Kube-api manifest file at master node.

/etc/kubernetes/manifest/kube-apiserver.yaml

39.3. seccomp path at worker | master node

/var/lib/kubelet/seccomp/profiles/name.json

In YAML file of pod/deployment we mention

localhostProfile: profiles/name.json

39.4 All apparmor profile file (not profile) are located at worker node

/etc/apparmod.d/

39.5 For runtime class configuration

39.5.1 with containerd

/etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.${HANDLER_NAME}]

39.5.2 with crio

/etc/crio/crio.conf

=============

38. module_request in SELinux

nice to read article: https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/

K8s. Flow

Ref: https://github.com/cncf/financial-user-group/tree/master/projects/k8s-threat-model

Security Context and PSP

Security Context

Only for container

allowPrivilegeEscalation: false . It is for setuid and setgid

allowPrivilegeEscalation:

if (CAP_SYS_ADMIN || Privileged mode) then AllowPrivilegeEscalation = SSeue

=============

securityContext:
capabilities:
drop:
- all
add: ["MKNOD"]

in PSP we have

allowedCapabilities:

- '*'

requiredDropCapabilities:

- ALL

=============

privileged: true avoid it.

=============

procMount: true is for nested container useful for in-cluster build process

=============

readOnlyRootFilesystem: true for immutable container

=============

For container and pod both

container settings will get precedence.

=============

runAsUser and runAsGroup are runtime configuration. If not defined then UID as per data in image

=============

If runAsNonRoot = True then image should have UID. Same UID should be in host.

=============

securityContext:

seLinuxOptions:
level: "s0:c123,c456"

we can specify level, role, type, user. They are labels for file, process and ports. They are collectively called called "context" in SELinux terms.

=============

type = Localhost | Unconfined | RuntimeDefault

seccompProfile:
type: Localhost
localhostProfile: profiles/name.json

The actual path of profile: <kubelet-root-dir>/seccomp/profiles/name.json

where kuelet-root-dir = /var/lib/kubelet .

It is configured with seccomp-profile-root flag for kubelet. This flag is deprecited since 1.19 . If seccomp-profile-root flag is not defined, the default path will be used, which is <root-dir>/seccomp where <root-dir> is specified by the --root-dir flag.

* type: Unconfined means no seccomp profile is applied

* type: RuntimeDefault means same seccomp profile as container runtime default is applied

* type: Localhost, then only, we need to set localhostProfile

We need to use annotation (optional)

seccomp.security.alpha.kubernetes.io/pod: localhost/profiles/audit.json

Path for seccomp profiles: https://k8s.io/examples/pods/security/seccomp/profiles/

=============

Only for Pod

fsGroup

the permission bit will be | with rw-rw----

ownership and permission change recursively for all content in mounted volume as per fsGroup

=============

fsGroupChangePolicy = OnRootMismatch | Always

no impact on emptyDir, secret and configMap

OnRootMismatch: It will save time. The permission and ownership only change if root level folder has mismatch with expected fsGroup.

If DelegateFSGroupToCSIDriver feature gate is enabled then this is done by CSI driver. CSI driver will not respect fsGroupChangePolicy

=============

supplementalGroups

A list of groups applied to the first process run in each container, in addition to the container's primary GID.

=============

sysctls

====================================================

Here Discretionary Access Control (DAC) is related fields are: runAsUser, runAsGroup, runAsNonRoot, readOnlyRootFilesystem

Here volume related fields are: fsGroup and seLinuxOptions. We set only level at seLinuxOptions and as per level, labels applies to all containers and volumes.

PodSecurityPolicy

Pod's scrutiny attributes. Defined at cluster level. It controls security sensitive aspects of the pod specification. It define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields.

It (1) restrict pod creation (2) restrict pod update (3) provide default value

=============

- "privileged" container (Privileged Mode). It is part of PSP. But it control container level "security context"

=============

- host-level ns (network, PID, IPC) "hostPID", "hostIPC", "hostNetwork". If hostPID allowed then container can escalate privilege using ptrace system call.

=============

- host ports "hostPorts"

hostPorts:
- min: 0
max: 65535

=============

- different types of volumes. E.g. "allowedFlexVolumes" "volumes"

=============

- host's filesystem E.g. "fsGroup"

fsGroup = MustRunAs | MayRunAs | RunAsAny

We shall specify 1+ range if fsGroup = MustRunAs | MayRunAs. In case of MustRunAs the fsGroup at Pod Security Context is set as min value. in case of MayRunAs, the default value for fsGroup at Pod Security Context, is unset

=============

- RO root filesystem for containers (DAC) "readOnlyRootFilesystem"

=============

- user IDs and group IDs (DAC) "runAsUser" "runAsGroup" "supplementalGroups"

runAsUser = MustRunAs | MustRunAsNonRoot | RunAsAny

runAsGroup = MustRunAs | MayRunAs | RunAsAny

supplementalGroups = MustRunAs | MayRunAs | RunAsAny

=============

- containers' privilege escalation "allowPrivilegeEscalation" "defaultAllowPrivilegeEscalation". Here defaultAllowPrivilegeEscalation, set values for allowPrivilegeEscalation, if not set.

=============

- containers' Linux capabilities (Linux Capabilities): "defaultAddCapabilities", "requiredDropCapabilities", "allowedCapabilities"

=============

- SELinux "seLinux"

RunAsAny means: Apparmor is used instead of SELinux.

=============

- seccomp and AppArmor profiles : using annotations.

seccomp.security.alpha.kubernetes.io/allowedProfileNames

apparmor.security.beta.kubernetes.io/defaultProfileName

If this annotation is not specified then default seccomp cannot be changed

=============

- sysctls that a pod can run: "forbiddenSysctls", "allowedUnsafeSysctls"

=============

- a proc mount type to be used. "allowedProcMountTypes" and "DefaultProcMount"

=============

- "AllowedHostPaths" : As name suggest

allowedHostPaths:
# This allows "/foo", "/foo/", "/foo/bar" etc., but
# disallows "/fool", "/etc/foo" etc.
# "/foo/../" is never valid.
- pathPrefix: "/foo"
readOnly: true # only allow read-only mounts

=============

PSP Policy Order

1. non-mutating policy in any order

2. mutating policy in alphabetical order of name

Good Reference:

A Good article: https://www.linux.com/training-tutorials/overview-linux-kernel-security-features/

Abbreviation

ASLR Address Space Layout Randomization

DAC Discretionary Access Control

IMA Integrity Measurement Architecture

MAC Mandatory Access Control

PAM Pluggable Authentication Modules

7. Workload Considerations : Falco

Falco by Sysdig: multiple components (user space program, configuration, driver) working together in order to evaluate system calls against rules, and generate alerts when a rule is broken:

rule has lists. rule can have reference to list. List can be part of macro and other list, in addition to part of rule.

rule has 5 k-v pairs. (1) name, (2) description , (3) condition : Filtering expression for events. (4) output, (5) priority. (emergency, alert, critical, error, warning, notice, informational, debug)

rule has 4 optional K-v pairs.(1) enabled. default is true (2) tags (filesystem, software_mgmt, process, database, host, shell, container, cis, users, network) . -T option to disable rules with given tag. -t option to enable. (3) warn_evttypes default is true. (4) skip-if-unknown-filter default is false. 5th one added (5) exceptions : a set of conditions that cause the rule to not generate an alert.

- Falco comes with many rules in /etc/falco/falco_rules.yaml file. They can be overwritten by /etc/falco/falco_rules.local.yaml file. E.g. to disable rule : We can add rule with same name and "append: true" + "enabled: false"

- evt.dir = < indicates end of system call and evt.dir = > indicates beginning of system call. dir = direction

- We have K8s related context: k8s.[pod | rc | svc | rs | deployment].[name | id \ label | labels] + many fields from K8s audit logs.

with macro part of rule can be re-used. There are many default macros.

Falco runs with K8s-audit on. So we need to specify audit policy file at API server argument --audit-policy-file

We can configure webhook in API server with this arguement

--audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig

This YAML file shall define Config kind.

We can see Falco output with journalctl command.

Reference :

https://github.com/falcosecurity/falco/blob/master/rules/k8s_audit_rules.yaml

https://v1-17.docs.kubernetes.io/docs/tasks/debug-application-cluster/falco/

Life is an exercise to express the InExpressible.

Express YourSelf !

minikube kubeconfig

CKS Tips

K8s. Flow

Security Context and PSP

7. Workload Considerations : Falco

Total Pageviews

Subscribe by E-mail

Labels

Popular Posts

Coming Soon...

Followers

My Social Network

Blog Archive