Pod Security Policy (PSP)

- A set of rules

- provide/modify default values for fields

- change pod

- PSP ordered by name before applied. 

- Deprecated in K8s 1.21

- will be removed in K8s 1.25

Even if you are only planning on changing a single value, the policy file must contain several entries. Sample PSP, where pod can do anything

apiVersion: policy/v1beta1

kind: PodSecurityPolicy
metadata:
   name: basicpolicy
spec:
privileged: true
runAsUser:
   rule: RunAsAny
seLinux:
  rule: RunAsAny
fsGroup:
  rule: RunAsAny
supplementalGroups:
  rule: RunAsAny
allowedCapabilities:
  - '*'
volumes:
  -'*'

Most commonly changed parameters

1. privileged

2. runAsUser

Reference: https://kubernetes.io/docs/concepts/policy/pod-security-policy/

For allowedUnsafeSysctls  and forbiddenSysctls 

• kernel (common prefix: kernel.)

• kernel.shm*,
• kernel.msg*,
• kernel.sem,
• networking (common prefix: net.)
• virtual memory (common prefix: vm.)
• MDADM (common prefix: dev.)

If we use RoleBinding instead of ClusterRoleBinding then it is for same namespace

The replicaset controller use default SA. So we should able to create deployment with about 2 commands also. 

If controller manager connects to API server using trusted/insecure port then all PSS allowed, as authorization (and authentication) is bypass. 

After enabling PodSecurityPolicy admission control plugin, we should have

1. This policy

2. We need clusterrole in target namespace

k -n team-red create clusterrole cr --verb=use --resource=psp

3. To add any new PSP, it should have min these fields

4. At each NS, we should have rolebidning.

k -n team-red create rolebinding rb --clusterrole=cr --user=system:serviceaccounts

OR

We can have clusterrolebinding

k -n team-red create clusterrolebinding crb --clusterrole=cr --user=system:serviceaccounts

References

https://banzaicloud.com/blog/pod-security-policy/

https://www.suse.com/c/rancher_blog/enhancing-kubernetes-security-with-pod-security-policies-part-2/

IAM using tools: keycloak , Active Directory, Amazon IAM

CIS It provides huge amount of free and paid resources to improve IT It provides security. tools, including benchmarks, scanning tools, threat tools, and hardened images. The CIS-CAT®Pro tool evaluates a target system against known issues and performance configurations. CIS also offers dashboards to view the ongoing state of compliance and security considerations.

For minikube setup we need to install kube-bench tool on individual node and run test. The test result recommend steps, for failure and warning cases. We can also run job.yaml at K8s cluster. 

Have a look to summary of CIS for K8s in this Excel file

Tips to protect etcd

- Only Kube-APIServer is allowed to communicate with etcd

- Over TLS

- configmaps and secrets must not be compromised, as they are injected to application pod. 

- Kube-APIServer must be updated to use encryption provider configuration 

Encryption Providers

1. identity : NULL encryption

2. aescbc : 

- Slow and strong

- Recommended 

- key length = 32 bits

- Algo: Public Key Cryptographic Standards (PKCS): Advanced Encryption Standard (AES) cipher algo in Cipher Block Chaining (CBC) mode. Refer RFC 3602

3. secretbox

- faster

- key length = 32 bits

Algo: XSalsa20 and Poly1305 encryption

4. aesgcm

- fastest

- key length = 16 bots or 32 bits

- key rotation every 200,000 writes

5. kms

- uses new Data Encryption Key (DEK) for every encryption along with AES-CBC

Sample Encryption Policy File

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
name: configureKeys
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: encKey
          secret: Ezqai0SIGChGOwf0VdbkFtYPUM2EYf1TAAQbDrfizJQ=
    - identity: {}

Assume the above YAML file is located at /var/lib/minikube/certs/ec.yaml then we shall mount this hostPath with same folder and specify to API Server with argument :

--encryption-provider-config=/var/lib/minikube/certs/ec.yaml

- One can encrypt many K8s objects instead of just secrets.

- new key should be at top of conifg file to encrypt. next line can be old key, to decrypt previously encrypted parameters. 

- Encryption key should be in separate KMS (key management system)

 Audit-log

* --audit-log-maxage Recommended: Minimum 30 (days)

* --audit-log-maxbackup Recommended: Minimum 10 (files)

* --audit-log-maxsize Recommended: 100 (MB) or appropriate

* --audit-policy-file=/etc/kubernetes/simple-policy.yaml

This declares where, inside the container, the policy file is located. You will also need to add a volume and mount point to the container.

* --audit-log-path Example value: /var/log/apiserver/audit.log

This declares where the logs should be kept inside the container. This will also need a volume and a mount point configured.

Audit-process

If audit is enabled then each API goes through audit process, even if it is filtered out. 

3 Phases

1. Call is received: RequestReceived

2. Handling the call: ResponseStarted (Only applicable for long running request like watch)

3. Response call is made: ResponseComplete

* Panic 

Each even has audit rule. Audit rule set audit level

- None

- Metadata

- Request

- RequestResponse

Sample policy files

Note : all resources are always in plural in all policy file. 

1.

2. monitor events for specific pods

3. Monitor all request and response from kubectl on minikube setup

The first matching rule sets the audit level of the event. The omitStages can be per rule or not per rule.

Audit backend

1. Log

2. Webhook

Step 1, create file  /etc/kubernetes/audit-webhook-kubeconfig on master node

phase of each API at Kube-APIServer

1. Authentication

2. Authorization

3. Admission Control

By default, kubeadm-built clusters allow API access without authentication and authorization.

* IP = 127.0.0.1 TLS = No Port = 8080. First two phases are skipped

* IP = First n/w interface IP TLS = Yes Port = 6443. All three phases are executed. 

Several arguments are passed to the kube-apiserver process via /etc/kubernetes/manifests/kube-apiserver.yaml

Sample file content

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.128.15.193
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true

Few parameters worth to highlight

* --advertise-address: Used by entire cluster. If absent, then use --bind-address. If that is also absent then use IP address of first network interface

* --allow-privileged : Privileged containers are isolated or not. If absent then value is false. kubeadm set it true. Change application, after modifying the flag. 

* --enable-bootstrap-token-auth : If true, bootstrapping tokens can be used as bearer token credentials. It allows requests to authenticate against the API server.

* --etcd-servers=https://127.0.0.1:2379

* --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

If Kube-APIServer need to communicate with Aggregated API (AA) Server:

* --proxy-client-cert-file

* --proxy-client-key-file

How Kube-APIServer and kube-proxy communicate with AA server.

* --requestheader-allowed-names=front-proxy-client

* --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt

* --requestheader-extra-headers-prefix=X-Remote-Extra-

* --requestheader-group-headers=X-Remote-Group

* --requestheader-username-headers=X-Remote-User

Many other parameters are captured at https://layers7.blogspot.com/2021/01/k8s-api-server-security-settings.html

Cluster Version

- Many open source software. Wanted and unwanted

- Each has may have its own security issue

- requires lots of documents and testing

- well defined roles and responsibilities : who will test, document and fix. 

K8s update means : existing YAML files and configuration become inoperable. 

Protect the kernel

Steps

1. choose OS

2. Choose kernel (1) customized (2) generic 

3. Advanced Pluggable Authentication Module PAM to be used or not.

4. Least privilege concept

- users added with no special access and add as and when needed

- install required software only. 

5. Use "Linux Exploit Suggester" script to compare the kernel version running on a system with a public kernel vulnerability database (CVE). 

* If kernel version < 3.2.1 then IGMP can cause DoS. Ref: CVE-2012-0207

Mitigating Kernel Vulnerabilities

1. Block Loading of Dynamic modules

Originally, a rootkit was a collection of tools that enabled administrator-level access to a computer or network. Now they are associated with malware. rootkit removal is practically impossible. If it resides in kernel then OS need to be reinstalled. If it resides in firmware then hardware change or special method. 

Solution is block loading of dynamic modules.

2. kernel.modules_disabled = 1 with root or CAP_SYS_MODULE capability. No module related operations are allowed till next reboot. 

3. Adaptive stack layout randomization. 

Software based hardening

 kernel.randomize_va_space sysctl

0: Disable

1: randomize stack, VDSO, shared memory address

2: randomize stack, VDSO, shared memory and data address

4. Hardware Security Features

NX (No eXecute): heap, stack etc. does not contain executable code. nx flag in /proc/cpuinfo It is hardware based, so no performance impact. 

VT-d (virtualization): VM can directly access h/w without support of OS/hypervisor. vmx or svm flag in /proc/cpuinfo. 

TPM (Trusted Platform Module): store checksum for boot volume, biometric data. later on used for authentication and comparison. TPM uses TSS (Trusted software stack) based on TrouSerS implementation https://sourceforge.net/projects/trousers/ Ref: https://sourceforge.net/projects/tpmdd/

TXT (Trusted Execution Technology): Isolate memory used by guest OS. It is coupled with TPM. 

SMAF (Secure Memory Allocation Feature), along with various support for hardware cryptographic devices.

5. ExecShield

protection against: stack overflow, buffer overflow, function pointer overflow. It is hardware based so more overhead.  

echo 0 > /proc/sys/kernel/exec-shield

6. Integrity Management

The Integrity Measurement Architecture (IMA) component performs runtime integrity measurements of files using cryptographic hashes. It can rely on TPM, if present. Look for CONFIG_IMA in /boot/config. Boot with kernel parameters ima_tcb and ima=on 

7. Integrity Management with dm-verity

Manages file integrity at block level. Intended to be part of verified boot process. Relies on kernel crypto API to detect device tempering. Check CONFIG_DM_VERITY at /boot/config

8. Linux Security Modules (LSM)

LSM API implements hooks at all security critical points in kernel. List of LSMs:

* AppArmor

* SELinux

* Smack (Simplified Mandatory Access Control Kernel) : Simpler than SELinux. It’s also implemented as a label-based scheme with a customizable policy. 

* TOMOYO: intended for end users; not system administrators. Another MAC scheme which implements path-based security rather than object labeling. Similar to AppArmor. TOMOYO records tree of process invocation, known as domain. 

* yama: It collected miscellaneous DAC security enhancements . 

* bpf

* audit: was designed to meet government certification requirements. 

* lockdown etc. 

9. Seccomp (Secure Computing Module)

Restricts access to system calls by process. So attack surface of kernel is reduced. 

mode1 provides access to: read, write, exit, sigreturn

mode2 used in Google Chrome OS