Express YourSelf !

Tips to protect etcd

- Only Kube-APIServer is allowed to communicate with etcd

- Over TLS

- configmaps and secrets must not be compromised, as they are injected to application pod. 

- Kube-APIServer must be updated to use encryption provider configuration 

Encryption Providers

1. identity : NULL encryption

2. aescbc : 

- Slow and strong

- Recommended 

- key length = 32 bits

- Algo: Public Key Cryptographic Standards (PKCS): Advanced Encryption Standard (AES) cipher algo in Cipher Block Chaining (CBC) mode. Refer RFC 3602

3. secretbox

- faster

- key length = 32 bits

Algo: XSalsa20 and Poly1305 encryption

4. aesgcm

- fastest

- key length = 16 bots or 32 bits

- key rotation every 200,000 writes

5. kms

- uses new Data Encryption Key (DEK) for every encryption along with AES-CBC

Sample Encryption Policy File

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
name: configureKeys
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: encKey
          secret: Ezqai0SIGChGOwf0VdbkFtYPUM2EYf1TAAQbDrfizJQ=
    - identity: {}

Assume the above YAML file is located at /var/lib/minikube/certs/ec.yaml then we shall mount this hostPath with same folder and specify to API Server with argument :

--encryption-provider-config=/var/lib/minikube/certs/ec.yaml

- One can encrypt many K8s objects instead of just secrets.

- new key should be at top of conifg file to encrypt. next line can be old key, to decrypt previously encrypted parameters. 

- Encryption key should be in separate KMS (key management system)

 Audit-log

* --audit-log-maxage Recommended: Minimum 30 (days)

* --audit-log-maxbackup Recommended: Minimum 10 (files)

* --audit-log-maxsize Recommended: 100 (MB) or appropriate

* --audit-policy-file=/etc/kubernetes/simple-policy.yaml

This declares where, inside the container, the policy file is located. You will also need to add a volume and mount point to the container.

* --audit-log-path Example value: /var/log/apiserver/audit.log

This declares where the logs should be kept inside the container. This will also need a volume and a mount point configured.

Audit-process

If audit is enabled then each API goes through audit process, even if it is filtered out. 

3 Phases

1. Call is received: RequestReceived

2. Handling the call: ResponseStarted (Only applicable for long running request like watch)

3. Response call is made: ResponseComplete

* Panic 

Each even has audit rule. Audit rule set audit level

- None

- Metadata

- Request

- RequestResponse

Sample policy files

Note : all resources are always in plural in all policy file. 

1.

2. monitor events for specific pods

3. Monitor all request and response from kubectl on minikube setup

The first matching rule sets the audit level of the event. The omitStages can be per rule or not per rule.

Audit backend

1. Log

2. Webhook

Step 1, create file  /etc/kubernetes/audit-webhook-kubeconfig on master node

phase of each API at Kube-APIServer

1. Authentication

2. Authorization

3. Admission Control

By default, kubeadm-built clusters allow API access without authentication and authorization.

* IP = 127.0.0.1 TLS = No Port = 8080. First two phases are skipped

* IP = First n/w interface IP TLS = Yes Port = 6443. All three phases are executed. 

Several arguments are passed to the kube-apiserver process via /etc/kubernetes/manifests/kube-apiserver.yaml

Sample file content

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.128.15.193
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true

Few parameters worth to highlight

* --advertise-address: Used by entire cluster. If absent, then use --bind-address. If that is also absent then use IP address of first network interface

* --allow-privileged : Privileged containers are isolated or not. If absent then value is false. kubeadm set it true. Change application, after modifying the flag. 

* --enable-bootstrap-token-auth : If true, bootstrapping tokens can be used as bearer token credentials. It allows requests to authenticate against the API server.

* --etcd-servers=https://127.0.0.1:2379

* --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

If Kube-APIServer need to communicate with Aggregated API (AA) Server:

* --proxy-client-cert-file

* --proxy-client-key-file

How Kube-APIServer and kube-proxy communicate with AA server.

* --requestheader-allowed-names=front-proxy-client

* --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt

* --requestheader-extra-headers-prefix=X-Remote-Extra-

* --requestheader-group-headers=X-Remote-Group

* --requestheader-username-headers=X-Remote-User

Many other parameters are captured at https://layers7.blogspot.com/2021/01/k8s-api-server-security-settings.html

Cluster Version

- Many open source software. Wanted and unwanted

- Each has may have its own security issue

- requires lots of documents and testing

- well defined roles and responsibilities : who will test, document and fix. 

K8s update means : existing YAML files and configuration become inoperable. 

Protect the kernel

Steps

1. choose OS

2. Choose kernel (1) customized (2) generic 

3. Advanced Pluggable Authentication Module PAM to be used or not.

4. Least privilege concept

- users added with no special access and add as and when needed

- install required software only. 

5. Use "Linux Exploit Suggester" script to compare the kernel version running on a system with a public kernel vulnerability database (CVE). 

* If kernel version < 3.2.1 then IGMP can cause DoS. Ref: CVE-2012-0207

Mitigating Kernel Vulnerabilities

1. Block Loading of Dynamic modules

Originally, a rootkit was a collection of tools that enabled administrator-level access to a computer or network. Now they are associated with malware. rootkit removal is practically impossible. If it resides in kernel then OS need to be reinstalled. If it resides in firmware then hardware change or special method. 

Solution is block loading of dynamic modules.

2. kernel.modules_disabled = 1 with root or CAP_SYS_MODULE capability. No module related operations are allowed till next reboot. 

3. Adaptive stack layout randomization. 

Software based hardening

 kernel.randomize_va_space sysctl

0: Disable

1: randomize stack, VDSO, shared memory address

2: randomize stack, VDSO, shared memory and data address

4. Hardware Security Features

NX (No eXecute): heap, stack etc. does not contain executable code. nx flag in /proc/cpuinfo It is hardware based, so no performance impact. 

VT-d (virtualization): VM can directly access h/w without support of OS/hypervisor. vmx or svm flag in /proc/cpuinfo. 

TPM (Trusted Platform Module): store checksum for boot volume, biometric data. later on used for authentication and comparison. TPM uses TSS (Trusted software stack) based on TrouSerS implementation https://sourceforge.net/projects/trousers/ Ref: https://sourceforge.net/projects/tpmdd/

TXT (Trusted Execution Technology): Isolate memory used by guest OS. It is coupled with TPM. 

SMAF (Secure Memory Allocation Feature), along with various support for hardware cryptographic devices.

5. ExecShield

protection against: stack overflow, buffer overflow, function pointer overflow. It is hardware based so more overhead.  

echo 0 > /proc/sys/kernel/exec-shield

6. Integrity Management

The Integrity Measurement Architecture (IMA) component performs runtime integrity measurements of files using cryptographic hashes. It can rely on TPM, if present. Look for CONFIG_IMA in /boot/config. Boot with kernel parameters ima_tcb and ima=on 

7. Integrity Management with dm-verity

Manages file integrity at block level. Intended to be part of verified boot process. Relies on kernel crypto API to detect device tempering. Check CONFIG_DM_VERITY at /boot/config

8. Linux Security Modules (LSM)

LSM API implements hooks at all security critical points in kernel. List of LSMs:

* AppArmor

* SELinux

* Smack (Simplified Mandatory Access Control Kernel) : Simpler than SELinux. It’s also implemented as a label-based scheme with a customizable policy. 

* TOMOYO: intended for end users; not system administrators. Another MAC scheme which implements path-based security rather than object labeling. Similar to AppArmor. TOMOYO records tree of process invocation, known as domain. 

* yama: It collected miscellaneous DAC security enhancements . 

* bpf

* audit: was designed to meet government certification requirements. 

* lockdown etc. 

9. Seccomp (Secure Computing Module)

Restricts access to system calls by process. So attack surface of kernel is reduced. 

mode1 provides access to: read, write, exit, sigreturn

mode2 used in Google Chrome OS

TUF & Notary

OPA & Gatekeeper

Gatekeeper project uses Admission Controller and "OPA constraint framework" (CRD). It enables users to customize admission control by configuration not by code. It enforces policies executed by OPA

Gatekeeper is extensible, parameterized policy library. constraint CRD instantiate the policy library. constraint template CRD extend the policy library. 

- Each Constraint is written with Rego, a declarative query language used by OPA. All Constraints are evaluated as a logical AND. If one Constraint is not satisfied, then the whole request is rejected.

- Each  constraint template describes both (1) the Rego logic that enforces the Constraint and  (2) the schema for the Constraint, which includes (2.1) the schema of the CRD and (2.2) the parameters that can be passed into a Constraint, much like arguments to a function.

- No namespace for constraint template. No namespace for constraint. 

The audit functionality enables periodic evaluations of replicated resources. It evaluates against the Constraints. So it detects pre-existing misconfigurations. Gatekeeper stores audit results as violations listed in the status field of the relevant Constraint. 

k get constraints "name of constraint" -o yaml

check status field here

- In k8s, OPA replaces PSP. OPA is also used for Envoy Proxy, SSH and sudo

- Gatekeeper is validation webhook + audit + CRD based policy and OPA is policy engine. 

- failurePolicy: Ignore means if webhook is down, then constraint will not be apply. Later on, audit shall determine such events. 

Use cases

• Validating
• Require specific labels on all resources. E.g. All namespaces must have a label that lists a point-of-contact
• Require container images come from the corporate image registry. (here, no error shown for deployment creation, not at at pod, not at rs. number of pod=0. when you describe deployment and rs, you will see error in rs creation. No, you can also see error at status of constraint. It is only for existing pods)
• Require all Pods specify resource requests and limits.
• Prevent conflicting Ingress objects from being created. E.g. All ingress hostnames must be globally unique
• No run as root
• No privileged container
• Mutating
• Inject sidecar containers into Pods.
• Set specific annotations on all resources.
• Rewrite container images to point at the corporate image registry.
• Include node and pod (anti-)affinity selectors on Deployments.