Express YourSelf !

phase of each API at Kube-APIServer

1. Authentication

2. Authorization

3. Admission Control

By default, kubeadm-built clusters allow API access without authentication and authorization.

* IP = 127.0.0.1 TLS = No Port = 8080. First two phases are skipped

* IP = First n/w interface IP TLS = Yes Port = 6443. All three phases are executed. 

Several arguments are passed to the kube-apiserver process via /etc/kubernetes/manifests/kube-apiserver.yaml

Sample file content

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.128.15.193
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true

Few parameters worth to highlight

* --advertise-address: Used by entire cluster. If absent, then use --bind-address. If that is also absent then use IP address of first network interface

* --allow-privileged : Privileged containers are isolated or not. If absent then value is false. kubeadm set it true. Change application, after modifying the flag. 

* --enable-bootstrap-token-auth : If true, bootstrapping tokens can be used as bearer token credentials. It allows requests to authenticate against the API server.

* --etcd-servers=https://127.0.0.1:2379

* --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

If Kube-APIServer need to communicate with Aggregated API (AA) Server:

* --proxy-client-cert-file

* --proxy-client-key-file

How Kube-APIServer and kube-proxy communicate with AA server.

* --requestheader-allowed-names=front-proxy-client

* --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt

* --requestheader-extra-headers-prefix=X-Remote-Extra-

* --requestheader-group-headers=X-Remote-Group

* --requestheader-username-headers=X-Remote-User

Many other parameters are captured at https://layers7.blogspot.com/2021/01/k8s-api-server-security-settings.html

Cluster Version

- Many open source software. Wanted and unwanted

- Each has may have its own security issue

- requires lots of documents and testing

- well defined roles and responsibilities : who will test, document and fix. 

K8s update means : existing YAML files and configuration become inoperable. 

Protect the kernel

Steps

1. choose OS

2. Choose kernel (1) customized (2) generic 

3. Advanced Pluggable Authentication Module PAM to be used or not.

4. Least privilege concept

- users added with no special access and add as and when needed

- install required software only. 

5. Use "Linux Exploit Suggester" script to compare the kernel version running on a system with a public kernel vulnerability database (CVE). 

* If kernel version < 3.2.1 then IGMP can cause DoS. Ref: CVE-2012-0207

Mitigating Kernel Vulnerabilities

1. Block Loading of Dynamic modules

Originally, a rootkit was a collection of tools that enabled administrator-level access to a computer or network. Now they are associated with malware. rootkit removal is practically impossible. If it resides in kernel then OS need to be reinstalled. If it resides in firmware then hardware change or special method. 

Solution is block loading of dynamic modules.

2. kernel.modules_disabled = 1 with root or CAP_SYS_MODULE capability. No module related operations are allowed till next reboot. 

3. Adaptive stack layout randomization. 

Software based hardening

 kernel.randomize_va_space sysctl

0: Disable

1: randomize stack, VDSO, shared memory address

2: randomize stack, VDSO, shared memory and data address

4. Hardware Security Features

NX (No eXecute): heap, stack etc. does not contain executable code. nx flag in /proc/cpuinfo It is hardware based, so no performance impact. 

VT-d (virtualization): VM can directly access h/w without support of OS/hypervisor. vmx or svm flag in /proc/cpuinfo. 

TPM (Trusted Platform Module): store checksum for boot volume, biometric data. later on used for authentication and comparison. TPM uses TSS (Trusted software stack) based on TrouSerS implementation https://sourceforge.net/projects/trousers/ Ref: https://sourceforge.net/projects/tpmdd/

TXT (Trusted Execution Technology): Isolate memory used by guest OS. It is coupled with TPM. 

SMAF (Secure Memory Allocation Feature), along with various support for hardware cryptographic devices.

5. ExecShield

protection against: stack overflow, buffer overflow, function pointer overflow. It is hardware based so more overhead.  

echo 0 > /proc/sys/kernel/exec-shield

6. Integrity Management

The Integrity Measurement Architecture (IMA) component performs runtime integrity measurements of files using cryptographic hashes. It can rely on TPM, if present. Look for CONFIG_IMA in /boot/config. Boot with kernel parameters ima_tcb and ima=on 

7. Integrity Management with dm-verity

Manages file integrity at block level. Intended to be part of verified boot process. Relies on kernel crypto API to detect device tempering. Check CONFIG_DM_VERITY at /boot/config

8. Linux Security Modules (LSM)

LSM API implements hooks at all security critical points in kernel. List of LSMs:

* AppArmor

* SELinux

* Smack (Simplified Mandatory Access Control Kernel) : Simpler than SELinux. It’s also implemented as a label-based scheme with a customizable policy. 

* TOMOYO: intended for end users; not system administrators. Another MAC scheme which implements path-based security rather than object labeling. Similar to AppArmor. TOMOYO records tree of process invocation, known as domain. 

* yama: It collected miscellaneous DAC security enhancements . 

* bpf

* audit: was designed to meet government certification requirements. 

* lockdown etc. 

9. Seccomp (Secure Computing Module)

Restricts access to system calls by process. So attack surface of kernel is reduced. 

mode1 provides access to: read, write, exit, sigreturn

mode2 used in Google Chrome OS

TUF & Notary

OPA & Gatekeeper

Gatekeeper project uses Admission Controller and "OPA constraint framework" (CRD). It enables users to customize admission control by configuration not by code. It enforces policies executed by OPA

Gatekeeper is extensible, parameterized policy library. constraint CRD instantiate the policy library. constraint template CRD extend the policy library. 

- Each Constraint is written with Rego, a declarative query language used by OPA. All Constraints are evaluated as a logical AND. If one Constraint is not satisfied, then the whole request is rejected.

- Each  constraint template describes both (1) the Rego logic that enforces the Constraint and  (2) the schema for the Constraint, which includes (2.1) the schema of the CRD and (2.2) the parameters that can be passed into a Constraint, much like arguments to a function.

- No namespace for constraint template. No namespace for constraint. 

The audit functionality enables periodic evaluations of replicated resources. It evaluates against the Constraints. So it detects pre-existing misconfigurations. Gatekeeper stores audit results as violations listed in the status field of the relevant Constraint. 

k get constraints "name of constraint" -o yaml

check status field here

- In k8s, OPA replaces PSP. OPA is also used for Envoy Proxy, SSH and sudo

- Gatekeeper is validation webhook + audit + CRD based policy and OPA is policy engine. 

- failurePolicy: Ignore means if webhook is down, then constraint will not be apply. Later on, audit shall determine such events. 

Use cases

• Validating
• Require specific labels on all resources. E.g. All namespaces must have a label that lists a point-of-contact
• Require container images come from the corporate image registry. (here, no error shown for deployment creation, not at at pod, not at rs. number of pod=0. when you describe deployment and rs, you will see error in rs creation. No, you can also see error at status of constraint. It is only for existing pods)
• Require all Pods specify resource requests and limits.
• Prevent conflicting Ingress objects from being created. E.g. All ingress hostnames must be globally unique
• No run as root
• No privileged container
• Mutating
• Inject sidecar containers into Pods.
• Set specific annotations on all resources.
• Rewrite container images to point at the corporate image registry.
• Include node and pod (anti-)affinity selectors on Deployments.

Traditional container shares host kernel

gVisor has application kernel Sentry to execute system calls, securely.

kata has hardware virtualization to provide kernel per container.

gVisor https://github.com/google/gvisor

- Application kernel (Sentry). It includes Netstack (user space network stack)

- implements OCI runtime runsc

- isolation between app and host kernel (Gofer for access to filesystem) with seccomp

- Each pod is gVisor instance and container inside pod is process. 

- Often used with SELinux or seccomp

- gVisor uses runsc binary instead of runc

- Sentry and Gofer communicates using 9P protocol (common n/w protocol by Bell Labs)

- more overhead to execute system calls using gVisor

Follow steps as per https://sbulav.github.io/certifications/cks-gvisor/ at all worker node. 

We can see on worker node runsc-gopher and runsc-sandbox (santry) processes are running. 

Kata https://katacontainers.io/

Speed of containers and security of VMs

Lightweight VM, plug into container eco system

hypervisors: QEMU, NEMU, Firecracker 

Works with Docker and cri-o architecture. cri-o and containerd engine.

Intel's clear container is now part of kata container.

Kata containers are part of OSF Open Stack Foundation. 

PouchContainer http://pouchcontainer.io/

Open source container engine by Alibaba.

Strong isolation

less runtime overhead

incremental image distribution

cross platform and cross OS

compatible with OCI spec

* Rich container: Can have multiple process + init process. Package fat application easily. 

* Strong isolation: Can create VM

* Compatible with many kernel versions

* P2P image distribution 

Use with K8s: https://github.com/alibaba/pouch/blob/master/docs/kubernetes/pouch_with_kubernetes_deploying.md

Firecracker https://github.com/firecracker-microvm/firecracker-containerd

to manage Firecracker microVM using containerd

Unik https://github.com/solo-io/unik

- Tool to compile app source into unikernels and microVM. Not binaries. 

Unikernels are lightweight, immutable operating systems compiled specifically to run a single application. No ssh/telnet. Mostly no File System. single binary | process | address space | mode, no forked. no shell, so no "shell injection". Less code, so less bug so less likely buffer overflow. 

- Simple docker-like CLI

- Good for IoT smart devices

1. Go code to Firecracker microVM

2. Python, Node.js, Go to rump (rumprun) unikernel

3. Java, Node.js, C, C++ to OSv unikernel

4. C++ to IncludeOS unikernel

5. OCaml to MirageOS unikernel

Supported Providers: Firecracker, Virtualbox, AWS, Google Cloud, vSphere, QEMU, UKVM, Xen, OpenStack, Photon Controller   

=========

RuntimeClass

Enable at (1) Kube-apiserver and (2) kubelet

High level container runtime containerd support it. (1) cri-o and (2) docker are yet to support custom runtime configuration. 

apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc //CRI Configuration 

=======

apiVersion: v1

kind: Pod

metadata:

  name: mypod

spec:

  runtimeClassName: gvisor

CRI Configuration 

* dockershim : Default. Does not support runtime handler 

* containerd: [plugins.cri.containerd.runtimes.${HANDLER_NAME}] section at file /etc/containerd/config.toml

Reference: https://github.com/containerd/cri/blob/master/docs/config.md

* cri-o: 

[crio.runtime.runtimes.${HANDLER_NAME}]

  runtime_path = "${PATH_TO_BINARY}"

section at file /etc/crio/crio.conf

Reference: https://github.com/cri-o/cri-o/blob/master/docs/crio.conf.5.md#crioruntime-table

Trow: Registry and image management

Prisma Cloud: Cloud Security Platform

NeuVector: scan container images for vulnerabilities. DPI, Firewall

Clair: static analysis for container image vulnerabilities. Open Source Project

Aqua: Complete enterprise security solution for container LCM

Trivy: scan container images for vulnerabilities.

use command

trivy image "image name" | more

trivy i "image name" | more

They provide all CVEs. Summaries number of CVE: high, medium, low, critical, unknown. 

Note, it is "trivy i" it is not "trivy -i" 

Trivy can find misconfiguration and vulnerabilities in (1) file system and (2) git repository. 

We can check all YAML file with command

trivy conf "folder path of YAML files"

Notary: TUF based CNCF project. Image signing

Harbor: secure registry and manage artifacts 

Kritis: Validation webhook of K8s admission controller and CRDs. It uses Grafeas and Voucher. 

1. GenericAttestationPolicy : kritis.grafeas.io/v1beata1

spec: attestationAuthoriy 

2. AttestationAuthority: kritis.grafeas.io/v1beata1

spec: privateKeySecretName, publicData, noteReference. 

3. ImageSecurityPolicy: kritis.grafeas.io/v1beata1

spec: imageWhitelist / imageAllowlist

spec: packageVulnerabilityRequirement

  maximumSeverity: 

  whitelistCVEs: / allowlistCVEs

4. Default admittance fallback policy

Grafeas: API for image vulnerabilities meta data, attestation meta data all other structured metadata for artifacts. 

Terminology

Note: High level description of type of metadata. E.g. CVE as Vulnerability Note. Note ID format: /projects/"project_id"/notes/"note_id"Note ID is unique within project. It can be CVE ID

Occurrence: instance of note in artifact E.g. CVE presence in an image. Occurrence ID format: /projects/<project_id>/occurrences/<occurrence_id> Occurrence ID is random number

Providers: E.g. Vulnerability scanning. add notes and occurrence in DB

Consumer: E.g. Kritis

Resource URL: Part of occurrence. It is identifier for artifact. Immutable. It ties occurrence to image

Kind Specific Shema: Strict and structured. ATTESTATION, BUILD, DEPLOYMENT, DISCOVERY, IMAGE, PACKAGE, VULNERABILITY, INTOTO. Total 8. Easy to add new. It helps to support multi-vendor. 

The metadata can be part of binary authorization tool. 

Architecture

Auth system for strict access control. 

Client: not part of Grafeas

Grafeas API: core component

DB: Pluggable storage backends

Voucher: Make sure image after build and image at deployment, both are same.