6. Networking
Session state: New, Established, Related : (1) related DNS queries, (2) netfilter need protocol specific module. E.g. FTP, VoIP require extra kernel module.
specify module "-m state --state" OR "-m conntrack --ctstate). state module is subest of conntrack module)
Invalid: out of sequence traffic.
Anatomy of filter
1. Where to apply filter? (input, output, forward) chain
2. Which traffic to filter? (source and destination match criteria)
3. What action? chain are grouped in tables, as per action (filter, NAT, mangle, raw, security)
Applicable to both firewall and nwpolicy
chain v/s action
| Action: | PreRouting | Input | Output | Forward | PostRouting |
|---|---|---|---|---|---|
| raw | Y | N | Y | N | N |
| mangle | Y | Y | Y | Y | Y |
| nat | Y | N | Y | N | Y |
| filter | N | Y | Y | Y | N |
| security | N | Y | Y | Y | N |
Calico-GlobalNetworkPolicy configure connectivity rules to join WokrloadEndpoint and HostEndPoint in all NS. It has precedence over Profiles. Profiles used before Calico-NetworkPolicy is functional.
Calico n/w policy has
* (1) policy ordering/priority, (2) deny rules, and (3) more flexible match rules, over default K8s policy.
* K8s n/w policy is only for pods. Calico n/w policy is for pod, VM, host interfaces.
* along with Istio it supports securing 5-7 layers match criteria & cryptographic identity.
* works for all cloud provider.
========
* Neither Ingress nor Egress is specified then default is Ingress
* If no policy then all traffic allowed for pod
* If Ingress policy then only those ingress traffic is allowed.
* If egress policy then only those egress traffic is allowed.
* If no policy then all traffic denied for node
Reference: https://docs.projectcalico.org/security/calico-network-policy
WireGuard: VPN
- easy to use
- less feature
- speed
- with Calico clusters
Ingress Controller: Envoy Proxy, NGINX, Traefik, Ambassador
We need to add annotations accordingly to "Ingress" resource
kubernetes.io/ingress.class: haproxy
kubernetes.io/ingress.class: nginx
We can use https://nip.io/ to convert IP and DNS which contain IP.
In local setup, without Load Balancer, when we use NodePort, we have to use higher port in HOST, while using curl to ingress controller. E.g.
curl http://192.168.49.3 -H 'Host: nginx.192.168.49.3.nip.io:32735'
Service Mesh: Istio (security features: peer authentication, authorization, identity management. Zero-Trust Networking), Linkerd (for security), Countour (VMWare), Aspen (old name nginx. F5 purchased nginx and renamed as Aspen)
5. Securing Kube-APIServer: PSP, IAM, CIS
Pod Security Policy (PSP)
- A set of rules
- provide/modify default values for fields
- change pod
- PSP ordered by name before applied.
- Deprecated in K8s 1.21
- will be removed in K8s 1.25
Even if you are only planning on changing a single value, the policy file must contain several entries. Sample PSP, where pod can do anything
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: basicpolicy
spec:
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
fsGroup:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
allowedCapabilities:
- '*'
volumes:
-'*'
Most commonly changed parameters
1. privileged
2. runAsUser
Reference: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
For allowedUnsafeSysctls and forbiddenSysctls
- kernel (common prefix: kernel.)
- kernel.shm*,
- kernel.msg*,
- kernel.sem,
- networking (common prefix: net.)
- virtual memory (common prefix: vm.)
- MDADM (common prefix: dev.)
If we use RoleBinding instead of ClusterRoleBinding then it is for same namespace
--resource=podsecuritypolicy \
--resource-name=" # This Is optional"
--serviceaccount=namespace:default
The replicaset controller use default SA. So we should able to create deployment with about 2 commands also.
If controller manager connects to API server using trusted/insecure port then all PSS allowed, as authorization (and authentication) is bypass.
After enabling PodSecurityPolicy admission control plugin, we should have
1. This policy
kind: PodSecurityPolicy
metadata:
name: default-allow-all
spec:
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
fsGroup:
rule: RunAsAny
hostIPC: true
hostNetwork: true
hostPID: true
hostPorts:
- max: 65535
min: 0
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
2. We need clusterrole in target namespace
k -n team-red create clusterrole cr --verb=use --resource=psp
3. To add any new PSP, it should have min these fields
kind: PodSecurityPolicy
metadata:
name: example
spec:
privileged: false # Don't allow privileged pods!
# The rest fills in some required fields.
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
volumes:
- '*'
4. At each NS, we should have rolebidning.
k -n team-red create rolebinding rb --clusterrole=cr --user=system:serviceaccounts
OR
We can have clusterrolebinding
k -n team-red create clusterrolebinding crb --clusterrole=cr --user=system:serviceaccounts
References
https://banzaicloud.com/blog/pod-security-policy/
https://www.suse.com/c/rancher_blog/enhancing-kubernetes-security-with-pod-security-policies-part-2/
IAM using tools: keycloak , Active Directory, Amazon IAM
CIS It provides huge amount of free and paid resources to improve IT It provides security. tools, including benchmarks, scanning tools, threat tools, and hardened images. The CIS-CAT®Pro tool evaluates a target system against known issues and performance configurations. CIS also offers dashboards to view the ongoing state of compliance and security considerations.
For minikube setup we need to install kube-bench tool on individual node and run test. The test result recommend steps, for failure and warning cases. We can also run job.yaml at K8s cluster.
Have a look to summary of CIS for K8s in this Excel file
5. Securing Kube-APIServer: etcd
Tips to protect etcd
- Only Kube-APIServer is allowed to communicate with etcd
- Over TLS
- configmaps and secrets must not be compromised, as they are injected to application pod.
- Kube-APIServer must be updated to use encryption provider configuration
Encryption Providers
1. identity : NULL encryption
2. aescbc :
- Slow and strong
- Recommended
- key length = 32 bits
- Algo: Public Key Cryptographic Standards (PKCS): Advanced Encryption Standard (AES) cipher algo in Cipher Block Chaining (CBC) mode. Refer RFC 3602
3. secretbox
- faster
- key length = 32 bits
Algo: XSalsa20 and Poly1305 encryption
4. aesgcm
- fastest
- key length = 16 bots or 32 bits
- key rotation every 200,000 writes
5. kms
- uses new Data Encryption Key (DEK) for every encryption along with AES-CBC
Sample Encryption Policy File
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
name: configureKeys
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: encKey
secret: Ezqai0SIGChGOwf0VdbkFtYPUM2EYf1TAAQbDrfizJQ=
- identity: {}
Assume the above YAML file is located at /var/lib/minikube/certs/ec.yaml then we shall mount this hostPath with same folder and specify to API Server with argument :
--encryption-provider-config=/var/lib/minikube/certs/ec.yaml
- One can encrypt many K8s objects instead of just secrets.
- new key should be at top of conifg file to encrypt. next line can be old key, to decrypt previously encrypted parameters.
- Encryption key should be in separate KMS (key management system)
5. Securing Kube-APIServer: Audit
Audit-log
* --audit-log-maxage Recommended: Minimum 30 (days)
* --audit-log-maxbackup Recommended: Minimum 10 (files)
* --audit-log-maxsize Recommended: 100 (MB) or appropriate
* --audit-policy-file=/etc/kubernetes/simple-policy.yaml
This declares where, inside the container, the policy file is located. You will also need to add a volume and mount point to the container.
* --audit-log-path Example value: /var/log/apiserver/audit.log
This declares where the logs should be kept inside the container. This will also need a volume and a mount point configured.
Audit-process
If audit is enabled then each API goes through audit process, even if it is filtered out.
3 Phases
1. Call is received: RequestReceived
2. Handling the call: ResponseStarted (Only applicable for long running request like watch)
3. Response call is made: ResponseComplete
* Panic
Each even has audit rule. Audit rule set audit level
- None
- Metadata
- Request
- RequestResponse
Sample policy files
Note : all resources are always in plural in all policy file.
1.
kind: Policy
rules:
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
- level: Metadata
omitStages:
- "RequestReceived"
2. monitor events for specific pods
kind: Policy
rules:
- level: RequestResponse
resources:
- group: "" # Version of group should NOT be included.
resources: ["pods"]
resourceNames: ["sp1"]
3. Monitor all request and response from kubectl on minikube setup
kind: Policy
rules:
- level: RequestResponse
users: ["minikube-user"]
omitStages: ["ResponseComplete"]
- level: RequestResponse
users: ["system:kube-proxy"]
userGroups: ["system:authenticated"]
namespaces: ["kube-system"]
verbs: ["watch"]
resources:
- group: "" # Version of group should NOT be included.
resources: ["pods"]
resourceNames: ["controller-leader"]
The first matching rule sets the audit level of the event. The omitStages can be per rule or not per rule.
Audit backend
1. Log
2. Webhook
Step 1, create file /etc/kubernetes/audit-webhook-kubeconfig on master node
kind: Config
clusters:
- cluster:
server: http://10.0.2.15:7777
name: falco
contexts:
- context:
cluster: falco
user: ""
name: default-context
current-context: default-context
preferences: {}
users: []
5. Securing Kube-APIServer: Parameters.
phase of each API at Kube-APIServer
1. Authentication
2. Authorization
3. Admission Control
By default, kubeadm-built clusters allow API access without authentication and authorization.
* IP = 127.0.0.1 TLS = No Port = 8080. First two phases are skipped
* IP = First n/w interface IP TLS = Yes Port = 6443. All three phases are executed.
Several arguments are passed to the kube-apiserver process via /etc/kubernetes/manifests/kube-apiserver.yaml
Sample file content
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=10.128.15.193
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
Few parameters worth to highlight
* --advertise-address: Used by entire cluster. If absent, then use --bind-address. If that is also absent then use IP address of first network interface
* --allow-privileged : Privileged containers are isolated or not. If absent then value is false. kubeadm set it true. Change application, after modifying the flag.
* --enable-bootstrap-token-auth : If true, bootstrapping tokens can be used as bearer token credentials. It allows requests to authenticate against the API server.
* --etcd-servers=https://127.0.0.1:2379
* --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
If Kube-APIServer need to communicate with Aggregated API (AA) Server:
* --proxy-client-cert-file
* --proxy-client-key-file
How Kube-APIServer and kube-proxy communicate with AA server.
* --requestheader-allowed-names=front-proxy-client
* --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
* --requestheader-extra-headers-prefix=X-Remote-Extra-
* --requestheader-group-headers=X-Remote-Group
* --requestheader-username-headers=X-Remote-User
Many other parameters are captured at https://layers7.blogspot.com/2021/01/k8s-api-server-security-settings.html
