7. Workload Considerations : Tracee
Tracee (uses eBPF) :monitors system call and kernel events.
- It captures : (1) precise time stamp, (2) uts_name, (3) UID, (4) Command COMM (5) PID, (6) TID/host (7) return code, RET (8) event, and (9) arguments.
- At least 3 volume locations needed. (1) /lib/modules , (2) /usr/src , (3) /tmp/tracee Tracee provides in-depth tracing of container or pod.
Tracee has multiple options. Important ones are
list: list of system calls and other events.
trace: events. specific pid, uid, uts, mntns, pidns, command (comm), system call etc. We can use comparison operator to filer.
Examples:
--trace pid=new | only trace events from new processes
--trace pid=510,1709 | only trace events from pid 510 or pid 1709
--trace p=510 --trace p=1709 | only trace events from pid 510 or pid 1709 (same as above)
--trace container=new | only trace events from newly created containers
--trace container | only trace events from containers
--trace c | only trace events from containers (same as above)
--trace '!container' | only trace events from the host
--trace uid=0 | only trace events from uid 0
--trace mntns=4026531840 | only trace events from mntns id 4026531840
--trace pidns!=4026531836 | only trace events from pidns id not equal to 4026531840
--trace 'uid>0' | only trace events from uids greater than 0
--trace 'pid>0' --trace 'pid<1000' | only trace events from pids between 0 and 1000
--trace 'u>0' --trace u!=1000 | only trace events from uids greater than 0 but not 1000
--trace event=execve,open | only trace execve and open events
--trace set=fs | trace all file-system related events
--trace s=fs --trace e!=open,openat | trace all file-system related events, but not open(at)
--trace uts!=ab356bc4dd554 | don't trace events from uts name ab356bc4dd554
--trace comm=ls | only trace events from ls command
--trace close.fd=5 | only trace 'close' events that have 'fd' equals 5
--trace openat.pathname=/tmp* | only trace 'openat' events that have 'pathname' prefixed by "/tmp"
--trace openat.pathname!=/tmp/1,/bin/ls | don't trace 'openat' events that have 'pathname' equals /tmp/1 or /bin/ls
--trace comm=bash --trace follow | trace all events that originated from bash or from one of the processes spawned by bash
--trace container=new | all the events from container created after issuing this command
5. Securing Kube-APIServer: RBAC
We can use
kubectl auth reconcile -f "filename.yaml"
to create missing objects and ns. It does not create sa
We can also run with
kubectl auth reconcile -f "filename.yaml" --dry-run=client
--remove-extra-permissions will remove extra permission in role
--remove-extra-subjects will remove extra subjects in binding
The kubectl auth reconcile command will ignore any resources that are not Role, RoleBinding, ClusterRole, and ClusterRoleBinding objects, so you can safely run reconcile on the full set of manifests. Next we can run kubectl apply command.
With kubectl apply command for rolebinding, we cannot update roleRef. it is immutable. However with this command kubectl auth reconcile, we can do it.
All the above points are applicable to ClusterRole and ClusterRoleBinding also.
Reference: https://www.mankier.com/1/kubectl-auth-reconcile
====================================
Regardless of namespace, by default, SA with name "default" is added to pod, in all namespace.
====================================
In rolebinding if kind = User then only name is sufficinet.
- kind: User
name: dan
6. Networking : Network Policy
- We cannot use namespaceSelector, for target pod. The namespaceSelector is for (1) to and (2) from
- if we do not mention about podSelector at all, then it means none of the pod.
- if we mention empty list , then also it means none of the pod. ingress: []
==================================
- For (1) to and (2) from, if you omit specifying a namespaceSelector it does not select any namespaces, which means it will allow traffic only from the namespace the NetworkPolicy is deployed to.
To allow all traffic from current namespace
ingress:
- from:
- podSelector: {}
==================================
- if we mention
ingress: {}
OR
ingress:
- {}
then it means network all pods from all namespace + outside K8s cluster
- if we mention
ingress:
- from:
- namespaceSelector: {}
==================================
- All policies are add / union. So there is no chance of conflict. Whitelist can be keep growing. Traffic is allowed, if we have at least one rule, that allow the traffic.
- By default, if no policies exist in a namespace, then all ingress and egress traffic is allowed to and from pods in that namespace
- Network Policy is connection level filter. It does not apply to packets
- Network Policy does not terminate established connection.
- cluster level network policy is not part of core API. It is implemented by Calico
==================================
Best practices
1. First block all ingress/egress in a namespace
2. start whitelisting for each app
3. While applying egress rule, we have to allow DNS, as it is needed in most cases, to resolve service FQDN
==================================
- If no policyTypes are specified on a NetworkPolicy then by default Ingress will always be set
- policyTypes= Egress will be set if the NetworkPolicy has any egress rules.
==================================
This is OR condition
kind: NetworkPolicy
metadata:
name: test-network-policy
namespace: default
ingress:
- from:
- ipBlock:
cidr: 172.17.0.0/16
except:
- 172.17.1.0/24
- namespaceSelector:
matchLabels:
KEY: VALUE
- podSelector:
matchLabels:
KEY: VALUE
Here : any pod whose namespace has label key=value OR any pod with namespace of Networkpolicy (default) who has label key=value OR pod has specific IP addresss
==================================
This is AND condition
ingress:
- namespaceSelector:
matchLabels:
user: alice
podSelector:
matchLabels:
role: client
This is also AND condition
- from:
ports:
- protocol: TCP
port: 6379
We have to use containerPort only.
We can have multiple rules by multiple "-from" and/or multiple "-to"
==================================
To allow all traffic from all namespace
(1)
ingress:
- from:
- podSelector: {}
namespaceSelector: {}
(2)
ingress:
- from:
- namespaceSelector: {}
Port is always destination port, for both ingress and egress.
==================================
We can block egress traffic go outside cluster, by (1) specifying allow to all namespace
egress:
- to:
- namespaceSelector: {}
(2) empty list
egress: []
==================================
First let's isolate Ingress and Egress both traffic to target pod as per podSelector. These pods belongs to same namespace, as the NetworkPolicy belong to. Here all pods with label role=db in default namespace are isolated.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: test-network-policy
namespace: default
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
Reference:
https://kubernetes.io/docs/concepts/services-networking/network-policies/
8. Issue Detection
Cyber Kill Chain
- Reconnaissance
- Weaponization: Client application data file: PDF, DOC
- Delivery: E-mail attachment, wesbite, USB removable media
- Exploitation:
- Installation:
- Command and Control
- Actions on Objectives
Acronyms
AIDE: Advanced Intrusion Detection Environment
C2: Command and Control
COOP: COntinuity of OPeration
CVEs: Common Vulnerabilities and Exposures
DR: Disaster Recovery
HIDS: Host Intrusion Detection System
IDS: Intrusion Detection System
IPS: Inline Intrusion Prevention System
LM-CIRT: Lockheed Martin Computer Incident Response Team
NIDS: Network Intrusion Detection System
NSM: Network Security Monitoring
NVD: National Vulnerability Database
PIDS: Physical Intrusion Detection System
US-CERT: United States Computer Emergency Readiness Team
Website
7. Workload Considerations : AppArmor
- AppArmor is less complete and simple.
- It is available on Debian and SUSE Linux distribution.
- It supplements UNIX Discretionary Access Control (DAC) model. It provides MAC (Mandatory Access Control).
- Its learning mode (complain mode) is similar to SELinux's Permissive mode. Here, profile violations are logged but not prevented. This log can be turn into profile.
- No security labels are needed, so it is filesystem-neutral
- Administrator can associate security profile to program.
- Unlike SELinux: instead of direct labeling of objects, security policy is applied to pathnames.
K8s
- AppArmor profile must be available at worker node, so pod can use it. With Ansible or Puppet, AppArmor profile can be added to worker node, during installation. OR use Daemonset.
- To disable AppArmor for entire cluster, pass --feature-gate=AppArmor=false.
- AppArmor profiles can be managed using PSP
- If AppArmor kernel module is available then
sudo systemctl [start|stop|restart|status] apparmor
- To load or not load at boot time
sudo systemctl [enable|disable] apparmor
- To see current status
sudo apparmor_status
Modes
1. Enforced mode
Default mode
aa-enforce
2. Complain
also called learning mode
aa-complain
Profiles
- Pre-package profile
- install along with new software install
- install with AppArmor package: apparmor-profile
- stored at /etc/apparmor.d
- "man apparmor.d" provides documentation.
Other utilities
- apparmor-notify: summary for AppArmor log messages
- disable: unload a single profile. and not load during boot
- easyprof: Help to setup a basic AppArmor profile for a program
- logprof: Scan log. If any AppArmor event found, that is not covered by existing profiles, then suggest.
- genprof: Createa new complain mode profile, using existing profiles as input parameter. Run logprof to scan AppArmor events. All entries in system log has option (A) Allow (D) Deny (I) Ignore (N) New (G) Glob last piece (Q) Quit. until Quit is selected. then new people is created.
- Bane: AppArmor profile generator for docker container. It automatically install profile in directory /etc/apparmor.d/containers/
List all AppArmor utilities using
rpm -qil apparmor-utils | grep bin
Access control to assign in AppArmor profile
- r : Read
- w : Write
- m : Memory map as executable
- k : File locking
- l : Create hard links
- ix : Execute and inherit this profile
- Px : Execute another profile after cleaning environment
- Ux : Execute unconfined after cleaning environment.
--feature-gates=AppArmor=t|f
Add this metadata to pod
container.apparmor.security.beta.kubernetes.io/<container_name>: <profile_ref>
Note: This is container name, not pod name
This profile name, not profile file name
E.g. container.apparmor.security.beta.kubernetes.io/hello: localhost/k8s-apparmor-example-deny-write
We should have some file at /etc/apparmod.d/ path, which should this profile k8s-apparmor-example-deny-write
profile_ref
- runtime/default
- localhost/<profile_name>
- unconfined
For PSP:
apparmor.security.beta.kubernetes.io/defaultProfileName: <profile_ref>
apparmor.security.beta.kubernetes.io/allowedProfileNames: <profile_ref>[,others...]
