Hashicorp User Group Bangalore Meetup #1 : Powering the Multi-Cloud Era
Alternatives for IDP
(1) https://github.com/JanssenProject/jans https://github.com/JanssenProject/jans/tree/main/jans-keycloak-link https://imshakil.medium.com/janssen-mod-auth-openidc-module-to-test-openid-connect-single-sign-on-s… It is by Glu
(2) Vault it self support OIDC https://developer.hashicorp.com/vault/docs/secrets/identity/oidc-provider https://brian-candler.medium.com/using-vault-as-an-openid-connect-identity-provider-ee0aaef2bba2
SQL++ is for JSON data. https://www.couchbase.com/sqlplusplus/
https://techmilap.com/ is free website for hosting event
Vault can provide dynamic temporary secrets to access data for each identity used by consumer. so later on, we can audit, who has accessed data. In our case, pods use ServiceAccount (SA). here we get dynamic secret per serviceaccount. So we cannot audit which pod accessed the data. we can only audit, data is accessed by which ServiceAccount. This dynamic secret has short life so one cannot use it again. SA we can use it as many time as we want.
Vault secure data in-transit with TLS and other encryption method that is called "encryption as a service"
In terraform, state file is the most confidential.
Nomad is alternative of K8s. It can manage VM also using QEMU driver. Consul is used for networking and service. Fabio is for ingress and load balancing in Nomad.
Identity Provider
https://github.com/pando85/kaniop Kaniop is a Kubernetes operator for managing Kanidm.
https://kanidm.com/ Kanidm is a modern, secure identity management system that provides authentication and authorization services with support for POSIX accounts, OAuth2, and more. It is simple and written in rust
IDP
(1)
https://github.com/JanssenProject/jans
https://github.com/JanssenProject/jans/tree/main/jans-keycloak-link
https://imshakil.medium.com/janssen-mod-auth-openidc-module-to-test-openid-connect-single-sign-on-s…
It is by Glu
(2) Vault it self support OIDC https://developer.hashicorp.com/vault/docs/secrets/identity/oidc-provider https://brian-candler.medium.com/using-vault-as-an-openid-connect-identity-provider-ee0aaef2bba2
-------------
Why Choose Keycloak?. Understanding the Need for an Identity… | by J3 | Jungletronics | Medium
Ory
GitHub - ory/k8s: Kubernetes Helm Charts for the ORY ecosystem. · GitHub
SPIFFE
SA is at cluster level.
So Nepheo could not use SA
Every CSP has workload identity
spiffe is standard:
- spiffe id. It is URL.
- spiffe verifiable documents (SVIDs): cert or toekn
- The spiffe workload API.
spire: spiffe Runtime Environment.
- A toolchain of API for establising trust based on spifee
- provides out of the box attestation plugins
Expiry is short. can be 4 hours. So no need of revocation
* spire agent can be colocated. it is dameonset in K8s.
=========
Nephio
ss7, sigtra, ngin, CN model (e.g. ORAN)
DISH is on AWS
CP based requirement for identity
Nephio SIG security wiki page has all details
Porch : Package Orchestration KPT
KPT does in place substitution
5G requirements / usecases
IMS, SMO , IMS
LF article about Nephio spifee implementation at LF wiki
Catalog packages at GitOps
Each cluster shall have its own repo
Identity federation is based on cert chain.
R3 Oct 23 of Nephio.
It is proposed solution. It will be upstream.
Workload identity solution shall not be native to specific cloud provider.
Identity federation across CSPs.
Google, E//, RedHat are in Nephio
SPIRE's alternative may be due to speicfic attestion plugin
What protocol between SPIRE Agent and SPIRE server? Bootstrap trust. it is pre-provision aspect. REST API and TLS. x.509 cert will be pulled. protocol is spire specific
Today's attestation is based on SA, pod labels, namespace.
CA, Cert Manager can be used.
eBPF
- eBPF programme in C, Rust or other programming languages.
- Compiler converts it to bytecode (obj file)
- Application pass this OBJ to kernel using eBPF library
- Kernel verifies it
- When event happens the program runs
- eBPF programme in C, Rust or other programming languages.
- Compiler converts it to bytecode (obj file)
- Application pass this OBJ to kernel using eBPF library
- Kernel verifies it
- JIT compiler converts byte code to machine code
- When event happens the program runs
- eBPF programme in C, Rust or other programming languages.
- Compiler converts it to bytecode (obj file)
- Application pass this OBJ to kernel using eBPF library
- Kernel verifies it
- JIT compiler converts byte code to machine code
- When event happens the program runs
- Multiple eBPF programs can access eBPF map at Kernel space
- Userspace App load config to map
- eBPF program pull config from map
- eBPF program sends metrics to map
- eBPF program stores/retrieve data from/to map
- eBPF map can store state in eBPF program
- BPF_TABLE
- BPF_HASH
- BPF_ARRAY
- BPF_HISTOGRAM
- BPF_PERF_ARRAY
- eBPF programme in C, Rust or other programming languages.
- Compiler converts it to bytecode (obj file)
- Application pass this OBJ to kernel using eBPF library
- Kernel verifies it
- JIT compiler converts byte code to machine code
- When event happens the program runs
- Multiple eBPF programs can access eBPF map at Kernel space
- Userspace App load config to map
- eBPF program pull config from map
- eBPF program sends metrics to map
- eBPF program stores/retrieve data from/to map
- eBPF map can store state in eBPF program
- BPF_TABLE
- BPF_HASH
- BPF_ARRAY
- BPF_HISTOGRAM
- BPF_PERF_ARRAY
- eBPF program can call Kernel function (eBPF helper)
Everything about PKI
entity = computer | user
Every entity has identity
Authentication is verification of claim by entity
======================================
Hash function: same input => same output. If input is different by even a bit, output is completely different. They are one way. MAC (Message Authentication Code) is hash function. MAC needs common key.
Signature is similar but uses key pair. If only one entity knows the private key you get a property called non-repudiation: the private key holder can't deny (repudiate) the fact that they signed some data.
sign with private key. verify with public key.
encrypt with public key. decrypt with private key
public key cryptography = asymmetric cryptography, as above
=====================================
subscriber or end entity is subject of certificate
CA is issuer of certificate
CA has root certificate | intermediate certificate
end entity has leaf certificate
relying party trust CA and verify certificate.
relying parties are pre-configured with a list of trusted root certificates (or trust anchors) in a trust store. Root certificates in trust stores are self-signed . Trust store by 4 major organization
1. Apple's root certificate
2. Microsoft's root certificate programm
3. Mozila's root certificate programm
2. Google's root certificate programm
Cloudflare's cfssl project maintains a github repository that includes the trusted certificates from various trust stores. https://github.com/cloudflare/cfssl_trust
=======================================
Certificate : issues says entity (subject) has public key
X.509, ASN.1, OIDs, DER, PEM, PKCS
X.509 builds on ASN.1. ASN.1 : DER (Binary) and BER
PEM: base64 encoded DER payload sandwiched between a header and a footer
Possible extension for PEM: .der, .pem, .crt, .cer
======================================
PKCS (Public Key Cryptography Standards) published by RSA labs
- * PKCS#7 is rebranded as Cryptographic Message Syntax (CMS) is by IETF. It is used by Java.
Possible extension for PKCS#7: .p7b, .p7c
- * PKCS#12 = certificate chain + private key
It is used by microsoft products
Possible extension for PKCS#12: .pfx, .p12
So possible formats for PKCS#7 and PKCS#12 are : raw der, pem, ber
raw der is most widely used.
- * PKCS#8 is for private key and its metadata
Possible extension for private key: .prv, .key, .pem
Possible extension for public key: .pub, .pem
They may includes headers: Proc-Type, DEK-Info
- * PKCS#10 is CSR
====================================
- SSH: certificate less PKI .It binds names to public key in files
- PGP: uses certificate, but not CA. It uses web-of-trust model
- Web PKI (Internet PKI or PKIX) works with browser. no control over important details like certificate lifetime, revocation mechanisms, renewal processes, key types, and algorithms
=====================================
Bundle of certificate: root intermediate leaf - forms certificate chain. More often, certificate chains are encoded as a simple sequence of line-separated PEM objects. Some stuff expects the certs to be ordered from leaf to root, other stuff expects root to leaf, and some stuff doesn't care. The relying party verifies the leaf and intermediate certificates in a process called certificate path validation.
=====================================
PKIX originally specified to use FQDN in DN common name. The modern best practices is to leverage SANs. There are four sorts of SANs in common use, all of which bind names that are broadly used and understood: domain names (DNS), email addresses, IP addresses, and URIs.
=====================================
- To configure a PKI relying party you tell it which root certificates to use
- To configure a PKI subscriber you tell it which certificate and private key to use (or tell it how to generate its own key pair and exchange a CSR for a certificate itself)
Apigee
1. integrated developer portal
2. convert a set of API into product: R/W access, rate per second, and how to use it. mix and match methods, operatoins from different APIs. code as pricing model. bronze/silver/gold customer. break out APIs in different packages.
3. Security
out of box policy
- authentication
- authorization
- rate limit
- spikers
- JSON based threat
- XML based threat
- use other Google cloud tool
-- cloud armor for DDoS
It sops absuive or bot traffic ASAP, before it enters into system.
New Advance API security feature
- Identify mis configured API.
-- security score for API
-- recommended action
-- abuse of API detection
-- API security dashboard for quick investigation and resolution
- Bot detection: API traffic pattern
4. Application Integration : 50+ connectors to 3rd party services,
API hub: API catalogue and API LCM
5. Measure and tune APIs: Analytic
6. engage and innovate with partner to create digital eco system.
7. full API LCM platform
- builder façade
- caching
Refernece: https://www.youtube.com/watch?v=KfJoQAKoQiM
minikube kubeconfig
- from .kube/config file, extract cluster.certificate-authority-data| base64 -d > ca.crt
openssl x509 -in ca.crt -text -noout > ca.crt.decode
From ca.crt.decode:
Issuer: CN = minikubeCA
Subject: CN = minikubeCA
- from .kube/config file, extract user.user. client-certificate-data | base64 -d > client.crt
openssl x509 -in client.crt -text -noout >
client.crt.decode
From client.crt.decode
Issuer: CN = minikubeCA
Subject: O = system:masters, CN = kubernetes-admin
CKS Tips
1. Shortcut
export now="--force --grace-period 0" # k delete pod x $now
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
10. securityContext.capabilities is only for container, not for pod
11. securityContext.readOnlyRootFilesystem is only for container, not for pod. Here we should use word "Root" and s is small in system.
12. Instead of applying (1) label to node and (2) then use nodeSelector, we can use nodeName in pod spec.
spec:
nodeName: cluster1-worker2 # add
13. To run command inside pod and take its output to outside pod. here the final command to be run inside the pod should be at the end.
14. To run etcdctl
ETCDCTL_API=3
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
14.3. Now do mapping. parameter value in kube-apiserver.yaml to input argument for etcd
--etcd-certfile mapped to --cert
--etcd-keyfile mapped to --key
system:serviceaccount:(singular) is the prefix for service account usernames.system:serviceaccounts:(plural) is the prefix for service account groups.
Security Context and PSP
Security Context
Only for container
allowPrivilegeEscalation: false . It is for setuid and setgid
allowPrivilegeEscalation:
if (CAP_SYS_ADMIN || Privileged mode) then AllowPrivilegeEscalation = SSeue
=============
capabilities:
drop:
- all
add: ["MKNOD"]
in PSP we have
allowedCapabilities:
- '*'
requiredDropCapabilities:
- ALL
=============
privileged: true avoid it.
=============
procMount: true is for nested container useful for in-cluster build process
=============
readOnlyRootFilesystem: true for immutable container
=============
For container and pod both
container settings will get precedence.
=============
runAsUser and runAsGroup are runtime configuration. If not defined then UID as per data in image
If runAsNonRoot = True then image should have UID. Same UID should be in host.
=============
securityContext:
level: "s0:c123,c456"
we can specify level, role, type, user. They are labels for file, process and ports. They are collectively called called "context" in SELinux terms.
=============
type = Localhost | Unconfined | RuntimeDefault
type: Localhost
localhostProfile: profiles/name.json
where kuelet-root-dir = /var/lib/kubelet .
It is configured with seccomp-profile-root flag for kubelet. This flag is deprecited since 1.19 . If seccomp-profile-root flag is not defined, the default path will be used, which is <root-dir>/seccomp where <root-dir> is specified by the --root-dir flag.
* type: RuntimeDefault means same seccomp profile as container runtime default is applied
* type: Localhost, then only, we need to set localhostProfile
We need to use annotation (optional)
seccomp.security.alpha.kubernetes.io/pod: localhost/profiles/audit.json
Path for seccomp profiles: https://k8s.io/examples/pods/security/seccomp/profiles/
=============
Only for Pod
fsGroup
the permission bit will be | with rw-rw----
ownership and permission change recursively for all content in mounted volume as per fsGroup
=============
fsGroupChangePolicy = OnRootMismatch | Always
no impact on emptyDir, secret and configMap
OnRootMismatch: It will save time. The permission and ownership only change if root level folder has mismatch with expected fsGroup.
If DelegateFSGroupToCSIDriver feature gate is enabled then this is done by CSI driver. CSI driver will not respect fsGroupChangePolicy
=============
supplementalGroups
A list of groups applied to the first process run in each container, in addition to the container's primary GID.
=============
sysctls
====================================================
Here Discretionary Access Control (DAC) is related fields are: runAsUser, runAsGroup, runAsNonRoot, readOnlyRootFilesystem
Here volume related fields are: fsGroup and seLinuxOptions. We set only level at seLinuxOptions and as per level, labels applies to all containers and volumes.
PodSecurityPolicy
Pod's scrutiny attributes. Defined at cluster level. It controls security sensitive aspects of the pod specification. It define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields.
It (1) restrict pod creation (2) restrict pod update (3) provide default value
=============
- "privileged" container (Privileged Mode). It is part of PSP. But it control container level "security context"
=============
- host-level ns (network, PID, IPC) "hostPID", "hostIPC", "hostNetwork". If hostPID allowed then container can escalate privilege using ptrace system call.
=============
- host ports "hostPorts"
- min: 0
max: 65535
- different types of volumes. E.g. "allowedFlexVolumes" "volumes"
=============
- host's filesystem E.g. "fsGroup"
fsGroup = MustRunAs | MayRunAs | RunAsAny
We shall specify 1+ range if fsGroup = MustRunAs | MayRunAs. In case of MustRunAs the fsGroup at Pod Security Context is set as min value. in case of MayRunAs, the default value for fsGroup at Pod Security Context, is unset
=============
- RO root filesystem for containers (DAC) "readOnlyRootFilesystem"
=============
- user IDs and group IDs (DAC) "runAsUser" "runAsGroup" "supplementalGroups"
runAsUser = MustRunAs | MustRunAsNonRoot | RunAsAny
runAsGroup = MustRunAs | MayRunAs | RunAsAny
supplementalGroups = MustRunAs | MayRunAs | RunAsAny
=============
- containers' privilege escalation "allowPrivilegeEscalation" "defaultAllowPrivilegeEscalation". Here defaultAllowPrivilegeEscalation, set values for allowPrivilegeEscalation, if not set.
=============
- containers' Linux capabilities (Linux Capabilities): "defaultAddCapabilities", "requiredDropCapabilities", "allowedCapabilities"
=============
- SELinux "seLinux"
RunAsAny means: Apparmor is used instead of SELinux.
=============
- seccomp and AppArmor profiles : using annotations.
seccomp.security.alpha.kubernetes.io/allowedProfileNames
apparmor.security.beta.kubernetes.io/defaultProfileName
If this annotation is not specified then default seccomp cannot be changed
=============
- sysctls that a pod can run: "forbiddenSysctls", "allowedUnsafeSysctls"
=============
- a proc mount type to be used. "allowedProcMountTypes" and "DefaultProcMount"
allowedHostPaths:
# This allows "/foo", "/foo/", "/foo/bar" etc., but
# disallows "/fool", "/etc/foo" etc.
# "/foo/../" is never valid.
- pathPrefix: "/foo"
readOnly: true # only allow read-only mounts
=============
PSP Policy Order
1. non-mutating policy in any order
2. mutating policy in alphabetical order of name
Good Reference:
A Good article: https://www.linux.com/training-tutorials/overview-linux-kernel-security-features/
Abbreviation
ASLR Address Space Layout Randomization
DAC Discretionary Access Control
IMA Integrity Measurement Architecture
MAC Mandatory Access Control
PAM Pluggable Authentication Modules
7. Workload Considerations : Falco
Falco by Sysdig: multiple components (user space program, configuration, driver) working together in order to evaluate system calls against rules, and generate alerts when a rule is broken:
rule has lists. rule can have reference to list. List can be part of macro and other list, in addition to part of rule.
rule has 5 k-v pairs. (1) name, (2) description , (3) condition : Filtering expression for events. (4) output, (5) priority. (emergency, alert, critical, error, warning, notice, informational, debug)
rule has 4 optional K-v pairs.(1) enabled. default is true (2) tags (filesystem, software_mgmt, process, database, host, shell, container, cis, users, network) . -T option to disable rules with given tag. -t option to enable. (3) warn_evttypes default is true. (4) skip-if-unknown-filter default is false. 5th one added (5) exceptions : a set of conditions that cause the rule to not generate an alert.
- Falco comes with many rules in /etc/falco/falco_rules.yaml file. They can be overwritten by /etc/falco/falco_rules.local.yaml file. E.g. to disable rule : We can add rule with same name and "append: true" + "enabled: false"
- evt.dir = < indicates end of system call and evt.dir = > indicates beginning of system call. dir = direction
- We have K8s related context: k8s.[pod | rc | svc | rs | deployment].[name | id \ label | labels] + many fields from K8s audit logs.
with macro part of rule can be re-used. There are many default macros.
Falco runs with K8s-audit on. So we need to specify audit policy file at API server argument --audit-policy-file
We can configure webhook in API server with this arguement
--audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
This YAML file shall define Config kind.
We can see Falco output with journalctl command.
Reference :
https://github.com/falcosecurity/falco/blob/master/rules/k8s_audit_rules.yaml
https://v1-17.docs.kubernetes.io/docs/tasks/debug-application-cluster/falco/
7. Workload Considerations : Tracee
Tracee (uses eBPF) :monitors system call and kernel events.
- It captures : (1) precise time stamp, (2) uts_name, (3) UID, (4) Command COMM (5) PID, (6) TID/host (7) return code, RET (8) event, and (9) arguments.
- At least 3 volume locations needed. (1) /lib/modules , (2) /usr/src , (3) /tmp/tracee Tracee provides in-depth tracing of container or pod.
Tracee has multiple options. Important ones are
list: list of system calls and other events.
trace: events. specific pid, uid, uts, mntns, pidns, command (comm), system call etc. We can use comparison operator to filer.
Examples:
--trace pid=new | only trace events from new processes
--trace pid=510,1709 | only trace events from pid 510 or pid 1709
--trace p=510 --trace p=1709 | only trace events from pid 510 or pid 1709 (same as above)
--trace container=new | only trace events from newly created containers
--trace container | only trace events from containers
--trace c | only trace events from containers (same as above)
--trace '!container' | only trace events from the host
--trace uid=0 | only trace events from uid 0
--trace mntns=4026531840 | only trace events from mntns id 4026531840
--trace pidns!=4026531836 | only trace events from pidns id not equal to 4026531840
--trace 'uid>0' | only trace events from uids greater than 0
--trace 'pid>0' --trace 'pid<1000' | only trace events from pids between 0 and 1000
--trace 'u>0' --trace u!=1000 | only trace events from uids greater than 0 but not 1000
--trace event=execve,open | only trace execve and open events
--trace set=fs | trace all file-system related events
--trace s=fs --trace e!=open,openat | trace all file-system related events, but not open(at)
--trace uts!=ab356bc4dd554 | don't trace events from uts name ab356bc4dd554
--trace comm=ls | only trace events from ls command
--trace close.fd=5 | only trace 'close' events that have 'fd' equals 5
--trace openat.pathname=/tmp* | only trace 'openat' events that have 'pathname' prefixed by "/tmp"
--trace openat.pathname!=/tmp/1,/bin/ls | don't trace 'openat' events that have 'pathname' equals /tmp/1 or /bin/ls
--trace comm=bash --trace follow | trace all events that originated from bash or from one of the processes spawned by bash
--trace container=new | all the events from container created after issuing this command
