GitOps


What is GitOps?

* GitOps is code-based infrastructure and operational procedures that rely on Git as a source control system

For Kubernetes GitOps means using git push instead of kubectl create/apply or helm install/upgrade.

 It’s an evolution of Infrastructure as Code (IaC) and a DevOps best practice that leverages Git as the single source of truth, and control mechanism for creating, updating, and deleting system architecture. 

* Git is a single source of truth for
- CD
- automated deployment
- monitoring
- management 
- entire state of the system

* GitOps extends pipelines with a feedback loop for observing and controlling the system .

* A set of practices to use "Git pull requests" (1) to manage infrastructure and (2) to manage application configurations (3) automatically deploy system infrastructure modifications. All git procedures are applies like: review, pull request, push requests, tagging, versioning etc. 

* GitOps practice
- Declarative description of system : stored in Git
- Pull requests modify the state of the Git repository. 
- Once approved and merged, the pull requests will automatically reconfigure and sync/reconcile the live infrastructure to the state of the repository. This is done by controller software. It self-heal the system or notify the drift. 
This live syncing pull request workflow is the core essence of GitOps.

* Declarative Description (YAML) of desired production infrastructure in git + automate to match production environment state with desired state at git

* Continuous Deployment for cloud native applications

GitOps is an incredibly powerful workflow pattern for managing modern cloud infrastructure

GitOps is an extension of IaC and declarative configuration

* git push + CI/CD toolchain + UI/UX

* Tools : Git , CD tools (tools for declarative infrastructure as code)

* Developer tools (read git) to drive operations. 

* GitOps provide more stability and reliability over typical CI/CD pipeline


The GitOps idea was first hatched and shared by WeaveWorks, an enterprise Kubernetes management firm

GitOps operator is a mechanism that sits between the pipeline and the orchestration system (read K8s) . A pull request starts the pipeline that then triggers the operator. The operator examines the state of the repository and the start of the orchestration and syncs them. 


Principles

1. The entire system is described declaratively. 
2. The canonical desired system state versioned in Git
3. Approved changes that can be automatically applied to the system. 
4. Software agents to ensure correctness and alert on divergence 

Tools:



  • ArgoCD: A GitOps operator for Kubernetes with a web interface. It supports multiple clusters and multiple Git repositories. 
  • Flux: The GitOps Kubernetes operator by the creators of GitOps — Weaveworks No central Management, No central UI. 
  • Gitkube: A tool for building and deploying docker images on Kubernetes using git push
  • kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.
  • BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner.
  • JenkinsX: Continuous Delivery on Kubernetes with built-in GitOps
  • Tkton Pipelines project provides k8s-style resources for declaring CI/CD-style pipelines.
  • Terragrunt: A wrapper for Terraform for keeping configurations DRY, and managing remote state
  • WKSctl: A tool for Kubernetes cluster configuration management based on GitOps principles
  • Helm Operator: An operator for using GitOps on K8s with Helm
  • Non K8s tools such as Terraform
  • harness.io 
  • Skaffold.dev handles the workflow for building, pushing and deploying application
  • kubediff
  • sealed-secrets by Bitnami OR Vault by Hashicorp

Gitops promoted by
* Weaveworks
* Cloudbees
* Bitnami (encrypt K8s secret) https://github.com/bitnami/sealed-secrets
* OpenFaaS
* Hasura
* Ocado
* Financial Times

Benefits

1. Cluster update are sequence of atomic transaction, that can fail or success. GitOps makes it easy so productivity increase. 
2. Git provides transaction logs for audit, rollback and teamwork
3. config repo and image repo acts as firewall. so even if CI pipeline is hacked, production environment is not hacked. 

CKA 6: Cluster Maintenance


Drain Cordon Uncordon  
If a worker node down, then K8s wait for "pod-eviction-timeout" period. Default value is 5 min
This value is passed to kube-control-manager as command line argument.

so if we can upgrade worker node within 5 min, it is ok. Else safer way is:
k drain "worker node name"

it also means the worker node is cordoned, i.e. no pod can be scheduled on it. So once it is back, we should uncorden it
k uncorden "worker node name"

K8s Version
It applies to: (1) kube-apiserver (2) controller-manager (3) kube-scheduler (4) kubelet (5) kube-proxy (6) kubectl
It does not apply to (1) etcd cluster (2) coreDNS

Cluster Upgrade
If Kube-apiserver has version X then
(1) controller-manager and (2) kube-scheduler can be with version [X - 1, X]
(1) kubelet and (2) kube-proxy can be with version [X - 2, X - 1, X]
kubectl can be with version [X - 1, X, X + 1]

* We should always upgrade one minor release at a time

kubeadm upgrade plan //Gives all the information about present version and latest available version
kubeadm upgrade apply "New version"

On master node

1. Upgrade kubeadm tool itself.
2. kubeadm upgrade plan "New version"hold hold is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed
3. kubeadm upgrade apply "New version"

k get nodes command show version of kublet, not version of kube-apiserver

3. if kubelet is present at master node then upgrade it
4. systemctl restart kubelet

on worker node

1. k drain "worker node name"
2. upgrade kubeadm
3. upgrade kubelet
4. kubeadm upgrade node config --kubelet-version "new version"
5. systemctl restart kubelet
6. k uncorden "worker node name"

OR

1. k drain "worker node name"
2. kubeadm upgrade node
3. upgrade kubelet
4. k uncorden "worker node name"

If we run k drain command for master node, then also it applies only to user applications. The K8s applications running on master node will not evacuated.

"apt-mark hold" is used to mark a package as held back, which will prevent the package from being automatically installed, upgraded or removed

Backup and Restore

1. YAML files


k get all -A -o yaml > all.yaml

OR use tools

ARK by HeptIO is now called Velero

instead of getting backup of YAML file, take back up etcd cluster. 

2. etcd cluster

2.1 etcd started with "--data-dir" option. Its default value is /var/lib/etcd

If the snapshot is copied from the data directory, there is no integrity hash and it will only restore by using --skip-hash-check.

2.2 ETCDCTL_API=3 etcdctl snapshot save snapshot.db
To restore

A. service kube-apiserver stop
B. ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --name=""  --data-dir "" --initial-cluster "" --initial-cluster-token "" --initial-advertise-peer-urls "" --data-dir=""
C. Preferably use different (1) initial-cluster-token  and (2) data-dir. change etcd configuration with these 2 new values
D. service etcd restart
E. service kube-apiserver start

We must specify (1) endpoints (2) cacert (3) cert and (4) key

In managed K8s, we may not have access to etcd. So back up of YAML file is better approach.

3. If etcd is running on a storage volume that supports backup, such as Amazon Elastic Block Store, back up etcd data by taking a snapshot of the storage volume.

Reference
https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md
https://www.youtube.com/watch?v=qRPNuT080Hk

CKA 5: Application Life Cycle Management


Init Containers

initContainers: 
It is a list
Each init container run, one at a time in sequential order. 
It is like job. Run to complete
If any one fails then pod will be restarted, unless pod has restartPolicy = Never
The init container does not support 
- lifecycle, 
- livenessProbe, 
- readinessProbe, 
- startupProbe

Pod-> activeDeadlineSeconds  and pod->container->liveness probe , prevents init containers from restarting from ever. 

Resources

Highest value among all init containers is considered for resources->request and resources->limit

For pod effective limit for resources->request and resources->limit = Highest ( Highest (init containers) , summation ( all normal containers) ) 

This effect limit is considered to schedule a pod

The resources reserved by init container(s) is not used once they are completed,  by other containers. 

If we change image of container then only that container is restarted. If we change image of init container then whole pod is restarted.

SPIFEE & SPIRE


SPIFEE Introduction 

SPIFEE standard is all about 
- How to encode SPIFEE ID into X.509 certificate
- Which field to use
- How to validate (1) X.509 certificate and (2) JWT token, when SPIFEE ID is inside. 

SPIFEE is for universal ID. SPIFEE is for how different components can trust each other in distributed system. SPIFEE was launched in KubeCon 2017. SPIFEE Federation API was main focus during 2018, 2019. 

SPIRE

SPIRE is open source project, that implements SPIFEE standards. 
1. It expose workload API
2. It is framework to manage issue of ID. 
3. It is for 'secure introduction' = 'credential zero' ='bootstrap credential' Workload authenticate itself with SPIRE agent and agent with server. 
4. trust bootstrapping 
SPIRE is stand alone. It has many custom plugins

SPIRE server
- Identity Mapping / Identity Registry. It exposes Registration API
- Node API using Node Attestation plugin +  Node Resolver plugin
- Federation API
- SVID Issuance (SPIFFE Verifiable Identity Document)
- signing key
- registry of workloads

It can have plugins like

1. Upstream CA plugin
2. Node Attestor plugin to validate node. Both SIRE agent and SPIRE server
3. Node Resolver plugin
4. Datastore plugin MySQL, SQLite 3 (default), or PostgresSQL
5. Key Manager plugin. To store private key to sign SVIDs (X.509 and JWT both)

It can be deployed as stateful set. It can have PV. In production environment, it can use DB: Pstgres or MySQL

SPIRE Agent

SPIRE Agent assign SPIFEE ID to workload and generates CSR to SPIRE server. The SPIRE server returns SPIFEE ID and trust bundle (a set of certificates to verify X.509-SVID OR public key to verify JWT). They gets transfer from SPIRE server to Node agent to workload. The private key of workload, never leave node. 

- Workload API
- Workload Attestation : Verify authenticity of caller. only SPIRE agent

It can have plugins like

1.  Multiple Workload attestor plugins 
      1.a Unix attestor (OS attestor). It use out-of-band Linux kernel to verify selector mentioned in request are genuine or not. 
      1.b K8s attestor. It communicate with kubelet. Verify it is genuine K8s workload. then ns, sa, docker image id etc. 
2. Node attestor plugin. It used bootstrap configuration. Server responds with SVID to agent. Also SPIFEE ID of node. It becomes parent ID for workload. 
3. key manager plugin. Generate and use private keys for X.509-SVID

It can be deployed as daemon set

Valid Node ID
1. cloud platform e.g. AWS Instance Identification Document IID, Azure Managed Service Identities, GCE Instance Identity Tokens
2. Private key stored at TPM = Trusted Platform Module or HSM = Hardware Security Module
3. manual verification through a joint token
4. SA token
5. etc. 

SVID (SPIFFE Verifiable Identity Document) has two format
1. X.509 certificate
2. JWT token 
 - it is susceptible to replay attacks
 - Use it when L7 proxy of L7 LB is on path. 
SPIRE supports a specific form of JWT that is specifically designed to encode SPIFFE IDs, the JWT-SVID. 

Workload registry entry fields
- Properties are called selector 
  (1) ns = namespace 
  (2) sa = service account 
  (3) docker image id 
- Parent ID can be K8s cluster name
- SPIFEE ID: Format spiffe://trust domain/workload 
- DNS Name: OR CN:
- TTL:
- Entry ID:

Usecases
1. DB Access
2. Access to cloud provider
3. identity translation, 
4. OAuth client authentication, 
5. mTLS "encryption everywhere" and 
6. workload observability.
7. Square talks about how Square uses SPIFFE and SPIRE to secure communications across hybrid infrastructure services: https://youtu.be/H5IlmYmEDKk?t=2585
8. Uber talks about integrating SPIRE with workload schedulers: https://youtu.be/H5IlmYmEDKk?t=4703
9. Tigera demonstrates how Calico, Envoy and SPIRE are used to deliver unified Layer 4 and Layer 7 authorization policies: https://youtu.be/H5IlmYmEDKk?t=7812
10. Bloomberg talks about TPM node attestation with SPIRE: https://youtu.be/30S0sKRxzjM
11. NGINX/F5 on how NGINX service mesh leverages SPIFFE and SPIRE https://youtu.be/plRkDK5xFpM

Other tools
1. Secret Stores
    1.1 Hashicorp Vault
    1.2 Square Keywhiz
2. Identity Provider
    2.1 ory.sh
    2.2 VMWare Lightwave
    2.3 WS02 Identity Serve
3. Authorization Policy Engines
    3.1 Open Policy Agent
4. Service Mesh 

In case of Istio: "Istio Node Agent" is "SPIRE Agent". The "SPIRE server" can have "Istio Node Attestor Plugin"

Reference:
https://www.youtube.com/watch?v=5m6kjzdysBI
https://www.youtube.com/watch?v=ikmxZdZRTio
https://www.youtube.com/watch?v=0LSaNrOabH4
https://www.thoughtworks.com/radar/platforms/spiffe
https://github.com/spiffe/spire/blob/master/ADOPTERS.md
https://www.youtube.com/watch?v=OHiPsqT1gcI

Replication Controller


Replication Controller

replicationcontrollerMetadata attributes:


  • clusterName: The name of the cluster which the object belongs to.
  • clusterName->namespace->name
  • deletionGracePeriodSeconds: Seconds allowed for gracefully terminate before forcefully removed from the system.
  • deletionTimestamp: RFC 3339 date and time at which this resource will be deleted. Is not directly settable by a client.
  • finalizers: Must be empty before the object is deleted from the registry.
  • generateName: Optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided.
  • generation: A sequence number representing a specific generation of the desired state. Populated by the system. Read-only.
  • initializers: An initializer is a controller which enforces some system invariant at object creation time.
  • labels: Map of string keys and values that can be used to organize and categorize (scope and select) objects.
  • ownerReferences: List of objects depended by this object.

replicationcontrollerSpec attributes:


  • minReadySeconds: Minimum number of seconds for which a newly created pod should be ready without any of its container crashing, for it to be considered available. Defaults to 0


By default, deployment will add a pod-template-hash to the name of RS that it creates.