K8s Operator


Introduction

An Operator is a method of 

- packaging, 
- deploying and 
- managing 
a Kubernetes application

Here a Kubernetes application is an application that is both 

- deployed on Kubernetes and 
- managed by Kubernetes
using the Kubernetes APIs and kubectl tooling.

* Operators are like runtime for K8s application


* Operators are more relevant for stateful applications. like

- databses
- caches
- Machine Learning
- Analytics
- monitoring systems

* Application native/specific. Operator is application specific controller, that extends k8s API for 

- packaging (Helm can use to package multiple k8s apps, that will be deployed and managed by their respective operator). 
- installation / deployment on k8s cluster
- run
- configure / re-configure
- updates
- upgrades
- rollback, in case of failure
- recover from fault
- backup
- Restore
- scaling
- Monitoring
self-service provisioning capabilities,
- more complex automation
of application. (particularly stateful applications). Operator does all these without data loss and without application unavailability. 

* Operator is like = Custom Resource Definition CRD + its controller. Here the custom resource is application instance. CRD adds new API for the application instance. Controller adds new behavior. 


* Operator is like = A set of cohesive APIs to extend K8s functionalities. This code does not go back to k8s repository. 


* operator is issued by coreos and the controller is used in Kubernetes. An Operator is an application-specific controller 

Kubernetes Operators are processes connecting to the master API and watching for events, typically on a limited number of resource types.
When a relevant event occurs, the operator reacts and performs a specific action. This may be limited to interacting with the master API only, but will often involve performing some action on some other systems (this could be either in cluster or off cluster resources). * CRD is nativae way of explaining k8s object.

* The kubectl commands work with CRD objects. 

* K8s manages K8s objects. With operator, k8s manage multiple application instances across multiple clusters. 


An operator is a way of building an application and driving an application on top of Kubernetes, behind Kubernetes APIs. Operator is more than K8s primitives like StateFulSet and PVC. Operator uses ReplicaSet, Services, DaemonSets, Deployments


Operator takes human operational knowledge (Read : DevOps Team's  knowledge  OR Site Reliability Engineer's knowledge about application lifecycle) and put it into software to easily package and share with consumers. This operational knowledge is embedded in CRD itself. 


* Operator is like a Controller that uses a CRD to encapsulate operational knowledge for a specific application

* As per Extending K8s article, K8s has multiple extension points as below figure


1. kubectl plugin
2. API access extension
3. CR
4. Scheduler extension
5. Controller. Controller often uses CR
6. Network plugin (CNI Calico)
7. Storage plugin (CSI Portwoks)

We should add Runtimeclass also. 

Operator is (3) CR + (5) controller 

* Here controllers have direct access to K8s API. So they can monitor the cluster, change pods/services, scale up/down and call endpoints of the running applications, all according to custom rules written inside those controllers.


* a custom controller acts and listens on native Kubernetes resources and events, an Operator works with Custom Resource Definitions (CRDs), resources that the user creates to solve complex business problems.

* Out of 4 najor modules of controller code generator (1) informer-gen and (2) list-gen modules are also called operator. They build basis of custom controller. 


* A powerful feature of an Operator is : 
to enable the desired state to be reflected in your application. 
As the application gets more complicated with more components, detecting these changes can be harder for a human, but easier for a computer.

* One can use existing tools to write your own operator. (Kubebuilder and Operator - SDK focus more on reconcile loop of custom controller)

1. KUDO (Kubernetes Universal Declarative Operator)
https://kudo.dev/
https://github.com/kudobuilder/operators

2. kubebuilder
https://book.kubebuilder.io/
https://github.com/operatify/operatify
Kubebuild is by Google

3. Metacontroller along with WebHooks that you implement yourself

https://metacontroller.app/

4. Operator Framework OR Operator - SDK
Operator Framework is by RedHat OpenShift (now IBM)

4.1. Operator SDK: Set of tools to accelerate the development of an Operator. 

Here is CLI guide https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md
Here is user guide : https://github.com/operator-framework/operator-sdk/blob/master/doc/user-guide.md

We can run our operator, outside the k8s cluster also as pretest. 

4.2. Operator Life Cycle Management
4.3. Operator Metering 

Here is K8s core API to write custom code for operator

Operator can be developed using

- GoLang
- Ansible Playbooks
- Helm Charts. It is more specific to stateless K8s application. Here deeper inspection to Pods, Services, Deployments and ConfigMaps will not happen

Types of Custom Resource Definition CRD

1. primary resources
2. secondary resources

* For ReplicaSet, primary resource is ReplicaSet itself docker image and secondary resource is pod

* For DaemonSet, primary resource is DaemonSet itself and secondary resource is pod+node
* For PodSet, primary resource is PodSet itself and secondary resource is pod.

PodSet owns the pod with controllerutil.SetControllerReference()


PodSet will be developed, that is similar to ReplicaSet. 


Controller


Controller works like this pseudo code. 


while True {
  values = check_real_values()
  tune_system(values, expected_state)
}

Controller is Informer + downstream store
Another such agent is SharedInformer. It creates shared cache of the state for multiple request. 





Reconsile() is major function of controller. K8s without operator simply create new pod. Controller does application specific code in addition to pod creation. 


Code Generator
- deepcopy-gen 
- client-gen
- informer-gen
- lister-gen
- conversion-gen
- defaulter-gen
- register-gen
- set-gen

gen-go  and CRD-code-generation are also another tool to generate code. 

Last two are also called operator. They are basis for building custom controller. 

Notification loop declare and describe desired state. Controller will find out how to reach there. 

Operator Examples


Good starting point: https://www.magalix.com/blog/creating-custom-kubernetes-operators


CoreOS provides sample operators as reference like (1) etcd Here is blog (2) Prometheus. Here is blog. K8s exports all of its internal metrics in the native Prometheus format. Here two third party resources (TPRs) are defined: Prometheus and ServiceMonitor.

Demo controller

Sample controller 
netperf operator

https://github.com/hrishin/podset-operator

Build replicaset kind of k8s object. Build controller from scratch to demonstrate  KuberBuilder or Operator SDK frameworks

There are many operators available. Here are the lists


https://github.com/operator-framework/awesome-operators

https://operatorhub.io/ one needs to install (1) minikube and (2) Operator Lifecycle Manger, to install any operator from Operator Hub. 
https://commons.openshift.org/sig/operators.html
https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-building-kubernetes-operators-and-stateful-apps
https://github.com/operator-framework/community-operators/tree/master/community-operators
https://github.com/operator-framework/community-operators/tree/master/community-operators/3scale-community-operator/0.4.0
Reference


https://slideplayer.com/slide/17861996/#.YVSYqHKnIyU.linkedin
https://coreos.com/operators/

https://coreos.com/blog/introducing-operator-framework
https://coreos.com/blog/introducing-operators.html
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/
https://medium.com/devopslinks/writing-your-first-kubernetes-operator-8f3df4453234
https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-building-kubernetes-operators-and-stateful-apps
https://enterprisersproject.com/article/2019/2/kubernetes-operators-plain-english
https://linuxhint.com/kubernetes_operator/
https://blog.couchbase.com/kubernetes-operators-game-changer/
https://github.com/cncf/tag-app-delivery/blob/main/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md


Reference for building controller


https://medium.com/@cloudark/kubernetes-custom-controllers-b6c7d0668fdf

https://medium.com/speechmatics/how-to-write-kubernetes-custom-controllers-in-go-8014c4a04235
https://github.com/piontec/k8s-demo-controller
https://github.com/kubernetes/sample-controller
https://blog.openshift.com/kubernetes-deep-dive-code-generation-customresources/
https://github.com/operator-framework/getting-started
https://www.linux.com/blog/learn/2018/12/demystifying-kubernetes-operators-operator-sdk-part-2

https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
https://blog.openshift.com/kubernetes-deep-dive-code-generation-customresources/
https://blog.openshift.com/make-a-kubernetes-operator-in-15-minutes-with-helm/
https://github.com/operator-framework/helm-app-operator-kit
https://www.tailored.cloud/kubernetes/write-a-kubernetes-controller-operator-sdk/
https://itnext.io/how-to-create-a-kubernetes-custom-controller-using-client-go-f36a7a7536cc

23rd Oct 2020

RedHat's Operator Framework is now part of CNCF
distroless is good for creating container image with only our distribution https://github.com/GoogleContainerTools/distroless

Python based K8s operator


0 comments:

Post a Comment