Managing Storage with Containers
https://www.meetup.com/Docker-Bangalore/events/253542738/
Flex Volume Drivers in Kubernetes and CSI, Peeyush Gupta, IBM
=============================================================
Storage in container
- stateless / stateful
- volumes
- dynamic provisioning
- PVC, PV and Storage Class
* Storage Class refers to dynamic provisioning
* PVC refers to Storage Class
* POD refers to PVC for volume.
Kubelets running on host, request Flex Driver. Flex Driver implements vendor specific APIs for storage/volume : 1. Mount 2. Unmount 3. attach 4. detach.
This binary need to be placed(copied) at specific path for each POD.
For CNI also cadico driver need to be placed(copied) at specific path for each POD. The better alternative is CSI = Container Storage Interface.
CO = Container Orachstrator. Example: Kubernetes (K8S), MESOS, Cloud Foundray, OpenShift (by RedHat).
CO has 1. node 2. controller 3. identity
There is a single binary for node and controller. Based on Identity either node or controller role can be played.
1. Indemotent APIs
2. Sidecar container
2.1 Driver register (Identity Service)
2.2 Extended Provisoner (watch create volume)
2.3 Extended Attacher (watch attach)
2.4 Liveness Probe
2.5 Extended snapshotter (started in mid July 2018)
Containerized Gluster Storage in Kubernetes - Saravanakumar, Red Hat
====================================================================
GlusterFS was born by oil industry. Oil industry need to process data from different hosts to detect presence of oil. Now it is more than 10 years old.
Steps (all steps as sudo)
1. install and start glusterd service on all host.
2. gluster peer status
3. gluster volume create
4. gluster volume start
This will start gluster on all host.
5. gluster volume status
6. mount -t glusterfs
PVC access mode
1. ROX: Read only by many nodes
2. RWO: Read/Write by single node
3. RWX: Read/Write by many nodes
Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes
Storage requirements for running Spark workloads on Kubernetes, Rachit Arora
============================================================================
Spark core engine runs over 1. Yarn, 2. MESOS, 3. Standalone Schedular, 4. K8S
1. Spark SQL 2. Spark Streaming 3. Spark Machine Learning Lib 4. GraphX runs over Spark Core Engine
* Data engineer 1. Ingest and store data from multiple source 2. Prepare Data.
* Data Scientis 2. Prepare Data 3. Analysze Data build model
* Application Developer 4. Visulize Data
Now new trend is serverless analytics.
'Spark over K8S' provides Jupyter-Kernel gateway for data scientist to analyze data
Distributed FS
1. NFS and BigNFS
2. HDFS
3. DBFS (Data Briks FS)
4. S3 / Object Storage
5. Portworx
6. GlusterFS
URLs:
datascience.ibm.com
www.ibm.com/analytics/us/en/watson-data-platform/tutorial
Tweeter Handle : @k8sBLR
Flex Volume Drivers in Kubernetes and CSI, Peeyush Gupta, IBM
=============================================================
Storage in container
- stateless / stateful
- volumes
- dynamic provisioning
- PVC, PV and Storage Class
* Storage Class refers to dynamic provisioning
* PVC refers to Storage Class
* POD refers to PVC for volume.
Kubelets running on host, request Flex Driver. Flex Driver implements vendor specific APIs for storage/volume : 1. Mount 2. Unmount 3. attach 4. detach.
This binary need to be placed(copied) at specific path for each POD.
For CNI also cadico driver need to be placed(copied) at specific path for each POD. The better alternative is CSI = Container Storage Interface.
CO = Container Orachstrator. Example: Kubernetes (K8S), MESOS, Cloud Foundray, OpenShift (by RedHat).
CO has 1. node 2. controller 3. identity
There is a single binary for node and controller. Based on Identity either node or controller role can be played.
1. Indemotent APIs
2. Sidecar container
2.1 Driver register (Identity Service)
2.2 Extended Provisoner (watch create volume)
2.3 Extended Attacher (watch attach)
2.4 Liveness Probe
2.5 Extended snapshotter (started in mid July 2018)
Containerized Gluster Storage in Kubernetes - Saravanakumar, Red Hat
====================================================================
GlusterFS was born by oil industry. Oil industry need to process data from different hosts to detect presence of oil. Now it is more than 10 years old.
Steps (all steps as sudo)
1. install and start glusterd service on all host.
2. gluster peer status
3. gluster volume create
4. gluster volume start
This will start gluster on all host.
5. gluster volume status
6. mount -t glusterfs
PVC access mode
1. ROX: Read only by many nodes
2. RWO: Read/Write by single node
3. RWX: Read/Write by many nodes
Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes
Storage requirements for running Spark workloads on Kubernetes, Rachit Arora
============================================================================
Spark core engine runs over 1. Yarn, 2. MESOS, 3. Standalone Schedular, 4. K8S
1. Spark SQL 2. Spark Streaming 3. Spark Machine Learning Lib 4. GraphX runs over Spark Core Engine
* Data engineer 1. Ingest and store data from multiple source 2. Prepare Data.
* Data Scientis 2. Prepare Data 3. Analysze Data build model
* Application Developer 4. Visulize Data
Now new trend is serverless analytics.
'Spark over K8S' provides Jupyter-Kernel gateway for data scientist to analyze data
Distributed FS
1. NFS and BigNFS
2. HDFS
3. DBFS (Data Briks FS)
4. S3 / Object Storage
5. Portworx
6. GlusterFS
URLs:
datascience.ibm.com
www.ibm.com/analytics/us/en/watson-data-platform/tutorial
Tweeter Handle : @k8sBLR
0 comments:
Post a Comment