My DIKSHAA Guru
DIKSHAA means initiation. One can not grasp the spiritual knowledge using intelligence. The God, or the consciousness or the BHRAMAN is beyond intelligence. It cannot be proved or disproved with arguments. The Guru make us enter into this realm, that is beyond intelligence by means of DIKSHAA. It is a traditional ceremony, rituals, where Guru gives a secret and sacred MANTRA to the disciple.
Today let me share an old photograph of my DIKSHA Guru, who initiate me with BHRAM-SAMBANDH may be around year 1993.
Today let me share an old photograph of my DIKSHA Guru, who initiate me with BHRAM-SAMBANDH may be around year 1993.
Pu. Go. Shri. Vallabharaiji Bava Shri
(Jamnagar - Chopasani)
Bengaluru Tech Summit : Day 1
Damiel Manuel talked about Cyber Security. In this world, millions and billions of people are active on various social media like Facebook, WhatsApp, Instagram, YouTube, LinkedIn etc. It rises "deep fake" video threat. The video footage that appears to be real but is produced by technology. Now, we can no longer believe our eyes. NIST is a cyber security framework. Link 1 and Link 2 Cyber security is also a layered architecture. Hardware and network security is at physical layer. Application/software security at logical layer. Data security includes protection of personal information, location, accessibility, integrity of data etc. Social aspects covers people, culture and processes. it needs more attention. Cyber security topic now moves to Cyber resilience
Interesting panel discussions about AI/ML.
Ajay Sharma : Machine used to crunch numbers and humans make decision. Now machines are capable of doing both. Use cases (1) use Intel Mobileye to prevent, predict accident zones. (2) CheXNeXt is deep learning for chest radiograph diagnosis. Discussion about Niti Ayog #AIforAll ICTAI (International Center for Transformative AI) etc.
Raj Cherubal: Smart City solutions : (1) Parking management system (2) public bicycle sharing (3) restoration of water bodies (ponds) (4) solar panel on government buildings (5) water meter (6) smart class (7) disaster management system etc. IUDX (Indian Urben Data Exchange) Link Whitepaper Link
Aksah Ravi (gnani) talked about NLP based voice assistant products that supports Hindi, Kannada etc.
Out of all AI/ML projects hardly 13 to 14 % projects are successful and only 8% of customer are happy. Some projects need their ML model to be updated every week, even every day. AI/ML business case is all about identifying and satisfying unmet and unmentioned need. Few challenges (1) AI/ML implementation needs cultural change, where man and machine both work together. (2) upskill of existing staff (3) Data is fuel for AI. More time goes to find out which data to use rather than building AI/ML model. Noise in data can cause bias. The chatbot may learn offensive language over time. (4) explain-ability of AI/ML outcome. (5) Robust governance is also important. At present, chatbots are becoming popular. Long way to go. Information architecture (IA) is must for AI/ML projects. IA is the structural design of shared information environments.
Queryplex and federation are effective tools from IBM to connect multiple data sources. AI OpenScale is another tool from IBM
Ashok Ayengar talked about shopx - buy and sale platform.
Last 25 years of history big enterprise adopted new technologies like ERP and small stratup companies built eco-system around it. Many fortune 500 companies have huge legacy in range of 20 to 100 years. They need machine learning even on end-of-life servers.
The collaboration at Bangalore is not seen even in developed country. Here we have hackathon, open source contribution, discussion at physical meetup etc. It is really proud to be in Bangalore.
The day 1 ended with thought leaders conclave on "TechScape 2035"
The following trends are emerged.
1. Digitalization
2. Rapid Urbanization
3. Customer does not want product, but wants solution
4. so more and more collaborations
The growth of Internet was not predicated.
1. Growth with many IoT devices
2. Growth for amusement e.g. Netflix
3. Many networks E.g. Car has many networks
During 90s, when Internet based innovative services and business was planned, no one predicated that everyone (4 billion people) will have smart phone. Now cost of computing come down. Technology is democratized. This growth will be accelerated further, as everything (e.g. speaker, light etc.) will have compute power, connectivity, intelligent (AI/ML). The enormous data is generated by our wearable. Evolution will happen in laboratory and new human being will be created. Excitement with fear.
Now the smart software makes diagnosis smarter than doctor. In next 5 to 10 years, it will be mandatory to take second opinion from machines. People will get incentive for not being sick by health insurance and hospitals. The Indian doctors and nurses are best in the world. This is a license driven industry.
Now in automobile people need : 0 emission, 0 downtime, 0 accident, and first solution is electric car. Electricity is alternate fuel. The omnipresent 3D printing will eliminate need to keep inventory of spare parts of cars. The digitization, GST have improved logistics a lot. We need more efficiency steps like: spare trailers will just replace the loaded one to save time.
All these trends will empower the bottom of pyramid, surprisingly.
In future, there is no concept of firm. All companies are becoming platform. Apple started as platform for iOS apps. People starts company just on WhatsApp group. Now we will see rapid consolidation of firms. So monopoly will be back.
The present generation is Digital Native and generation Z Their working style is completely different.
Interesting panel discussions about AI/ML.
Ajay Sharma : Machine used to crunch numbers and humans make decision. Now machines are capable of doing both. Use cases (1) use Intel Mobileye to prevent, predict accident zones. (2) CheXNeXt is deep learning for chest radiograph diagnosis. Discussion about Niti Ayog #AIforAll ICTAI (International Center for Transformative AI) etc.
Raj Cherubal: Smart City solutions : (1) Parking management system (2) public bicycle sharing (3) restoration of water bodies (ponds) (4) solar panel on government buildings (5) water meter (6) smart class (7) disaster management system etc. IUDX (Indian Urben Data Exchange) Link Whitepaper Link
Aksah Ravi (gnani) talked about NLP based voice assistant products that supports Hindi, Kannada etc.
Out of all AI/ML projects hardly 13 to 14 % projects are successful and only 8% of customer are happy. Some projects need their ML model to be updated every week, even every day. AI/ML business case is all about identifying and satisfying unmet and unmentioned need. Few challenges (1) AI/ML implementation needs cultural change, where man and machine both work together. (2) upskill of existing staff (3) Data is fuel for AI. More time goes to find out which data to use rather than building AI/ML model. Noise in data can cause bias. The chatbot may learn offensive language over time. (4) explain-ability of AI/ML outcome. (5) Robust governance is also important. At present, chatbots are becoming popular. Long way to go. Information architecture (IA) is must for AI/ML projects. IA is the structural design of shared information environments.
Queryplex and federation are effective tools from IBM to connect multiple data sources. AI OpenScale is another tool from IBM
Ashok Ayengar talked about shopx - buy and sale platform.
Last 25 years of history big enterprise adopted new technologies like ERP and small stratup companies built eco-system around it. Many fortune 500 companies have huge legacy in range of 20 to 100 years. They need machine learning even on end-of-life servers.
The collaboration at Bangalore is not seen even in developed country. Here we have hackathon, open source contribution, discussion at physical meetup etc. It is really proud to be in Bangalore.
The day 1 ended with thought leaders conclave on "TechScape 2035"
The following trends are emerged.
1. Digitalization
2. Rapid Urbanization
3. Customer does not want product, but wants solution
4. so more and more collaborations
The growth of Internet was not predicated.
1. Growth with many IoT devices
2. Growth for amusement e.g. Netflix
3. Many networks E.g. Car has many networks
During 90s, when Internet based innovative services and business was planned, no one predicated that everyone (4 billion people) will have smart phone. Now cost of computing come down. Technology is democratized. This growth will be accelerated further, as everything (e.g. speaker, light etc.) will have compute power, connectivity, intelligent (AI/ML). The enormous data is generated by our wearable. Evolution will happen in laboratory and new human being will be created. Excitement with fear.
Now the smart software makes diagnosis smarter than doctor. In next 5 to 10 years, it will be mandatory to take second opinion from machines. People will get incentive for not being sick by health insurance and hospitals. The Indian doctors and nurses are best in the world. This is a license driven industry.
Now in automobile people need : 0 emission, 0 downtime, 0 accident, and first solution is electric car. Electricity is alternate fuel. The omnipresent 3D printing will eliminate need to keep inventory of spare parts of cars. The digitization, GST have improved logistics a lot. We need more efficiency steps like: spare trailers will just replace the loaded one to save time.
All these trends will empower the bottom of pyramid, surprisingly.
In future, there is no concept of firm. All companies are becoming platform. Apple started as platform for iOS apps. People starts company just on WhatsApp group. Now we will see rapid consolidation of firms. So monopoly will be back.
The present generation is Digital Native and generation Z Their working style is completely different.
Continuous delivery : 2nd DevOps Pillar
CI = build and UT after code checkin
Six Principle
2.1.1 Build less than 5 min
2.1.2 Commit small bits
2.1.3 Don't leave the build broken
2.1.4 Trunk based development flow
2.1.5 Don't allow flaky test. Fix them
2.1.6 build should return status, log and artifact
CDelivery = CI + deploy at production like environment (staging) + automate integration testing + automate acceptance testing + smoke testing
Principles
2.2.1 Build artifact once
2.2.2 Artifact should be immutable
2.2.3 Deployment to production like environment = load balancers + n/w settings + security controls + data that matches production
2.2.4 stop if previous step fails
2.2.5 Deployment should be idempotent
CDeployment = CDelivery + Full automated testing + Deployed to production
For higher performances teams
* Less than 3 branch (ideally 1 branch)
* Their life span is less than a day and then merge to trunk.
It is not about how much you can deliver but it is about how little you can
CDelivery the build is not always release to production environment.
CDeployment, the build is always release to production environment.
Tools
Build Systems : Jenkins, Bamboo, Teamcity etc
Online build : Circle CI, Travis CI
Monitoring tool: Datadog
./hooks/git-pre-commit.hooks
webhooks : Notify external service with POST request about changes in repository.
Jenkins :
* We can add, remove steps in build "configure"
* We can add/remove plugins for different languages : C, Go etc
* We can configure those plugins using global tools configuration.
* Jenkins plugin repository https://plugins.jenkins.io/ We can add custom plugins. Plugins to notify on slack about build failure. Based on installed plugin we can get options in build
* Pipeline feature converts GUI to "build as code".
* BUILD_NUMBER is env. variable. We can add this number as tag and commit back to git repo for backward compatibility.
Test terms
* Shift Left : test at developer's machine by developer. Left side in L to R pipeline
* Test Fixture : e.g. sample data, Linux server configuration. They are also artifacts.
Test Philosophy
TDD: first right negative TCs
BDD: Tool Cucumber
ATD : Acceptance Test Driven
KDD: Keyword driven testing
Test Split
70 % Unit Testing
20 % Integration Testing
10 % E2E Testing
JQ is good tool to deal with JSON data
YQ converts YAML to JSON.
REST API Markup Language
Tools: Swagger, RAML ; abao, Mocha, chai are also relevant tools
GUI Testing
1. Selenium
2. Robot f/w
2.1 Selenium2Library is plugin for Robot f/w, that connects both.
2.2 PageObject design pattern
Dynamic Security Testing
1. Gauntlt : Given, When, Then conventions
2. arachni
3. nmap
4. Fortify
5. Vericode
6. Brakeman
7. Bundler
8. Audit
9. Retire.js
Reference
https://testing.googleblog.com
Crazy Fast Build: https://www.infoq.com/presentations/Crazy-Fast-Build-Times-or-When-10-Seconds-Starts-to-Make-You-Nervous
http://dan.bodar.com/2012/02/28/crazy-fast-build-times-or-when-10-seconds-starts-to-make-you-nervous/
Six Principle
2.1.1 Build less than 5 min
2.1.2 Commit small bits
2.1.3 Don't leave the build broken
2.1.4 Trunk based development flow
2.1.5 Don't allow flaky test. Fix them
2.1.6 build should return status, log and artifact
CDelivery = CI + deploy at production like environment (staging) + automate integration testing + automate acceptance testing + smoke testing
Principles
2.2.1 Build artifact once
2.2.2 Artifact should be immutable
2.2.3 Deployment to production like environment = load balancers + n/w settings + security controls + data that matches production
2.2.4 stop if previous step fails
2.2.5 Deployment should be idempotent
CDeployment = CDelivery + Full automated testing + Deployed to production
For higher performances teams
* Less than 3 branch (ideally 1 branch)
* Their life span is less than a day and then merge to trunk.
It is not about how much you can deliver but it is about how little you can
CDelivery the build is not always release to production environment.
CDeployment, the build is always release to production environment.
Tools
Build Systems : Jenkins, Bamboo, Teamcity etc
Online build : Circle CI, Travis CI
Monitoring tool: Datadog
./hooks/git-pre-commit.hooks
webhooks : Notify external service with POST request about changes in repository.
Jenkins :
* We can add, remove steps in build "configure"
* We can add/remove plugins for different languages : C, Go etc
* We can configure those plugins using global tools configuration.
* Jenkins plugin repository https://plugins.jenkins.io/ We can add custom plugins. Plugins to notify on slack about build failure. Based on installed plugin we can get options in build
* Pipeline feature converts GUI to "build as code".
* BUILD_NUMBER is env. variable. We can add this number as tag and commit back to git repo for backward compatibility.
Test terms
* Shift Left : test at developer's machine by developer. Left side in L to R pipeline
* Test Fixture : e.g. sample data, Linux server configuration. They are also artifacts.
Test Philosophy
TDD: first right negative TCs
BDD: Tool Cucumber
ATD : Acceptance Test Driven
KDD: Keyword driven testing
Test Split
70 % Unit Testing
20 % Integration Testing
10 % E2E Testing
JQ is good tool to deal with JSON data
YQ converts YAML to JSON.
REST API Markup Language
Tools: Swagger, RAML ; abao, Mocha, chai are also relevant tools
GUI Testing
1. Selenium
2. Robot f/w
2.1 Selenium2Library is plugin for Robot f/w, that connects both.
2.2 PageObject design pattern
Dynamic Security Testing
1. Gauntlt : Given, When, Then conventions
2. arachni
3. nmap
4. Fortify
5. Vericode
6. Brakeman
7. Bundler
8. Audit
9. Retire.js
Reference
https://testing.googleblog.com
Crazy Fast Build: https://www.infoq.com/presentations/Crazy-Fast-Build-Times-or-When-10-Seconds-Starts-to-Make-You-Nervous
http://dan.bodar.com/2012/02/28/crazy-fast-build-times-or-when-10-seconds-starts-to-make-you-nervous/
Infrastructure automation : 1st DevOps Pillar
Operation team should use source control
Define your infrastructure as model or drive it using RESTful API
Model OR Recipes = Hardware + OS(Operating System) + OS Dependencies + System Configuration + Accounts + SSL Certs + Application
Important terms
* Imperative (procedural) Approach where commands desired to produce a state are defined and then executed. E.g. BASH Shell script, AWS CLI, Python Boto library
* declarative (functional) Approach where you define the desired state, and the tool converges the existing system on the model. E.g. Makefile, Puppet manifest, SQL Queries.
declarative is more efficient.
* Convergent : CM automation that converges the system to desired state.
* Idempotent : The ability to execute the CM procedure repeatedly. And end up in the same state each time. The second run will not make any change.
* Immutable : Not change after deployment. Can be redeploy if needed. E.g. Docker container.
* Self service : The ability for an end user to kick off one of these processes without having to go through other people
http://www.infrastructures.org/
Configuration Management
* Provisioning : Server ready with hardware, OS, system services, network
Two types of provisioning : Model based provisioning and Convergent Driver Provisioning
1. Model based provisioning
1.1 declarative model: n/w, node, db etc. E.g. using template YAML
or imperative manner, step by step
Tools
* ColudFormation JSON file for AWS
* Azure Resource Manager template for Azure
* HashiCorp's Terraform for multiple cloud providers. It maintains external state store
* Ubuntu's Juju = Infra + services (common in docker)
** Orchestration : Coordinated operations across multiple systems
Provisioners and providers are building blocks of Vagrant.
1. Vagrant has provisioners like Puppe, Chef, Ansible etc.
2. Vagrant has Providers like VirtualBox, Hyper-V, Docker. VMWare and AWS via plugin.
vagrantfile
commands
vagrant destroy
vagrant status
2. Convergent Driver Provisioning.
It is declarative and idempotent
It handles orchestration by state tracking and master registry. Each node reports system information to the registry.
Chef components
1. Run-list
2. Roles
3. Recipes belongs to cookbook.
Cookbook public repository (Chef): https://supermarket.chef.io/
Puppet public repository : Puppet Forge https://forge.puppet.com/
purpose of Ohai is to provide node data to Chef-client, to use within cookbooks.
commands
kitchen converge
kitchen list
kitchen destroy
Tools
Chef
Lint: Foodcritic
Unit Testing : Chefspec
Integration Testing : KitchenCI
Puppet
Ansible
Salt
CFEngine
Packer
Shell Provisioner
Evolution of provisioning tools
Adhoc script OR Capistrano framework by developers to push changes.
Provisioning Tools: CFEngine to Puppet to Chef ,
Commercial IT Provisioning tool : Ghost,
Enterprise tool : HP
Then Golden Image or Foil Ball : This result in (1) image sprawl or (2) configuration drift. So stem cell system approach with minimum initial provisioning and update later on : Chef DSL,
Capistrano to RunDeck
Rundeck is a container, which listen to on 4440
docker run -p 4440:4440 -e SERVER_URL=http://localhost:4440 --name AnyName -t /jordan/rundeck:latest
Orchestrated deployment techniques
* Canary Deployment
* Blue / Green Deployment
* Cluster immune system deployment
* Immutable deployment (1) for Container = OS + Dependency + App code (2) not for data store
Artifact :
JAR to WAR to
-Deb
-RPM
to
- Amazon Machine Image (AMI for amazon)
- VM image (using packer)
- Docker image (using docker file)
Maven's jdub plugins: for Debian image
FPM : deb and RPM package etc. format
Artifact Repository
* Nexus
* JFrog Artifactory
* Apache Archiva
Testing:
- Unit (includes formatters and linters)
- Integration
- Security
We can include tools related to formatter and linter too as part of CUT (coding and Unit Testing) phase for developer.
Formatter : Rubocop. It is for Ruby. Chef and Puppet are on top of Ruby
Linter: Foodcritic for Chef
Unit Test : ChefSpec is extension of RSpec framework (for Ruby)
Integration Test : Serverspec (written in RSpec). Before that we run TestKitchen (kitchen converge) then Serverspec (kitchen verify)
Security Test: InSpec is audit based tool from Chef (kitchen verify) We can get (1) CIS hardening compliance and (2) visual reports. Another tool Gauntlt perform cyber attack at system
CMDB :
* etcd
* ZooKeepr
* Consul
Container As a Services
* Rancher
* Google Cloud Platform
* AWS ECS
Habitat : Application automation = Build + Deploy. It extends Chef.
Usecase
Just Recommendations:
1. Serverless : Basic needs.
2. container, docker, k8s : for medium complex system
3. Chef or Puppet: large distributed environment, network with heterogeneous systems and variety of apps (convergence model)
4. Ansible Orchestration: Website with high uptime requirement
5. Rundeck with Puppet: Update with coordination with other services.
Cloud
SaaS : TurboTax, Office365 Salesforce
PaaS: Google App Engine, Microsoft Azure App Service, Pivotal Web Service
IaaS: Google Compute Engine, Microsoft Azure VM, AWS EC2
FaaS : Amazon Lambda, Google Cloud Function, Azure Function
Hypervisor : VMWare, ZeN, Hyper-V, VirtualBox
Tools for cloud:
1. Hashicorp's Packer to create images
2. OpenStack
3. CloudFoundry by Pivotal
4. Open Source tools : MAAS (for Ubuntu's Juju), cobbler, Hanlon, helps in Pixie n/w booting
5. Then provisioning by Chef, Puppet, VMWare vCenter
6. Foreman is based on Puppet framework
7. Crowbar (Now Zeus cloud) : OpenStack and Hadoop installation
8. digital_REBAR : Provisioning + Orchestration
Define your infrastructure as model or drive it using RESTful API
Model OR Recipes = Hardware + OS(Operating System) + OS Dependencies + System Configuration + Accounts + SSL Certs + Application
Important terms
* Imperative (procedural) Approach where commands desired to produce a state are defined and then executed. E.g. BASH Shell script, AWS CLI, Python Boto library
* declarative (functional) Approach where you define the desired state, and the tool converges the existing system on the model. E.g. Makefile, Puppet manifest, SQL Queries.
declarative is more efficient.
* Convergent : CM automation that converges the system to desired state.
* Idempotent : The ability to execute the CM procedure repeatedly. And end up in the same state each time. The second run will not make any change.
* Immutable : Not change after deployment. Can be redeploy if needed. E.g. Docker container.
* Self service : The ability for an end user to kick off one of these processes without having to go through other people
http://www.infrastructures.org/
Configuration Management
* Provisioning : Server ready with hardware, OS, system services, network
Two types of provisioning : Model based provisioning and Convergent Driver Provisioning
1. Model based provisioning
1.1 declarative model: n/w, node, db etc. E.g. using template YAML
or imperative manner, step by step
Tools
* ColudFormation JSON file for AWS
* Azure Resource Manager template for Azure
* HashiCorp's Terraform for multiple cloud providers. It maintains external state store
* Ubuntu's Juju = Infra + services (common in docker)
* Vagrant tool emulate production environment.
** Deployment : Automatic deploy / upgrade application on server** Orchestration : Coordinated operations across multiple systems
Provisioners and providers are building blocks of Vagrant.
1. Vagrant has provisioners like Puppe, Chef, Ansible etc.
2. Vagrant has Providers like VirtualBox, Hyper-V, Docker. VMWare and AWS via plugin.
vagrantfile
commands
vagrant destroy
vagrant status
2. Convergent Driver Provisioning.
It is declarative and idempotent
It handles orchestration by state tracking and master registry. Each node reports system information to the registry.
Chef components
1. Run-list
2. Roles
3. Recipes belongs to cookbook.
Cookbook public repository (Chef): https://supermarket.chef.io/
Puppet public repository : Puppet Forge https://forge.puppet.com/
purpose of Ohai is to provide node data to Chef-client, to use within cookbooks.
commands
kitchen converge
kitchen list
kitchen destroy
Tools
Chef
Lint: Foodcritic
Unit Testing : Chefspec
Integration Testing : KitchenCI
Puppet
Ansible
Salt
CFEngine
Packer
Shell Provisioner
Evolution of provisioning tools
Adhoc script OR Capistrano framework by developers to push changes.
Provisioning Tools: CFEngine to Puppet to Chef ,
Commercial IT Provisioning tool : Ghost,
Enterprise tool : HP
Then Golden Image or Foil Ball : This result in (1) image sprawl or (2) configuration drift. So stem cell system approach with minimum initial provisioning and update later on : Chef DSL,
Capistrano to RunDeck
Rundeck is a container, which listen to on 4440
docker run -p 4440:4440 -e SERVER_URL=http://localhost:4440 --name AnyName -t /jordan/rundeck:latest
Orchestrated deployment techniques
* Canary Deployment
* Blue / Green Deployment
* Cluster immune system deployment
* Immutable deployment (1) for Container = OS + Dependency + App code (2) not for data store
Artifact :
JAR to WAR to
-Deb
-RPM
to
- Amazon Machine Image (AMI for amazon)
- VM image (using packer)
- Docker image (using docker file)
Maven's jdub plugins: for Debian image
FPM : deb and RPM package etc. format
Artifact Repository
* Nexus
* JFrog Artifactory
* Apache Archiva
Testing:
- Unit (includes formatters and linters)
- Integration
- Security
We can include tools related to formatter and linter too as part of CUT (coding and Unit Testing) phase for developer.
Formatter : Rubocop. It is for Ruby. Chef and Puppet are on top of Ruby
Linter: Foodcritic for Chef
Unit Test : ChefSpec is extension of RSpec framework (for Ruby)
Integration Test : Serverspec (written in RSpec). Before that we run TestKitchen (kitchen converge) then Serverspec (kitchen verify)
Security Test: InSpec is audit based tool from Chef (kitchen verify) We can get (1) CIS hardening compliance and (2) visual reports. Another tool Gauntlt perform cyber attack at system
CMDB :
- DB
- then Puppet + Chef gave node registry
- then ZooKeeper from Haddop
- then etcd and HashiCorp's Consul emerged. They powers Mesos, Swarm and Kubernetes.
- Tumblr's Collins project
- Chef + Ohai = searchable CMDB
* etcd
* ZooKeepr
* Consul
Container As a Services
* Rancher
* Google Cloud Platform
* AWS ECS
Habitat : Application automation = Build + Deploy. It extends Chef.
Usecase
Just Recommendations:
1. Serverless : Basic needs.
2. container, docker, k8s : for medium complex system
3. Chef or Puppet: large distributed environment, network with heterogeneous systems and variety of apps (convergence model)
4. Ansible Orchestration: Website with high uptime requirement
5. Rundeck with Puppet: Update with coordination with other services.
Cloud
SaaS : TurboTax, Office365 Salesforce
PaaS: Google App Engine, Microsoft Azure App Service, Pivotal Web Service
IaaS: Google Compute Engine, Microsoft Azure VM, AWS EC2
FaaS : Amazon Lambda, Google Cloud Function, Azure Function
Hypervisor : VMWare, ZeN, Hyper-V, VirtualBox
Tools for cloud:
1. Hashicorp's Packer to create images
2. OpenStack
3. CloudFoundry by Pivotal
4. Open Source tools : MAAS (for Ubuntu's Juju), cobbler, Hanlon, helps in Pixie n/w booting
5. Then provisioning by Chef, Puppet, VMWare vCenter
6. Foreman is based on Puppet framework
7. Crowbar (Now Zeus cloud) : OpenStack and Hadoop installation
8. digital_REBAR : Provisioning + Orchestration
Levels of DevOps Practice
1. Values:
======
CAMS Culture Automation Measurement and Sharing
1.1 culture
1.2 automation,
1.3 measurement,
* MTTR
* cycle Time
* Costs
* Revenue
* Employee Satisfaction
1.4 sharing: Kizen
Later on, L for Lean is also added.
CALMS Culture Automation Lean Measurement and Sharing
========
2.1 System thinking : Concept to Cash
2.2 Amplifying feedback loop
2.3 Continuous experiments and learning
3. Methods
=======
3.1 People (Who is responsible) , Process, Tools
3.2 Continuous Delivery
3.3 Lean Management
3.4 Change Control
3.5 Infrastructure as code
4. Practices
========
4.1 Chaos Monkey
4.2 Blue / Green Deployment
4.3 Dependency Injection (Inversion of control) using service discovery
4.4 Andon Cord
4.5 Cloud : API driven way
4.6 Embedded Teams
4.7 Blameless Postmortem (industrial safety) should happen within 24 or 48 hours
4.8 Public Status Page is also about Transparent uptime
4.9 Developers on call
4.10 Incident Command System (Like fire department)
5. Tools
=====
Criteria for choosing tools
1. Tools should be programmable. Can invoke by API and CLI, not only GUI.
2. verifiable : events and matrix
3. well behave: deploy automatically, acceptance test case. it can be in form for .deb OR chef cookbook
5.1 Code – code development and review, source code management tools, code merging
5.2 Build – continuous integration tools, build status
5.3 Test – continuous testing tools that provide feedback on business risks
5.4 Package – artifact repository, application pre-deployment staging
5.5 Release – change management, release approvals, release automation
5.6 Configure – infrastructure configuration and management, infrastructure as code tools (e.g. Ansible, Puppet, Chef)
5.7 Monitor – applications performance monitoring, end-user experience (e.g. NewRelic, Kibana, Datadog)
Six Phases of tool chain
5.1 Version Control : Git (Github OR Bigbucket) as SAS or enterprise version. Hands-on http://try.github.io/
5.2 CI systems : Jenkins, commercial Jenkins from CloudBees, GoCD, Bamboo, Shippable, TeamCity. CI as Service : Travis CI, Circle CI
5.3 Build : Make, Rake, Maven (Workflow approach), Gulp for front end, HashiCorp's Packer for infrastructure
5.4 Test : Testing for developers before check in
5.4.1 Unit Test and code hygiene : JUnit, golint/gofmt, RuboCop,
Integration Testing : Robot, Protractor, Cucumber, Selenium (for UI testing), Sauce Labs from SAS for front end acceptance testing
5.4.2 Infrastructure Testing : Kitchen Chef
5.4.3 Performance Testing : Apache Bench, Meter, Apache JMeter
5.4.4 Security Testing : Brakeman, VeraCode, gauntlet
Vagrant, Otto, Docker Compose
5.5 Artifact Repository
Artifactory
Nexus
DockerHub
AWS S3
5.6 Deployment
RunDeck
UrbenCode
ThoughtWorks
Deployinator
Elastic Beanstalk
Octopus
Vamp
Saltstack
rerun
Most important tools are : Continuous Integration (e.g. Jenkins) and Infrastructure as a code (e.g. Puppet)
Containerization — Docker, Rocket, Unik
Orchestration — Kubernetes, Swarm, Mesos
Organisational culture
1. Pathological (Power oriented)
2. Bureaucratic (rule oriented)
3. Generative (performance oriented) cooperation and inquiry
Processes are like training wheels: So go for minimal viable process
How to build Trust, Respect and eliminate blame and hostility ?
* Some people may introvert
* People think others job is easy to do
* bring transparency "chat ops" : Hip Chat, slack, ICR, Hubot, Lita, Cog, Err
Organisational boundary is also communication boundary.
Progress Charts
1. Epic burn down charts
2. Release burn down charts
3. Sprint burn down charts
4. Velocity charts (committed v/s completed)
Kanban based matrix and charts
1. WIP : Cumulative Flow Diagram
2. throughput
3. lead time = work is requested till delivered
4. cycle time = amount of time we spend working on it.
"The Phoenix Project" Modeled on "The Goal" but DevOps specific
Reference
https://theagileadmin.com/
