Effective PPT
The goal for any presentation can be to create slides that lead to yes !
Yes for question about Why, How and What
Design Aspect
Content of PPT
Data
1. Relationship
Bubble chart (2 variables)
Scatter chart (3 variables)
2. Composition
2.1 Changing over time
Stacked 100% column chart
Stacked column chart
Stacked 100% area chart
Stacked area chart
2.2 static
Pie chart
Waterfall chart
Stacked 100% column chart with subcomponents
3. Distribution
Histogram (1 variable few items)
Line Histogram (1 variable many items)
Scatter Chart (2 variable)
3D Area chart (3 variable)
4. Comparison
Cyclic Area Chart
Line Chart
Column Chart
Single Bar Chart
Variable Width Column Chart
Reference Books
1. "brain rules" by John Medina
2. slide:ology
3. "Presentation Zen" by Garr Reynolds
4. "PresentationZen Design" by Garr Reynolds
5. "The Wall Street Journal Guide to Information Graphics" by Wong, Dona M.
Yes for question about Why, How and What
- The audience can retain limited information. The audience want to listen to you, instead of too much content on PPT slides.
- Show important points with image. Human process 60,000 x faster images compare to text. We can also recall information from image 6x time more easily compare to test.
Design Aspect
- Consistent font, font size, line length, spacing, margin, text alignment, kerning. For consistent arrangements of components, grid functionality can be used.
- Arial and Verdana fonts are better compare to Times New Roman. Avoid ALL CAPS
- Use any and only one highlighting method
- Simple and visually appealing slide is better compare to a cluttered slide to avoid distract and confusion. Avoid picture for decoration. Avoid too much of 3D, clip arts, word art, photos, complex visuals, lengthy text and animations.
- Add logo, slide number and copyright notice
- Select picture that support your message and story line in graphs.
Content of PPT
- Organize points in groups and use SmartArt
- Use concise slide title
- Use brand color as primary color and avoid using so many different colors. Maintain contrast between foreground and background color.
Data
- Use data to highlight important points
- Use table with detail view and summary view (total) both. Highlight key values.
- Mention the source of data
1. Relationship
Bubble chart (2 variables)
Scatter chart (3 variables)
2. Composition
2.1 Changing over time
Stacked 100% column chart
Stacked column chart
Stacked 100% area chart
Stacked area chart
2.2 static
Pie chart
Waterfall chart
Stacked 100% column chart with subcomponents
3. Distribution
Histogram (1 variable few items)
Line Histogram (1 variable many items)
Scatter Chart (2 variable)
3D Area chart (3 variable)
4. Comparison
Cyclic Area Chart
Line Chart
Column Chart
Single Bar Chart
Variable Width Column Chart
Reference Books
1. "brain rules" by John Medina
2. slide:ology
3. "Presentation Zen" by Garr Reynolds
4. "PresentationZen Design" by Garr Reynolds
5. "The Wall Street Journal Guide to Information Graphics" by Wong, Dona M.
Bengaluru Tech Summit : Day 3 - Part 1
We had interesting panel discussion on the topic Innovations driving next generation enterprise
Trends
* Now, the personal computing is more affordable compare to 1960s and 1980s due to to reduction is computing cost, storage cose and emergence of open source culture.
* In 1990s all the computers are interconnected and the Internet emerged along with new business models for finance/banking, retail e-commerce and e-mail. New enterprises also got emerged. Today we are living in digital age or mobile age.
* The people who born around 1995, have entered as work force around 2015. Generation Z, how to influence these 2 billion people, with different set of expectations.
* IoT
* Emergence of rural area. Now rural people also have access to information.
* Now people feel pride to work at startups.
* A new startup can be started just using WhatsApp groups.
* Supportive governments and Law. Few examples of few funny laws, that exist even today. Laws are reaction to 150 years of industrial revolution. The government should not stop the growth. So Karnatak is first state in India with regulatory sandbox, where you are allowed to break (only state) law for duration ranges from 3 months to 2 years, powered by Karnataka Innovation Authority. More can be achieve on startup front with de-regularization like allowing crypto currency.
* All the new innovations are becoming necessities.
* Big giant companies like IBM are supporting startups by finance, by connecting them to large customer base, scaling up. They cannot continue with their own R&D center so they come to startups. Block chain based virtual idea market may emerged in next few months.
* Online transactions will ultimately eliminate the need for banks.
* Education: 500 Mbps bandwidth at each home, will eliminate universities. all students will study online. Then even 6th standard students will talk about Artificial Intelligence, Machine Learning, IoT, Block Chain etc.
* The rule based and law based audit will change. Already after GST and demonetization the B.Com degree has more value compare to BBA or BBM. Now working people need to play different roles to remain in job market. Innovations at large enterprises are reducing jobs. On the other hand, same innovation increases the jobs at startups.
The factors that drives innovations are changed based on above trends. The generation z is consumer as well as is at startups, so disruption is bound to happen. Now all enterprise needs innovation to survive or to make more profit in areas of their products, service and/or process.
Discovery v/s Invention v/s Innovation
We know discovery is about something that already exists. Invention is one step further, by combining multiple few discoveries. Innovation goes even further by combining multiple inventions and build a use case to satisfy needs. The startups are all about building faster, better, cheaper solutions with help of inventions. Today major innovations are around (1) Artificial Intelligence, (2) data science and (3) block chain. Again startups need to focus on usage rather than the technology. 90% of startups fail due to lecuna about how to take it forward. All entrepreneurs cannot scale up their venture. Startups are risky, but design driven approach, reduces the risk.
It is not important, what you study (in college/school to get your degree). The important aspect is what you continue to study. Today everyone should know, how to learn new topics, unlearn old topics and re-learn as per emerging trends.
Bengaluru Tech Summit : Day 2
There was an interesting experiment on a patient, who daily forgets her past life. Everyday, she introduce herself, shake hands like meeting for the first time. The scientist doctor, started hurting her with small pin, during shake hand. Just in few days, she stopped shake hands with the scientist, even though she could not recover any memory associated about him. There are many research on topics of fear, fact, emotions, memory etc.
Brain Research in the age of information technology
As per Moravec's paradox, what we human beings find very easy is difficult for computers and what is easy for computers, that we find the most difficult. Like computer compare two images, pixel by pixel and so it cannot identify two objects are same, if the images are from different angle. On other way, two images of up-side-down faces, we human beings find identical, but when we rotate 180 degree, we realize they are different. It is easy for computer to tell, that they are different, as it compare pixel by pixel. Today 80 % of object detection is done by AI/ML.The 40% of our brain are is involved in visual processing. The brains does not look at brightness of pixel. We also discussed about famous image, about a dress, where a group people says white and blue, while others say gold and black. The color depends on individual perception about color of light falling on that dress white or yellow. Symmetry in deep learning network, increases accuracy from 1 to 10 %. Few initiatives:
* OpenAI is a non-profit AI research company, discovering and enacting the path to safe artificial general intelligence. Many projects by OpenAI on github.
* VisionLab@IISc, Bangalore. Website and Facebook page
Ashwini Godbole emphasize on trans-disciplinary approach of brain health as per Ayurveda and modern biomedicine for brain. She also explained about clinic to lab approach. She mentioend about meaningful eating.
Brain Research in the age of information technology
As per Moravec's paradox, what we human beings find very easy is difficult for computers and what is easy for computers, that we find the most difficult. Like computer compare two images, pixel by pixel and so it cannot identify two objects are same, if the images are from different angle. On other way, two images of up-side-down faces, we human beings find identical, but when we rotate 180 degree, we realize they are different. It is easy for computer to tell, that they are different, as it compare pixel by pixel. Today 80 % of object detection is done by AI/ML.The 40% of our brain are is involved in visual processing. The brains does not look at brightness of pixel. We also discussed about famous image, about a dress, where a group people says white and blue, while others say gold and black. The color depends on individual perception about color of light falling on that dress white or yellow. Symmetry in deep learning network, increases accuracy from 1 to 10 %. Few initiatives:
* OpenAI is a non-profit AI research company, discovering and enacting the path to safe artificial general intelligence. Many projects by OpenAI on github.
* VisionLab@IISc, Bangalore. Website and Facebook page
Ashwini Godbole emphasize on trans-disciplinary approach of brain health as per Ayurveda and modern biomedicine for brain. She also explained about clinic to lab approach. She mentioend about meaningful eating.
My DIKSHAA Guru
DIKSHAA means initiation. One can not grasp the spiritual knowledge using intelligence. The God, or the consciousness or the BHRAMAN is beyond intelligence. It cannot be proved or disproved with arguments. The Guru make us enter into this realm, that is beyond intelligence by means of DIKSHAA. It is a traditional ceremony, rituals, where Guru gives a secret and sacred MANTRA to the disciple.
Today let me share an old photograph of my DIKSHA Guru, who initiate me with BHRAM-SAMBANDH may be around year 1993.
Today let me share an old photograph of my DIKSHA Guru, who initiate me with BHRAM-SAMBANDH may be around year 1993.
Pu. Go. Shri. Vallabharaiji Bava Shri
(Jamnagar - Chopasani)
Bengaluru Tech Summit : Day 1
Damiel Manuel talked about Cyber Security. In this world, millions and billions of people are active on various social media like Facebook, WhatsApp, Instagram, YouTube, LinkedIn etc. It rises "deep fake" video threat. The video footage that appears to be real but is produced by technology. Now, we can no longer believe our eyes. NIST is a cyber security framework. Link 1 and Link 2 Cyber security is also a layered architecture. Hardware and network security is at physical layer. Application/software security at logical layer. Data security includes protection of personal information, location, accessibility, integrity of data etc. Social aspects covers people, culture and processes. it needs more attention. Cyber security topic now moves to Cyber resilience
Interesting panel discussions about AI/ML.
Ajay Sharma : Machine used to crunch numbers and humans make decision. Now machines are capable of doing both. Use cases (1) use Intel Mobileye to prevent, predict accident zones. (2) CheXNeXt is deep learning for chest radiograph diagnosis. Discussion about Niti Ayog #AIforAll ICTAI (International Center for Transformative AI) etc.
Raj Cherubal: Smart City solutions : (1) Parking management system (2) public bicycle sharing (3) restoration of water bodies (ponds) (4) solar panel on government buildings (5) water meter (6) smart class (7) disaster management system etc. IUDX (Indian Urben Data Exchange) Link Whitepaper Link
Aksah Ravi (gnani) talked about NLP based voice assistant products that supports Hindi, Kannada etc.
Out of all AI/ML projects hardly 13 to 14 % projects are successful and only 8% of customer are happy. Some projects need their ML model to be updated every week, even every day. AI/ML business case is all about identifying and satisfying unmet and unmentioned need. Few challenges (1) AI/ML implementation needs cultural change, where man and machine both work together. (2) upskill of existing staff (3) Data is fuel for AI. More time goes to find out which data to use rather than building AI/ML model. Noise in data can cause bias. The chatbot may learn offensive language over time. (4) explain-ability of AI/ML outcome. (5) Robust governance is also important. At present, chatbots are becoming popular. Long way to go. Information architecture (IA) is must for AI/ML projects. IA is the structural design of shared information environments.
Queryplex and federation are effective tools from IBM to connect multiple data sources. AI OpenScale is another tool from IBM
Ashok Ayengar talked about shopx - buy and sale platform.
Last 25 years of history big enterprise adopted new technologies like ERP and small stratup companies built eco-system around it. Many fortune 500 companies have huge legacy in range of 20 to 100 years. They need machine learning even on end-of-life servers.
The collaboration at Bangalore is not seen even in developed country. Here we have hackathon, open source contribution, discussion at physical meetup etc. It is really proud to be in Bangalore.
The day 1 ended with thought leaders conclave on "TechScape 2035"
The following trends are emerged.
1. Digitalization
2. Rapid Urbanization
3. Customer does not want product, but wants solution
4. so more and more collaborations
The growth of Internet was not predicated.
1. Growth with many IoT devices
2. Growth for amusement e.g. Netflix
3. Many networks E.g. Car has many networks
During 90s, when Internet based innovative services and business was planned, no one predicated that everyone (4 billion people) will have smart phone. Now cost of computing come down. Technology is democratized. This growth will be accelerated further, as everything (e.g. speaker, light etc.) will have compute power, connectivity, intelligent (AI/ML). The enormous data is generated by our wearable. Evolution will happen in laboratory and new human being will be created. Excitement with fear.
Now the smart software makes diagnosis smarter than doctor. In next 5 to 10 years, it will be mandatory to take second opinion from machines. People will get incentive for not being sick by health insurance and hospitals. The Indian doctors and nurses are best in the world. This is a license driven industry.
Now in automobile people need : 0 emission, 0 downtime, 0 accident, and first solution is electric car. Electricity is alternate fuel. The omnipresent 3D printing will eliminate need to keep inventory of spare parts of cars. The digitization, GST have improved logistics a lot. We need more efficiency steps like: spare trailers will just replace the loaded one to save time.
All these trends will empower the bottom of pyramid, surprisingly.
In future, there is no concept of firm. All companies are becoming platform. Apple started as platform for iOS apps. People starts company just on WhatsApp group. Now we will see rapid consolidation of firms. So monopoly will be back.
The present generation is Digital Native and generation Z Their working style is completely different.
Interesting panel discussions about AI/ML.
Ajay Sharma : Machine used to crunch numbers and humans make decision. Now machines are capable of doing both. Use cases (1) use Intel Mobileye to prevent, predict accident zones. (2) CheXNeXt is deep learning for chest radiograph diagnosis. Discussion about Niti Ayog #AIforAll ICTAI (International Center for Transformative AI) etc.
Raj Cherubal: Smart City solutions : (1) Parking management system (2) public bicycle sharing (3) restoration of water bodies (ponds) (4) solar panel on government buildings (5) water meter (6) smart class (7) disaster management system etc. IUDX (Indian Urben Data Exchange) Link Whitepaper Link
Aksah Ravi (gnani) talked about NLP based voice assistant products that supports Hindi, Kannada etc.
Out of all AI/ML projects hardly 13 to 14 % projects are successful and only 8% of customer are happy. Some projects need their ML model to be updated every week, even every day. AI/ML business case is all about identifying and satisfying unmet and unmentioned need. Few challenges (1) AI/ML implementation needs cultural change, where man and machine both work together. (2) upskill of existing staff (3) Data is fuel for AI. More time goes to find out which data to use rather than building AI/ML model. Noise in data can cause bias. The chatbot may learn offensive language over time. (4) explain-ability of AI/ML outcome. (5) Robust governance is also important. At present, chatbots are becoming popular. Long way to go. Information architecture (IA) is must for AI/ML projects. IA is the structural design of shared information environments.
Queryplex and federation are effective tools from IBM to connect multiple data sources. AI OpenScale is another tool from IBM
Ashok Ayengar talked about shopx - buy and sale platform.
Last 25 years of history big enterprise adopted new technologies like ERP and small stratup companies built eco-system around it. Many fortune 500 companies have huge legacy in range of 20 to 100 years. They need machine learning even on end-of-life servers.
The collaboration at Bangalore is not seen even in developed country. Here we have hackathon, open source contribution, discussion at physical meetup etc. It is really proud to be in Bangalore.
The day 1 ended with thought leaders conclave on "TechScape 2035"
The following trends are emerged.
1. Digitalization
2. Rapid Urbanization
3. Customer does not want product, but wants solution
4. so more and more collaborations
The growth of Internet was not predicated.
1. Growth with many IoT devices
2. Growth for amusement e.g. Netflix
3. Many networks E.g. Car has many networks
During 90s, when Internet based innovative services and business was planned, no one predicated that everyone (4 billion people) will have smart phone. Now cost of computing come down. Technology is democratized. This growth will be accelerated further, as everything (e.g. speaker, light etc.) will have compute power, connectivity, intelligent (AI/ML). The enormous data is generated by our wearable. Evolution will happen in laboratory and new human being will be created. Excitement with fear.
Now the smart software makes diagnosis smarter than doctor. In next 5 to 10 years, it will be mandatory to take second opinion from machines. People will get incentive for not being sick by health insurance and hospitals. The Indian doctors and nurses are best in the world. This is a license driven industry.
Now in automobile people need : 0 emission, 0 downtime, 0 accident, and first solution is electric car. Electricity is alternate fuel. The omnipresent 3D printing will eliminate need to keep inventory of spare parts of cars. The digitization, GST have improved logistics a lot. We need more efficiency steps like: spare trailers will just replace the loaded one to save time.
All these trends will empower the bottom of pyramid, surprisingly.
In future, there is no concept of firm. All companies are becoming platform. Apple started as platform for iOS apps. People starts company just on WhatsApp group. Now we will see rapid consolidation of firms. So monopoly will be back.
The present generation is Digital Native and generation Z Their working style is completely different.
Continuous delivery : 2nd DevOps Pillar
CI = build and UT after code checkin
Six Principle
2.1.1 Build less than 5 min
2.1.2 Commit small bits
2.1.3 Don't leave the build broken
2.1.4 Trunk based development flow
2.1.5 Don't allow flaky test. Fix them
2.1.6 build should return status, log and artifact
CDelivery = CI + deploy at production like environment (staging) + automate integration testing + automate acceptance testing + smoke testing
Principles
2.2.1 Build artifact once
2.2.2 Artifact should be immutable
2.2.3 Deployment to production like environment = load balancers + n/w settings + security controls + data that matches production
2.2.4 stop if previous step fails
2.2.5 Deployment should be idempotent
CDeployment = CDelivery + Full automated testing + Deployed to production
For higher performances teams
* Less than 3 branch (ideally 1 branch)
* Their life span is less than a day and then merge to trunk.
It is not about how much you can deliver but it is about how little you can
CDelivery the build is not always release to production environment.
CDeployment, the build is always release to production environment.
Tools
Build Systems : Jenkins, Bamboo, Teamcity etc
Online build : Circle CI, Travis CI
Monitoring tool: Datadog
./hooks/git-pre-commit.hooks
webhooks : Notify external service with POST request about changes in repository.
Jenkins :
* We can add, remove steps in build "configure"
* We can add/remove plugins for different languages : C, Go etc
* We can configure those plugins using global tools configuration.
* Jenkins plugin repository https://plugins.jenkins.io/ We can add custom plugins. Plugins to notify on slack about build failure. Based on installed plugin we can get options in build
* Pipeline feature converts GUI to "build as code".
* BUILD_NUMBER is env. variable. We can add this number as tag and commit back to git repo for backward compatibility.
Test terms
* Shift Left : test at developer's machine by developer. Left side in L to R pipeline
* Test Fixture : e.g. sample data, Linux server configuration. They are also artifacts.
Test Philosophy
TDD: first right negative TCs
BDD: Tool Cucumber
ATD : Acceptance Test Driven
KDD: Keyword driven testing
Test Split
70 % Unit Testing
20 % Integration Testing
10 % E2E Testing
JQ is good tool to deal with JSON data
YQ converts YAML to JSON.
REST API Markup Language
Tools: Swagger, RAML ; abao, Mocha, chai are also relevant tools
GUI Testing
1. Selenium
2. Robot f/w
2.1 Selenium2Library is plugin for Robot f/w, that connects both.
2.2 PageObject design pattern
Dynamic Security Testing
1. Gauntlt : Given, When, Then conventions
2. arachni
3. nmap
4. Fortify
5. Vericode
6. Brakeman
7. Bundler
8. Audit
9. Retire.js
Reference
https://testing.googleblog.com
Crazy Fast Build: https://www.infoq.com/presentations/Crazy-Fast-Build-Times-or-When-10-Seconds-Starts-to-Make-You-Nervous
http://dan.bodar.com/2012/02/28/crazy-fast-build-times-or-when-10-seconds-starts-to-make-you-nervous/
Six Principle
2.1.1 Build less than 5 min
2.1.2 Commit small bits
2.1.3 Don't leave the build broken
2.1.4 Trunk based development flow
2.1.5 Don't allow flaky test. Fix them
2.1.6 build should return status, log and artifact
CDelivery = CI + deploy at production like environment (staging) + automate integration testing + automate acceptance testing + smoke testing
Principles
2.2.1 Build artifact once
2.2.2 Artifact should be immutable
2.2.3 Deployment to production like environment = load balancers + n/w settings + security controls + data that matches production
2.2.4 stop if previous step fails
2.2.5 Deployment should be idempotent
CDeployment = CDelivery + Full automated testing + Deployed to production
For higher performances teams
* Less than 3 branch (ideally 1 branch)
* Their life span is less than a day and then merge to trunk.
It is not about how much you can deliver but it is about how little you can
CDelivery the build is not always release to production environment.
CDeployment, the build is always release to production environment.
Tools
Build Systems : Jenkins, Bamboo, Teamcity etc
Online build : Circle CI, Travis CI
Monitoring tool: Datadog
./hooks/git-pre-commit.hooks
webhooks : Notify external service with POST request about changes in repository.
Jenkins :
* We can add, remove steps in build "configure"
* We can add/remove plugins for different languages : C, Go etc
* We can configure those plugins using global tools configuration.
* Jenkins plugin repository https://plugins.jenkins.io/ We can add custom plugins. Plugins to notify on slack about build failure. Based on installed plugin we can get options in build
* Pipeline feature converts GUI to "build as code".
* BUILD_NUMBER is env. variable. We can add this number as tag and commit back to git repo for backward compatibility.
Test terms
* Shift Left : test at developer's machine by developer. Left side in L to R pipeline
* Test Fixture : e.g. sample data, Linux server configuration. They are also artifacts.
Test Philosophy
TDD: first right negative TCs
BDD: Tool Cucumber
ATD : Acceptance Test Driven
KDD: Keyword driven testing
Test Split
70 % Unit Testing
20 % Integration Testing
10 % E2E Testing
JQ is good tool to deal with JSON data
YQ converts YAML to JSON.
REST API Markup Language
Tools: Swagger, RAML ; abao, Mocha, chai are also relevant tools
GUI Testing
1. Selenium
2. Robot f/w
2.1 Selenium2Library is plugin for Robot f/w, that connects both.
2.2 PageObject design pattern
Dynamic Security Testing
1. Gauntlt : Given, When, Then conventions
2. arachni
3. nmap
4. Fortify
5. Vericode
6. Brakeman
7. Bundler
8. Audit
9. Retire.js
Reference
https://testing.googleblog.com
Crazy Fast Build: https://www.infoq.com/presentations/Crazy-Fast-Build-Times-or-When-10-Seconds-Starts-to-Make-You-Nervous
http://dan.bodar.com/2012/02/28/crazy-fast-build-times-or-when-10-seconds-starts-to-make-you-nervous/
Infrastructure automation : 1st DevOps Pillar
Operation team should use source control
Define your infrastructure as model or drive it using RESTful API
Model OR Recipes = Hardware + OS(Operating System) + OS Dependencies + System Configuration + Accounts + SSL Certs + Application
Important terms
* Imperative (procedural) Approach where commands desired to produce a state are defined and then executed. E.g. BASH Shell script, AWS CLI, Python Boto library
* declarative (functional) Approach where you define the desired state, and the tool converges the existing system on the model. E.g. Makefile, Puppet manifest, SQL Queries.
declarative is more efficient.
* Convergent : CM automation that converges the system to desired state.
* Idempotent : The ability to execute the CM procedure repeatedly. And end up in the same state each time. The second run will not make any change.
* Immutable : Not change after deployment. Can be redeploy if needed. E.g. Docker container.
* Self service : The ability for an end user to kick off one of these processes without having to go through other people
http://www.infrastructures.org/
Configuration Management
* Provisioning : Server ready with hardware, OS, system services, network
Two types of provisioning : Model based provisioning and Convergent Driver Provisioning
1. Model based provisioning
1.1 declarative model: n/w, node, db etc. E.g. using template YAML
or imperative manner, step by step
Tools
* ColudFormation JSON file for AWS
* Azure Resource Manager template for Azure
* HashiCorp's Terraform for multiple cloud providers. It maintains external state store
* Ubuntu's Juju = Infra + services (common in docker)
** Orchestration : Coordinated operations across multiple systems
Provisioners and providers are building blocks of Vagrant.
1. Vagrant has provisioners like Puppe, Chef, Ansible etc.
2. Vagrant has Providers like VirtualBox, Hyper-V, Docker. VMWare and AWS via plugin.
vagrantfile
commands
vagrant destroy
vagrant status
2. Convergent Driver Provisioning.
It is declarative and idempotent
It handles orchestration by state tracking and master registry. Each node reports system information to the registry.
Chef components
1. Run-list
2. Roles
3. Recipes belongs to cookbook.
Cookbook public repository (Chef): https://supermarket.chef.io/
Puppet public repository : Puppet Forge https://forge.puppet.com/
purpose of Ohai is to provide node data to Chef-client, to use within cookbooks.
commands
kitchen converge
kitchen list
kitchen destroy
Tools
Chef
Lint: Foodcritic
Unit Testing : Chefspec
Integration Testing : KitchenCI
Puppet
Ansible
Salt
CFEngine
Packer
Shell Provisioner
Evolution of provisioning tools
Adhoc script OR Capistrano framework by developers to push changes.
Provisioning Tools: CFEngine to Puppet to Chef ,
Commercial IT Provisioning tool : Ghost,
Enterprise tool : HP
Then Golden Image or Foil Ball : This result in (1) image sprawl or (2) configuration drift. So stem cell system approach with minimum initial provisioning and update later on : Chef DSL,
Capistrano to RunDeck
Rundeck is a container, which listen to on 4440
docker run -p 4440:4440 -e SERVER_URL=http://localhost:4440 --name AnyName -t /jordan/rundeck:latest
Orchestrated deployment techniques
* Canary Deployment
* Blue / Green Deployment
* Cluster immune system deployment
* Immutable deployment (1) for Container = OS + Dependency + App code (2) not for data store
Artifact :
JAR to WAR to
-Deb
-RPM
to
- Amazon Machine Image (AMI for amazon)
- VM image (using packer)
- Docker image (using docker file)
Maven's jdub plugins: for Debian image
FPM : deb and RPM package etc. format
Artifact Repository
* Nexus
* JFrog Artifactory
* Apache Archiva
Testing:
- Unit (includes formatters and linters)
- Integration
- Security
We can include tools related to formatter and linter too as part of CUT (coding and Unit Testing) phase for developer.
Formatter : Rubocop. It is for Ruby. Chef and Puppet are on top of Ruby
Linter: Foodcritic for Chef
Unit Test : ChefSpec is extension of RSpec framework (for Ruby)
Integration Test : Serverspec (written in RSpec). Before that we run TestKitchen (kitchen converge) then Serverspec (kitchen verify)
Security Test: InSpec is audit based tool from Chef (kitchen verify) We can get (1) CIS hardening compliance and (2) visual reports. Another tool Gauntlt perform cyber attack at system
CMDB :
* etcd
* ZooKeepr
* Consul
Container As a Services
* Rancher
* Google Cloud Platform
* AWS ECS
Habitat : Application automation = Build + Deploy. It extends Chef.
Usecase
Just Recommendations:
1. Serverless : Basic needs.
2. container, docker, k8s : for medium complex system
3. Chef or Puppet: large distributed environment, network with heterogeneous systems and variety of apps (convergence model)
4. Ansible Orchestration: Website with high uptime requirement
5. Rundeck with Puppet: Update with coordination with other services.
Cloud
SaaS : TurboTax, Office365 Salesforce
PaaS: Google App Engine, Microsoft Azure App Service, Pivotal Web Service
IaaS: Google Compute Engine, Microsoft Azure VM, AWS EC2
FaaS : Amazon Lambda, Google Cloud Function, Azure Function
Hypervisor : VMWare, ZeN, Hyper-V, VirtualBox
Tools for cloud:
1. Hashicorp's Packer to create images
2. OpenStack
3. CloudFoundry by Pivotal
4. Open Source tools : MAAS (for Ubuntu's Juju), cobbler, Hanlon, helps in Pixie n/w booting
5. Then provisioning by Chef, Puppet, VMWare vCenter
6. Foreman is based on Puppet framework
7. Crowbar (Now Zeus cloud) : OpenStack and Hadoop installation
8. digital_REBAR : Provisioning + Orchestration
Define your infrastructure as model or drive it using RESTful API
Model OR Recipes = Hardware + OS(Operating System) + OS Dependencies + System Configuration + Accounts + SSL Certs + Application
Important terms
* Imperative (procedural) Approach where commands desired to produce a state are defined and then executed. E.g. BASH Shell script, AWS CLI, Python Boto library
* declarative (functional) Approach where you define the desired state, and the tool converges the existing system on the model. E.g. Makefile, Puppet manifest, SQL Queries.
declarative is more efficient.
* Convergent : CM automation that converges the system to desired state.
* Idempotent : The ability to execute the CM procedure repeatedly. And end up in the same state each time. The second run will not make any change.
* Immutable : Not change after deployment. Can be redeploy if needed. E.g. Docker container.
* Self service : The ability for an end user to kick off one of these processes without having to go through other people
http://www.infrastructures.org/
Configuration Management
* Provisioning : Server ready with hardware, OS, system services, network
Two types of provisioning : Model based provisioning and Convergent Driver Provisioning
1. Model based provisioning
1.1 declarative model: n/w, node, db etc. E.g. using template YAML
or imperative manner, step by step
Tools
* ColudFormation JSON file for AWS
* Azure Resource Manager template for Azure
* HashiCorp's Terraform for multiple cloud providers. It maintains external state store
* Ubuntu's Juju = Infra + services (common in docker)
* Vagrant tool emulate production environment.
** Deployment : Automatic deploy / upgrade application on server** Orchestration : Coordinated operations across multiple systems
Provisioners and providers are building blocks of Vagrant.
1. Vagrant has provisioners like Puppe, Chef, Ansible etc.
2. Vagrant has Providers like VirtualBox, Hyper-V, Docker. VMWare and AWS via plugin.
vagrantfile
commands
vagrant destroy
vagrant status
2. Convergent Driver Provisioning.
It is declarative and idempotent
It handles orchestration by state tracking and master registry. Each node reports system information to the registry.
Chef components
1. Run-list
2. Roles
3. Recipes belongs to cookbook.
Cookbook public repository (Chef): https://supermarket.chef.io/
Puppet public repository : Puppet Forge https://forge.puppet.com/
purpose of Ohai is to provide node data to Chef-client, to use within cookbooks.
commands
kitchen converge
kitchen list
kitchen destroy
Tools
Chef
Lint: Foodcritic
Unit Testing : Chefspec
Integration Testing : KitchenCI
Puppet
Ansible
Salt
CFEngine
Packer
Shell Provisioner
Evolution of provisioning tools
Adhoc script OR Capistrano framework by developers to push changes.
Provisioning Tools: CFEngine to Puppet to Chef ,
Commercial IT Provisioning tool : Ghost,
Enterprise tool : HP
Then Golden Image or Foil Ball : This result in (1) image sprawl or (2) configuration drift. So stem cell system approach with minimum initial provisioning and update later on : Chef DSL,
Capistrano to RunDeck
Rundeck is a container, which listen to on 4440
docker run -p 4440:4440 -e SERVER_URL=http://localhost:4440 --name AnyName -t /jordan/rundeck:latest
Orchestrated deployment techniques
* Canary Deployment
* Blue / Green Deployment
* Cluster immune system deployment
* Immutable deployment (1) for Container = OS + Dependency + App code (2) not for data store
Artifact :
JAR to WAR to
-Deb
-RPM
to
- Amazon Machine Image (AMI for amazon)
- VM image (using packer)
- Docker image (using docker file)
Maven's jdub plugins: for Debian image
FPM : deb and RPM package etc. format
Artifact Repository
* Nexus
* JFrog Artifactory
* Apache Archiva
Testing:
- Unit (includes formatters and linters)
- Integration
- Security
We can include tools related to formatter and linter too as part of CUT (coding and Unit Testing) phase for developer.
Formatter : Rubocop. It is for Ruby. Chef and Puppet are on top of Ruby
Linter: Foodcritic for Chef
Unit Test : ChefSpec is extension of RSpec framework (for Ruby)
Integration Test : Serverspec (written in RSpec). Before that we run TestKitchen (kitchen converge) then Serverspec (kitchen verify)
Security Test: InSpec is audit based tool from Chef (kitchen verify) We can get (1) CIS hardening compliance and (2) visual reports. Another tool Gauntlt perform cyber attack at system
CMDB :
- DB
- then Puppet + Chef gave node registry
- then ZooKeeper from Haddop
- then etcd and HashiCorp's Consul emerged. They powers Mesos, Swarm and Kubernetes.
- Tumblr's Collins project
- Chef + Ohai = searchable CMDB
* etcd
* ZooKeepr
* Consul
Container As a Services
* Rancher
* Google Cloud Platform
* AWS ECS
Habitat : Application automation = Build + Deploy. It extends Chef.
Usecase
Just Recommendations:
1. Serverless : Basic needs.
2. container, docker, k8s : for medium complex system
3. Chef or Puppet: large distributed environment, network with heterogeneous systems and variety of apps (convergence model)
4. Ansible Orchestration: Website with high uptime requirement
5. Rundeck with Puppet: Update with coordination with other services.
Cloud
SaaS : TurboTax, Office365 Salesforce
PaaS: Google App Engine, Microsoft Azure App Service, Pivotal Web Service
IaaS: Google Compute Engine, Microsoft Azure VM, AWS EC2
FaaS : Amazon Lambda, Google Cloud Function, Azure Function
Hypervisor : VMWare, ZeN, Hyper-V, VirtualBox
Tools for cloud:
1. Hashicorp's Packer to create images
2. OpenStack
3. CloudFoundry by Pivotal
4. Open Source tools : MAAS (for Ubuntu's Juju), cobbler, Hanlon, helps in Pixie n/w booting
5. Then provisioning by Chef, Puppet, VMWare vCenter
6. Foreman is based on Puppet framework
7. Crowbar (Now Zeus cloud) : OpenStack and Hadoop installation
8. digital_REBAR : Provisioning + Orchestration
Levels of DevOps Practice
1. Values:
======
CAMS Culture Automation Measurement and Sharing
1.1 culture
1.2 automation,
1.3 measurement,
* MTTR
* cycle Time
* Costs
* Revenue
* Employee Satisfaction
1.4 sharing: Kizen
Later on, L for Lean is also added.
CALMS Culture Automation Lean Measurement and Sharing
========
2.1 System thinking : Concept to Cash
2.2 Amplifying feedback loop
2.3 Continuous experiments and learning
3. Methods
=======
3.1 People (Who is responsible) , Process, Tools
3.2 Continuous Delivery
3.3 Lean Management
3.4 Change Control
3.5 Infrastructure as code
4. Practices
========
4.1 Chaos Monkey
4.2 Blue / Green Deployment
4.3 Dependency Injection (Inversion of control) using service discovery
4.4 Andon Cord
4.5 Cloud : API driven way
4.6 Embedded Teams
4.7 Blameless Postmortem (industrial safety) should happen within 24 or 48 hours
4.8 Public Status Page is also about Transparent uptime
4.9 Developers on call
4.10 Incident Command System (Like fire department)
5. Tools
=====
Criteria for choosing tools
1. Tools should be programmable. Can invoke by API and CLI, not only GUI.
2. verifiable : events and matrix
3. well behave: deploy automatically, acceptance test case. it can be in form for .deb OR chef cookbook
5.1 Code – code development and review, source code management tools, code merging
5.2 Build – continuous integration tools, build status
5.3 Test – continuous testing tools that provide feedback on business risks
5.4 Package – artifact repository, application pre-deployment staging
5.5 Release – change management, release approvals, release automation
5.6 Configure – infrastructure configuration and management, infrastructure as code tools (e.g. Ansible, Puppet, Chef)
5.7 Monitor – applications performance monitoring, end-user experience (e.g. NewRelic, Kibana, Datadog)
Six Phases of tool chain
5.1 Version Control : Git (Github OR Bigbucket) as SAS or enterprise version. Hands-on http://try.github.io/
5.2 CI systems : Jenkins, commercial Jenkins from CloudBees, GoCD, Bamboo, Shippable, TeamCity. CI as Service : Travis CI, Circle CI
5.3 Build : Make, Rake, Maven (Workflow approach), Gulp for front end, HashiCorp's Packer for infrastructure
5.4 Test : Testing for developers before check in
5.4.1 Unit Test and code hygiene : JUnit, golint/gofmt, RuboCop,
Integration Testing : Robot, Protractor, Cucumber, Selenium (for UI testing), Sauce Labs from SAS for front end acceptance testing
5.4.2 Infrastructure Testing : Kitchen Chef
5.4.3 Performance Testing : Apache Bench, Meter, Apache JMeter
5.4.4 Security Testing : Brakeman, VeraCode, gauntlet
Vagrant, Otto, Docker Compose
5.5 Artifact Repository
Artifactory
Nexus
DockerHub
AWS S3
5.6 Deployment
RunDeck
UrbenCode
ThoughtWorks
Deployinator
Elastic Beanstalk
Octopus
Vamp
Saltstack
rerun
Most important tools are : Continuous Integration (e.g. Jenkins) and Infrastructure as a code (e.g. Puppet)
Containerization — Docker, Rocket, Unik
Orchestration — Kubernetes, Swarm, Mesos
Organisational culture
1. Pathological (Power oriented)
2. Bureaucratic (rule oriented)
3. Generative (performance oriented) cooperation and inquiry
Processes are like training wheels: So go for minimal viable process
How to build Trust, Respect and eliminate blame and hostility ?
* Some people may introvert
* People think others job is easy to do
* bring transparency "chat ops" : Hip Chat, slack, ICR, Hubot, Lita, Cog, Err
Organisational boundary is also communication boundary.
Progress Charts
1. Epic burn down charts
2. Release burn down charts
3. Sprint burn down charts
4. Velocity charts (committed v/s completed)
Kanban based matrix and charts
1. WIP : Cumulative Flow Diagram
2. throughput
3. lead time = work is requested till delivered
4. cycle time = amount of time we spend working on it.
"The Phoenix Project" Modeled on "The Goal" but DevOps specific
Reference
https://theagileadmin.com/
Building Blocks of DevOps
DevOps includes both
1. New project management principles : agile and lean
2. old-school principles : ITIL (Information Technology Infrastructure Library) , ITSM (IT Service Management), and SDLC
Let's see three building blocks of DevOps.
1. DevOps should be extensions of Agile
Iterative and incremental process
Agile Manifesto
1.1 Individual and interactions > processes and tools
1.2 working s/w > comprehensive documentation
1.3 customer collaboration > contract negotiona
1.4 Responding to change > following plan
12 principles
1.1 highest priority is satisfy customer with early + continuous valuable s/w delivery
1.2 welcome requirement change
1.3 frequent delivery of working s/w
1.4 business people and developer works together daily
1.5 motivated individual
1.6 face to face conversation
1.7 working s/w is primary measure of progress
1.8 constant pace and sustainable development
1.9 technical excellence and good design
1.10 simplicity
1.11 self organising team
1.12 retrospection
Methodologies
Scrum
Kanabn
DSDM : Dynamic System Development Method
FDD : Feature Driven Development
ASD
eXtreme Programming
Lean
Crystal
Rational Unified Process
Widely used are scrum, Kanban and lean.
1.1 scrum
from rugby game
meaning: tight player formation that is trying to move the ball towards goal
Scrum Life Cycle : Product backlog -> Sprint Backlog -> Sprint -> Valuable product
Sprint = timebox of a month or less
1.2 kanban
5 principles
1.2.1 Visualize workflow
1.2.2 Limit WIP
1.2.3 Manage the flow
1.2.4 Make process policies explicit
1.2.5 Improve collaboratively
SAFe (Scaled Agile Framework) to reduce complexity for large scale implementation.
2. Lean :
Lean inspired by SPC, Kizen, DoE, JIT, ToC, quality tools
Value Stream Mapping
From Supplier to Customer : What Information and Materials are flowing
Seven principles
2.1 Eliminating waste
waste = The effort spend on anything other than creating value
* Muda : Absorbs resources but no value. 7 wastes
Type 1: Technical waste for compliance
Type 2: Plain waste
* Mura : unevenness , No regular flow of work. Overburden
* Muri : overburden process, person or system. unevenness
2.1.1 Partially done work
2.1.2 Extra Features
2.1.3 Re-Learning
2.1.4 Handoffs
2.1.5 Delays
2.1.6 Task Switching
2.1.7 Defects
2.2 Amplify Learning
2.3 Decide as late as possible
2.4 Deliver as fast as possible
2.5 Empower the team
2.6 Build an integrity
2.7 See the whole
Plan Do Check Act of Kaizen
is
Build Measure Learn Repeat in Lean
Four Principles
2.1. Pull scheduling
2.2. Kanban
2.3. Visual Management
2.4. Employee empowerment
For software industry
2.1 Limiting WIP
2.2 Kanban Visibility to flow
2.3 Customer f/b
2.4 Empowered team
Upcoming : Lean UX
3. Six principles of Kaizen
3.1. Good process brings good results
3.2. Gemba
3.3. Speak with data
3.4. Take action to correct root cause
* Five Whys
* Focus on causes not symptoms
* Fish Bone : Equipment, Process, People, Materials, Environment, Management
* Do not accept (1) No time (2) human error . People does not fail. Process fails.
3.5. Team work
3.6. Kaizen is for everyone
4. ITIL (Information Technology Infrastructure Library), ITSM (IT Service Management) and SDLC
1. New project management principles : agile and lean
2. old-school principles : ITIL (Information Technology Infrastructure Library) , ITSM (IT Service Management), and SDLC
Let's see three building blocks of DevOps.
1. DevOps should be extensions of Agile
Iterative and incremental process
Agile Manifesto
1.1 Individual and interactions > processes and tools
1.2 working s/w > comprehensive documentation
1.3 customer collaboration > contract negotiona
1.4 Responding to change > following plan
12 principles
1.1 highest priority is satisfy customer with early + continuous valuable s/w delivery
1.2 welcome requirement change
1.3 frequent delivery of working s/w
1.4 business people and developer works together daily
1.5 motivated individual
1.6 face to face conversation
1.7 working s/w is primary measure of progress
1.8 constant pace and sustainable development
1.9 technical excellence and good design
1.10 simplicity
1.11 self organising team
1.12 retrospection
Methodologies
Scrum
Kanabn
DSDM : Dynamic System Development Method
FDD : Feature Driven Development
ASD
eXtreme Programming
Lean
Crystal
Rational Unified Process
Widely used are scrum, Kanban and lean.
1.1 scrum
from rugby game
meaning: tight player formation that is trying to move the ball towards goal
Scrum Life Cycle : Product backlog -> Sprint Backlog -> Sprint -> Valuable product
Sprint = timebox of a month or less
1.2 kanban
5 principles
1.2.1 Visualize workflow
1.2.2 Limit WIP
1.2.3 Manage the flow
1.2.4 Make process policies explicit
1.2.5 Improve collaboratively
SAFe (Scaled Agile Framework) to reduce complexity for large scale implementation.
2. Lean :
Lean inspired by SPC, Kizen, DoE, JIT, ToC, quality tools
Value Stream Mapping
From Supplier to Customer : What Information and Materials are flowing
Seven principles
2.1 Eliminating waste
waste = The effort spend on anything other than creating value
* Muda : Absorbs resources but no value. 7 wastes
Type 1: Technical waste for compliance
Type 2: Plain waste
* Mura : unevenness , No regular flow of work. Overburden
* Muri : overburden process, person or system. unevenness
2.1.1 Partially done work
2.1.2 Extra Features
2.1.3 Re-Learning
2.1.4 Handoffs
2.1.5 Delays
2.1.6 Task Switching
2.1.7 Defects
2.2 Amplify Learning
2.3 Decide as late as possible
2.4 Deliver as fast as possible
2.5 Empower the team
2.6 Build an integrity
2.7 See the whole
Plan Do Check Act of Kaizen
is
Build Measure Learn Repeat in Lean
Four Principles
2.1. Pull scheduling
2.2. Kanban
2.3. Visual Management
2.4. Employee empowerment
For software industry
2.1 Limiting WIP
2.2 Kanban Visibility to flow
2.3 Customer f/b
2.4 Empowered team
Upcoming : Lean UX
3. Six principles of Kaizen
3.1. Good process brings good results
3.2. Gemba
3.3. Speak with data
3.4. Take action to correct root cause
* Five Whys
* Focus on causes not symptoms
* Fish Bone : Equipment, Process, People, Materials, Environment, Management
* Do not accept (1) No time (2) human error . People does not fail. Process fails.
3.5. Team work
3.6. Kaizen is for everyone
4. ITIL (Information Technology Infrastructure Library), ITSM (IT Service Management) and SDLC
PyCharm : My Notes
PyCharm is an IDE for Python development. Here are few key take away points:
$ pip freeze > requirements.txt
This will create a requirements.txt file, which contains a simple list of all the packages in the current environment, and their respective versions. Later, when a different developer (or you, if you need to re- create the environment) can install the same packages, with the same versions by running
$ pip install -r requirements.txt
================================================================================
Code Completion
1. Basic : Ctrl + Space
2. Smart : Ctrl + Shift + Space
3. Within context : Atl + / Atl + Shift + /
4. Post Fix completion : Space OR Enter OR Tab
Ctrl + Shift + Enter to make all auto corrections
================================================================================
Pipenv is a tool that provides all necessary means to create a virtual environment for your Python project. It automatically manages project packages through the Pipfile file as you install or uninstall packages.
================================================================================
Two settings
1. Project level settings are applied to current project and stored at :
C:\Users\\PycharmProjects\schoop_app\.idea
2. Global settings are like template, it applies to new projects. Files-> Settings for new projects
================================================================================
Keyboard short cut
Double Shift : search
Ctrl+N
Ctrl+Shift+N
Ctrl+Shift+Alt+N : Find and jump to the desired class, file, or symbol.
Ctrl+E : View Recent Files-
================================================================================
AI-ML Algorithm
- Choosing the right estimator (scikit-learn) : http://scikit-learn.org/stable/tutorial/machine_learning_map/
- How to choose algorithms for Azure Machine Learning Studio : https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-choice
- A Tour of Machine Learning Algorithms : https://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/
- Guidelines to Select Machine Learning Scheme for Classification of Biomedical Datasets http://nexginrc.org/Publications/pub_files/evobio09-ajay.pdf
- A Tour of The Top 10 Algorithms for Machine Learning Newbies : https://towardsdatascience.com/a-tour-of-the-top-10-algorithms-for-machine-learning-newbies-dde4edffae11
- Machine Learning Algorithms: Which One to Choose for Your Problem : https://blog.statsbot.co/machine-learning-algorithms-183cc73197c
Yantra
Yantra (यन्त्र) (Sanskrit) (literally "machine, contraption" is a mystical diagram, mainly from the Tantric traditions of the Indian religions.
They are used for the worship of deities in temples or at home; as an aid in meditation; used for the benefits given by their supposed occult powers based on Hindu astrology and tantric texts. They are also used for adornment of temple floors, due mainly to their aesthetic and symmetric qualities. Specific yantras are traditionally associated with specific deities.
They are used for the worship of deities in temples or at home; as an aid in meditation; used for the benefits given by their supposed occult powers based on Hindu astrology and tantric texts. They are also used for adornment of temple floors, due mainly to their aesthetic and symmetric qualities. Specific yantras are traditionally associated with specific deities.
Yantras are usually associated with a particular deity and are used for specific benefits, such as: for meditation; protection from harmful influences; development of particular powers; attraction of wealth or success, etc.They are often used in daily ritual worship at home or in temples, and sometimes worn as a talisman.
As an aid to meditation, yantras represent the deity that is the object of meditation. These yantras emanate from the central point, the bindu. The yantra typically has several geometric shapes radiating concentrically from the center, including triangles, circles, hexagons, octagons, and symbolic lotus petals. The outside often includes a square representing the four cardinal directions, with doors to each of them
Yantras can be on a flat surface or three dimensional. Yantras can be drawn or painted on paper, engraved on metal, or any flat surface. They tend to be smaller in size than the similar mandala, and traditionally use less color than mandalas.
Occult yantras are used as good luck charms, to ward off evil, as preventative medicine, in exorcism, etc., by using their magical power. When used as a talisman, the yantra is seen to represent a deity who can be called on at will by the user. They are traditionally consecrated and energized by a priest, including the use of mantras which are closely associated to the specific deity and yantra. Practitioners believe that a yantra that is not energized with mantra is lifeless
A yantra comprises geometric shapes, images, and written mantra. Triangles and hexagrams are common, as are circles and lotuses of 4 to 1,000 petals.
Reference : https://en.wikipedia.org/wiki/Yantra
Image Courtesy : Desktop calendar of year 2017 by Ganesha Housing Corporation Limited
