Infrastructure automation : 1st DevOps Pillar
Operation team should use source control
Define your infrastructure as model or drive it using RESTful API
Model OR Recipes = Hardware + OS(Operating System) + OS Dependencies + System Configuration + Accounts + SSL Certs + Application
Important terms
* Imperative (procedural) Approach where commands desired to produce a state are defined and then executed. E.g. BASH Shell script, AWS CLI, Python Boto library
* declarative (functional) Approach where you define the desired state, and the tool converges the existing system on the model. E.g. Makefile, Puppet manifest, SQL Queries.
declarative is more efficient.
* Convergent : CM automation that converges the system to desired state.
* Idempotent : The ability to execute the CM procedure repeatedly. And end up in the same state each time. The second run will not make any change.
* Immutable : Not change after deployment. Can be redeploy if needed. E.g. Docker container.
* Self service : The ability for an end user to kick off one of these processes without having to go through other people
http://www.infrastructures.org/
Configuration Management
* Provisioning : Server ready with hardware, OS, system services, network
Two types of provisioning : Model based provisioning and Convergent Driver Provisioning
1. Model based provisioning
1.1 declarative model: n/w, node, db etc. E.g. using template YAML
or imperative manner, step by step
Tools
* ColudFormation JSON file for AWS
* Azure Resource Manager template for Azure
* HashiCorp's Terraform for multiple cloud providers. It maintains external state store
* Ubuntu's Juju = Infra + services (common in docker)
** Orchestration : Coordinated operations across multiple systems
Provisioners and providers are building blocks of Vagrant.
1. Vagrant has provisioners like Puppe, Chef, Ansible etc.
2. Vagrant has Providers like VirtualBox, Hyper-V, Docker. VMWare and AWS via plugin.
vagrantfile
commands
vagrant destroy
vagrant status
2. Convergent Driver Provisioning.
It is declarative and idempotent
It handles orchestration by state tracking and master registry. Each node reports system information to the registry.
Chef components
1. Run-list
2. Roles
3. Recipes belongs to cookbook.
Cookbook public repository (Chef): https://supermarket.chef.io/
Puppet public repository : Puppet Forge https://forge.puppet.com/
purpose of Ohai is to provide node data to Chef-client, to use within cookbooks.
commands
kitchen converge
kitchen list
kitchen destroy
Tools
Chef
Lint: Foodcritic
Unit Testing : Chefspec
Integration Testing : KitchenCI
Puppet
Ansible
Salt
CFEngine
Packer
Shell Provisioner
Evolution of provisioning tools
Adhoc script OR Capistrano framework by developers to push changes.
Provisioning Tools: CFEngine to Puppet to Chef ,
Commercial IT Provisioning tool : Ghost,
Enterprise tool : HP
Then Golden Image or Foil Ball : This result in (1) image sprawl or (2) configuration drift. So stem cell system approach with minimum initial provisioning and update later on : Chef DSL,
Capistrano to RunDeck
Rundeck is a container, which listen to on 4440
docker run -p 4440:4440 -e SERVER_URL=http://localhost:4440 --name AnyName -t /jordan/rundeck:latest
Orchestrated deployment techniques
* Canary Deployment
* Blue / Green Deployment
* Cluster immune system deployment
* Immutable deployment (1) for Container = OS + Dependency + App code (2) not for data store
Artifact :
JAR to WAR to
-Deb
-RPM
to
- Amazon Machine Image (AMI for amazon)
- VM image (using packer)
- Docker image (using docker file)
Maven's jdub plugins: for Debian image
FPM : deb and RPM package etc. format
Artifact Repository
* Nexus
* JFrog Artifactory
* Apache Archiva
Testing:
- Unit (includes formatters and linters)
- Integration
- Security
We can include tools related to formatter and linter too as part of CUT (coding and Unit Testing) phase for developer.
Formatter : Rubocop. It is for Ruby. Chef and Puppet are on top of Ruby
Linter: Foodcritic for Chef
Unit Test : ChefSpec is extension of RSpec framework (for Ruby)
Integration Test : Serverspec (written in RSpec). Before that we run TestKitchen (kitchen converge) then Serverspec (kitchen verify)
Security Test: InSpec is audit based tool from Chef (kitchen verify) We can get (1) CIS hardening compliance and (2) visual reports. Another tool Gauntlt perform cyber attack at system
CMDB :
* etcd
* ZooKeepr
* Consul
Container As a Services
* Rancher
* Google Cloud Platform
* AWS ECS
Habitat : Application automation = Build + Deploy. It extends Chef.
Usecase
Just Recommendations:
1. Serverless : Basic needs.
2. container, docker, k8s : for medium complex system
3. Chef or Puppet: large distributed environment, network with heterogeneous systems and variety of apps (convergence model)
4. Ansible Orchestration: Website with high uptime requirement
5. Rundeck with Puppet: Update with coordination with other services.
Cloud
SaaS : TurboTax, Office365 Salesforce
PaaS: Google App Engine, Microsoft Azure App Service, Pivotal Web Service
IaaS: Google Compute Engine, Microsoft Azure VM, AWS EC2
FaaS : Amazon Lambda, Google Cloud Function, Azure Function
Hypervisor : VMWare, ZeN, Hyper-V, VirtualBox
Tools for cloud:
1. Hashicorp's Packer to create images
2. OpenStack
3. CloudFoundry by Pivotal
4. Open Source tools : MAAS (for Ubuntu's Juju), cobbler, Hanlon, helps in Pixie n/w booting
5. Then provisioning by Chef, Puppet, VMWare vCenter
6. Foreman is based on Puppet framework
7. Crowbar (Now Zeus cloud) : OpenStack and Hadoop installation
8. digital_REBAR : Provisioning + Orchestration
Define your infrastructure as model or drive it using RESTful API
Model OR Recipes = Hardware + OS(Operating System) + OS Dependencies + System Configuration + Accounts + SSL Certs + Application
Important terms
* Imperative (procedural) Approach where commands desired to produce a state are defined and then executed. E.g. BASH Shell script, AWS CLI, Python Boto library
* declarative (functional) Approach where you define the desired state, and the tool converges the existing system on the model. E.g. Makefile, Puppet manifest, SQL Queries.
declarative is more efficient.
* Convergent : CM automation that converges the system to desired state.
* Idempotent : The ability to execute the CM procedure repeatedly. And end up in the same state each time. The second run will not make any change.
* Immutable : Not change after deployment. Can be redeploy if needed. E.g. Docker container.
* Self service : The ability for an end user to kick off one of these processes without having to go through other people
http://www.infrastructures.org/
Configuration Management
* Provisioning : Server ready with hardware, OS, system services, network
Two types of provisioning : Model based provisioning and Convergent Driver Provisioning
1. Model based provisioning
1.1 declarative model: n/w, node, db etc. E.g. using template YAML
or imperative manner, step by step
Tools
* ColudFormation JSON file for AWS
* Azure Resource Manager template for Azure
* HashiCorp's Terraform for multiple cloud providers. It maintains external state store
* Ubuntu's Juju = Infra + services (common in docker)
* Vagrant tool emulate production environment.
** Deployment : Automatic deploy / upgrade application on server** Orchestration : Coordinated operations across multiple systems
Provisioners and providers are building blocks of Vagrant.
1. Vagrant has provisioners like Puppe, Chef, Ansible etc.
2. Vagrant has Providers like VirtualBox, Hyper-V, Docker. VMWare and AWS via plugin.
vagrantfile
commands
vagrant destroy
vagrant status
2. Convergent Driver Provisioning.
It is declarative and idempotent
It handles orchestration by state tracking and master registry. Each node reports system information to the registry.
Chef components
1. Run-list
2. Roles
3. Recipes belongs to cookbook.
Cookbook public repository (Chef): https://supermarket.chef.io/
Puppet public repository : Puppet Forge https://forge.puppet.com/
purpose of Ohai is to provide node data to Chef-client, to use within cookbooks.
commands
kitchen converge
kitchen list
kitchen destroy
Tools
Chef
Lint: Foodcritic
Unit Testing : Chefspec
Integration Testing : KitchenCI
Puppet
Ansible
Salt
CFEngine
Packer
Shell Provisioner
Evolution of provisioning tools
Adhoc script OR Capistrano framework by developers to push changes.
Provisioning Tools: CFEngine to Puppet to Chef ,
Commercial IT Provisioning tool : Ghost,
Enterprise tool : HP
Then Golden Image or Foil Ball : This result in (1) image sprawl or (2) configuration drift. So stem cell system approach with minimum initial provisioning and update later on : Chef DSL,
Capistrano to RunDeck
Rundeck is a container, which listen to on 4440
docker run -p 4440:4440 -e SERVER_URL=http://localhost:4440 --name AnyName -t /jordan/rundeck:latest
Orchestrated deployment techniques
* Canary Deployment
* Blue / Green Deployment
* Cluster immune system deployment
* Immutable deployment (1) for Container = OS + Dependency + App code (2) not for data store
Artifact :
JAR to WAR to
-Deb
-RPM
to
- Amazon Machine Image (AMI for amazon)
- VM image (using packer)
- Docker image (using docker file)
Maven's jdub plugins: for Debian image
FPM : deb and RPM package etc. format
Artifact Repository
* Nexus
* JFrog Artifactory
* Apache Archiva
Testing:
- Unit (includes formatters and linters)
- Integration
- Security
We can include tools related to formatter and linter too as part of CUT (coding and Unit Testing) phase for developer.
Formatter : Rubocop. It is for Ruby. Chef and Puppet are on top of Ruby
Linter: Foodcritic for Chef
Unit Test : ChefSpec is extension of RSpec framework (for Ruby)
Integration Test : Serverspec (written in RSpec). Before that we run TestKitchen (kitchen converge) then Serverspec (kitchen verify)
Security Test: InSpec is audit based tool from Chef (kitchen verify) We can get (1) CIS hardening compliance and (2) visual reports. Another tool Gauntlt perform cyber attack at system
CMDB :
- DB
- then Puppet + Chef gave node registry
- then ZooKeeper from Haddop
- then etcd and HashiCorp's Consul emerged. They powers Mesos, Swarm and Kubernetes.
- Tumblr's Collins project
- Chef + Ohai = searchable CMDB
* etcd
* ZooKeepr
* Consul
Container As a Services
* Rancher
* Google Cloud Platform
* AWS ECS
Habitat : Application automation = Build + Deploy. It extends Chef.
Usecase
Just Recommendations:
1. Serverless : Basic needs.
2. container, docker, k8s : for medium complex system
3. Chef or Puppet: large distributed environment, network with heterogeneous systems and variety of apps (convergence model)
4. Ansible Orchestration: Website with high uptime requirement
5. Rundeck with Puppet: Update with coordination with other services.
Cloud
SaaS : TurboTax, Office365 Salesforce
PaaS: Google App Engine, Microsoft Azure App Service, Pivotal Web Service
IaaS: Google Compute Engine, Microsoft Azure VM, AWS EC2
FaaS : Amazon Lambda, Google Cloud Function, Azure Function
Hypervisor : VMWare, ZeN, Hyper-V, VirtualBox
Tools for cloud:
1. Hashicorp's Packer to create images
2. OpenStack
3. CloudFoundry by Pivotal
4. Open Source tools : MAAS (for Ubuntu's Juju), cobbler, Hanlon, helps in Pixie n/w booting
5. Then provisioning by Chef, Puppet, VMWare vCenter
6. Foreman is based on Puppet framework
7. Crowbar (Now Zeus cloud) : OpenStack and Hadoop installation
8. digital_REBAR : Provisioning + Orchestration
0 comments:
Post a Comment