Infrastructure automation : 1st DevOps Pillar


Operation team should use source control

Define your infrastructure as model or drive it using RESTful API

Model OR Recipes  = Hardware + OS(Operating System) + OS Dependencies + System Configuration + Accounts + SSL Certs + Application

Important terms

Imperative (procedural) Approach where commands desired to produce a state are defined and then executed. E.g. BASH Shell script, AWS CLI, Python Boto library 
* declarative (functional) Approach where you define the desired state, and the tool converges the existing system on the model. E.g. Makefile, Puppet manifest, SQL Queries.

declarative is more efficient.

* Convergent : CM automation that converges the system to desired state. 
* Idempotent : The ability to execute the CM procedure repeatedly. And end up in the same state each time. The second run will not make any change. 
* Immutable : Not change after deployment. Can be redeploy if needed. E.g. Docker container. 

* Self service : The ability for an end user to kick off one of these processes without having to go through other people

http://www.infrastructures.org/

Configuration Management 

* Provisioning : Server ready with hardware, OS, system services, network 

Two types of provisioning : Model based provisioning and Convergent Driver Provisioning

1. Model based provisioning 

1.1 declarative model: n/w, node, db etc. E.g. using template YAML
or imperative manner, step by step 

Tools

* ColudFormation JSON file for AWS
* Azure Resource Manager template for Azure
HashiCorp's Terraform for multiple cloud providers. It maintains external state store 

* Ubuntu's Juju = Infra + services (common in docker) 
Vagrant tool emulate production environment. 
** Deployment : Automatic deploy / upgrade application on server
** Orchestration : Coordinated operations across multiple systems
Provisioners and providers are building blocks of Vagrant. 
1. Vagrant has provisioners like Puppe, Chef, Ansible etc. 
2. Vagrant has Providers like VirtualBox, Hyper-V, Docker. VMWare and AWS via plugin. 

vagrantfile

commands
vagrant destroy
vagrant status

2. Convergent Driver Provisioning. 

It is declarative and idempotent 
It handles orchestration by state tracking and master registry. Each node reports system information to the registry. 

Chef components
1. Run-list
2. Roles
3. Recipes belongs  to cookbook. 

Cookbook public repository (Chef): https://supermarket.chef.io/
Puppet public repository : Puppet Forge https://forge.puppet.com/

purpose of Ohai is to provide node data to Chef-client, to use within cookbooks. 

commands
kitchen converge
kitchen list
kitchen destroy

Tools

Chef 
     Lint: Foodcritic 
     Unit Testing : Chefspec
     Integration Testing : KitchenCI
Puppet
Ansible
Salt
CFEngine
Packer
Shell Provisioner 

Evolution of provisioning tools

Adhoc script OR Capistrano framework by developers to push changes. 

Provisioning Tools: CFEngine to Puppet to Chef , 
Commercial IT Provisioning tool : Ghost, 
Enterprise tool : HP

Then Golden Image or Foil Ball : This result in (1) image sprawl or (2) configuration drift. So stem cell system approach with minimum initial provisioning and update later on : Chef DSL, 

Capistrano to RunDeck

Rundeck is a container, which listen to on 4440

docker run -p 4440:4440 -e SERVER_URL=http://localhost:4440 --name AnyName -t /jordan/rundeck:latest

Orchestrated deployment techniques

* Canary Deployment
* Blue / Green Deployment
* Cluster immune system deployment
* Immutable deployment (1) for Container  = OS + Dependency + App code (2) not for data store

Artifact : 
JAR to WAR to 
 -Deb
 -RPM
to 
 - Amazon Machine Image (AMI for amazon) 
 - VM image (using packer)
 - Docker image (using docker file) 

Maven's jdub plugins: for Debian image
FPM : deb and RPM package etc. format

Artifact Repository
* Nexus
* JFrog Artifactory
* Apache Archiva 

Testing: 

- Unit (includes formatters and linters)
- Integration
- Security

We can include tools related to formatter and linter too as part of CUT (coding and Unit Testing) phase for developer. 

Formatter : Rubocop. It is for Ruby. Chef and Puppet are on top of Ruby

Linter: Foodcritic for Chef

Unit Test : ChefSpec is extension of RSpec framework (for Ruby) 

Integration Test : Serverspec (written in RSpec). Before that we run TestKitchen (kitchen converge) then Serverspec (kitchen verify)

Security Test: InSpec is audit based tool from Chef (kitchen verify) We can get (1) CIS hardening compliance and (2) visual reports. Another tool Gauntlt perform cyber attack at system

CMDB : 
  • DB 
  • then Puppet + Chef gave node registry 
  • then ZooKeeper from Haddop 
  • then etcd  and HashiCorp's Consul emerged. They powers Mesos, Swarm and Kubernetes.  
  • Tumblr's Collins project
  • Chef + Ohai = searchable CMDB
Service Discovery Tools

* etcd
* ZooKeepr
* Consul

Container As a Services

* Rancher
* Google Cloud Platform
* AWS ECS

Habitat : Application automation = Build + Deploy. It extends Chef. 

Usecase

Just Recommendations:

1. Serverless : Basic needs. 
2. container, docker, k8s : for medium complex system
3. Chef or Puppet: large distributed environment, network with heterogeneous systems and variety of apps (convergence model) 
4. Ansible Orchestration:  Website with high uptime requirement 
5. Rundeck with Puppet: Update with coordination with other services. 

Cloud

SaaS : TurboTax, Office365 Salesforce
PaaS: Google App Engine, Microsoft Azure App Service, Pivotal Web Service
IaaS: Google Compute Engine, Microsoft Azure VM, AWS EC2
FaaS : Amazon Lambda, Google Cloud Function, Azure Function

Hypervisor : VMWare, ZeN, Hyper-V, VirtualBox

Tools for cloud: 

1. Hashicorp's Packer to create images

2. OpenStack

3. CloudFoundry by Pivotal

4. Open Source tools : MAAS (for Ubuntu's Juju), cobbler, Hanlon, helps in Pixie n/w booting 

5. Then provisioning by Chef, Puppet, VMWare vCenter

6. Foreman is based on Puppet framework

7. Crowbar (Now Zeus cloud) : OpenStack and Hadoop installation 

8. digital_REBAR : Provisioning + Orchestration 

0 comments:

Post a Comment