Container Runtime


Low-Level container run time:  At their core, low-level container runtimes are responsible for 
* setting up these namespaces and cgroups for containers, 
* allow for setting resource limits on the cgroup, 
* setting up a root filesystem, 
* chrooting the container's process to the root file system, and 
* then running commands inside those namespaces and cgroups
using the standard Linux cgcreate, cgset, cgexec, chroot and unshare commands.

Example
1. runC is reference implementation of OCI specification https://github.com/opencontainers/runtime-spec sudo runc run mycontainerid
2. lxc
3. lmctfy by Google. It has the ability to run sub-task containers under a pre-allocated set of resources on a server, and thus achieve more stringent SLOs than could be provided by the runtime itself. It supports container hierarchies that use cgroup hierarchies via the container names


High-level container run time (= container run time) supports: 
* image management, 
* image transport
* image unpack
* pass image to low-level container run time to run it. 
* provide daemon application
* gRPC API. gRPC is a modern, open source, high-performance remote procedure call (RPC) framework 
* Web API. 

Example
1. containerd from docker. ctr is command line client for containerd. https://github.com/docker/containerd/blob/master/design/architecture.md
2. rkt by CoreOS 
3. cri-o

Relations

* rkt is a runtime containing both high level runtime and low level runtime. 
containerd and cri-o are on top of runC
* containerd does not have support for building container image. 
* rkt can build docker image, however does not provide remote API. 
* docker provides everything
* rkt can become alternative of docker. K8s can use rkt instead of docker. K8s can also use cri-o + runC instead of docker. 


Reference : https://www.youtube.com/watch?v=Utf-A4rODH8

Docker


Under the hood

Docker is programme written in Go

Docker manage kernel features (1) cgroups (2) namespace (3) copy-on-write COW file system

1. cgroups : limits the amount of resources (CPU, Memory) 
2. namespace = mount, IPC, net (IP address, route and iptables) , PID, user (user and group) and UTS (=Unix Time Sharing. Host name and domain name). 
Namespace API has 3 system calls I. clone II. unshare III. setns 

https://jvns.ca/blog/2016/10/10/what-even-is-a-container/

Docker needs Linux kernel version 3.10 or more. 

Docker is 2 programme = client + server. The server can be on remote machine

Docker creates bridges. Docker sets routes. docker uses iptables

Docker Dependencies

ca-certificates package contains certificates provided by the Certificate Authorities.


apt-transport-https allows the use of repositories accessed and downloaded via https protocol


software-properties-common provides some useful scripts for adding and removing PPAs + the DBUS backends. Without it, you would need to add and remove repositories (such as PPAs) manually by editing /etc/apt/sources.list and/or any subsidiary files in /etc/apt/sources.list.d


To add non-root user

sudo groupadd docker

sudo gpasswd -a docker
sudo usermod -aG docker
sudo setfacl -m user::rw /var/run/docker.sock

Docker Flow

docker run puts image to container

1. docker commit creates new image from container
Here commit will also tag image. default tag value is 'latest"
2. docker tag "old name" "new name may contain registry url"
Here "tag" is optional. 
Instead of 1, and 2 use single command
"docker commit "

docker image and container both have IDs. They are different. 


Docker run command

docker run --memory --cpu-share --cpu-quota -p [outside port:]"inside port"[/tcp|udp] --link --name -ti --privileged=true --net="host | some name" --ip "ip address" -v [absolute local path:] --restart=always --pid=host --env KEY=VALUE --volumes-from --rm -d [:tag default is latest]
ti = interactive terminal
rm = remove container, once it is done.
d = detach
-p here outside port is optional
-p :/ default is tcp
-P = --publish-all For exposing all ports
--link = it will add "ip address another container name" at /etc/hosts. It will auto detect IP address of another container. Assume the IP address does not change. 
-v for volume
--privileged to get full access of host OS
--pid will give more privilege to control PID of host OS
--restart : To restart if it dies. 
--net : To specify network name space

^d to exit container. 


https://docs.docker.com/engine/reference/commandline/run/

docker run = docker create + docker start

Attach and detach 


attach using

docker attach <container name>

to detach, when you are inside container, press ^p, ^q


Run process at container


docker -exec

Cannot add ports, volumes etc.
docker -exec -ti my_container sh -c "echo a && echo b"

Running more than one process or service in a container is discouraged for maximum efficiency & isolation. 

Logs


docker logs


Remove and Kill


docker kill

Container moves to stopped state. 

docker rm  

This will remove the container

docker ps -l

l = last running container. 

Network


docker port

It will list out external v/s internal ports. Same like iptables

For dynamic link, first create network
docker network create  

Then use --net while creating both networks.

here a name server will be added to network and it will take care of new ip address. 

If we specify --net=host then we can see, all the bridges inside container also with "brctl show" command

Docker images


docker images

List downloaded image. 

Docker Registry 

name = registry FQDN : port /organization name/image name:version tag

short name = organization name/image name


docker rmi


docker login
docker pull
docker push "name as per used in docker tag"
docker search

https://hub.docker.com/

Nexus

port 5000 for push, pull etc. 


https://docs.docker.com/registry/deploying/#copy-an-image-from-docker-hub-to-your-registry

The docker images can be stored at local host, docker, AWS, Google, Microsoft

docker save -o "tar.gz file" "one or more than one docker image:tag"
docker load -i "tar.gz file"

Volumes

virtual disc

persistent (-v) v/s ephemeral (--volumes-from)
not part of image

mount -o bind "original folder" "new name of folder. May be existing folder"


Host file system over guest can be mounter. Not vica versa. 

Dockerfile


No. Purpose docker file "docker run" command
1 Base image FROM
2 "Author" as output of "docker inspect" command MAINTAINER
3 Debug statement RUN
4 ADD "URL | local path" "container path"
5 During build time and run time ENV --env (or -e)KEY=VALUE
--env-file
6 It is like CMD. It is first programe to run inside container ENTRYPOINT --entrypoint It will override from DockerFile
7 CMD Last Argument is combination of ENTRYPOINT + CMD
8 Multi project file  COPY
9 sets working dir for build and for container. it is like CD WORKDIR --workdir
10 binary argument Shell Form
11 ["bianry", "argument"] Exec Form
12 Opening outgoing port at iptables firewall EXPOSE portNumber -p [outside port:]"inside port" P=publish. Publish is for outside network. Expose is for inside network.
13 Avoid local path VOLUME ["optional local path" "container path] -v and --volume-from
14 the container will run as user USER sudo

Reference : https://docs.docker.com/engine/reference/builder/ and https://docs.docker.com/engine/reference/commandline/docker/


Playground

https://labs.play-with-docker.com/


https://nickjanetakis.com/blog/a-linux-dev-environment-on-windows-with-wsl-docker-tmux-and-vscode
https://nickjanetakis.com/blog/using-wsl-and-mobaxterm-to-create-a-linux-dev-environment-on-windows

Questions

1. How to run xserver in container ?
2. How to connect using docker client to remote docker server?
3. how to expose my docker server on network? How to bind it to tcp socket instead of unix socket
4. Is LXD replacement of docker? 
https://linuxcontainers.org/lxd/introduction/
https://us.images.linuxcontainers.org/
https://linuxcontainers.org/lxd/getting-started-cli/

DevOps Resources


Introduction


DevOps approach includes automation and event monitoring at each stage of build. DevOps is set of practice to reduce time to deploy committed code to production. It is cross-function mode of working among development, QA and Operations team. Its variants are:

1. ArchOps for architecture
2. DataOps for data engineers
3. DevSecOps with security
4. WinOps for MicroSoft centric view. etc. 



Conferences

1. DevOpsDays : Low Cost. Bangalore also
2. DevOps Enterprise Summit (DOES) : High cost
3. Velocity.

Technology Specific events

1. ChefConf
2. PuppetConf
3. AWS re:invent
4. Monitorama
5. Surge
6. ScaleConf
7. Structure 

NewsLatter : https://www.devopsweekly.com/

Twitter Handles

@garethr
@devopsdotcom
@ashimmy
@botchgalupe
@damonedwards
@nathenharvey
@devopsmaster
@ernestmueller
@wickett
@iteration1

Zappa is serverless Python web services
https://github.com/Miserlou/Zappa
https://www.zappa.io/
Use case: Reporting system

Security in DevOps

Rugged Manifesto for secure coding
DevOps Audit Defense Toolkit
Threat Stack builds cloud security solution
gauntlt security testing
Signal Sciences is web application firewall
Alien Vault manages cyber attacks. 

Reliability Engineering : 3rd Pillar of DevOps


Reliability : system or component function under stated condition for specified period of time. It includes :

- Availability
- Performance
- Security

These should not be part of non-functional requirements. 

Key Areas of DevOps

1. Extending delivery to production
2. Extending feedback from operations to development (operate for design)
3. Embedding development into operations
4. Embedding operations into development

Dev comes from the school and Ops comes from the street. Reliability engineering = design for operate + operate for design

As per "Site Reliability Engineering" book  by Google, development team handles 100% and after the service reach maturity and stability 5% of operational workload 

Design for operation
  • "Design Pattern" by Gang of Four is for software design, design pattern and architecture. Similar design pattern book for stability is "Release it" by Michale Nygard. Another good book is "The Twelve Factor App" https://12factor.net/
  • Failure at integration point : Hystrix is open source library by Netflix that wrap a call to integration point with circuit breaker. 
  • config state should be separate from app code and store in environment variable 
  • Factorish github project to move legacy app to 12 factor app. 
  • Follow https://martinfowler.com/
  • Replication avoid single point of failures
  • Performance testing can be part of build pipeline. 
  • User profiler tool and APM (Application Performance Management) tools to locate performance bottleneck. 

Operate for Design

Monitoring

Monitoring metrics

  1. service performance & uptime
  2. software components metrics 
  3. system metrics (time series metrics about host) 
  4. app metrics
  5. performance
  6. security
    1. System security : Bad TLS/SSL settings, open ports, system configuration probelms
    2. Application security : XSS/SQL injection, custom events like password reset, invalid logins, new account creation. 
    3. Anomalies 
Monitoring Tools


1. Legacy tools : Nagios, Savics 
2. Simple endpoint monitoring : Pingdom, 
3. system and metric monitoring : Datadog, Netuitive, Ruxit, and Librato, 
4. full application performance management tools: New Relic and AppDynamics. 
5. Open Source tools : graphite, grafana, statsd, gangila, InfluxDB, OpenTSDB, mitrics.dropwizard.io
6. Open source solutions : icinga, sensu similar to nagios
7. container monitoring open source tools: prometheus, sysdig
8. security monitoring tools



Metrics
Avoid so many metrics. 

Logging

5 principles of logging

1. Do not collect log, that you will not use
2. Retain log data as long as it needed to retain by regulatory authority
3. Log all you can, but alert only when actions needed. Define log levels 
4. application availability and security is more needed, compare to logging availability and security 
5. Logs change. Format, content. Let's all take ownership of their log and keep at centralized system. 

Books related to Logging

1. Logging and Log Management
2. The practice of cloud system administration
3. Web Operations.

Log Management Tools

1. Legacy tool : Splunk
2. Open source Elk stack = Elastic search + log stashin + kibana
3. SaaS incident management tool :  Pagerduty and VictorOps
4. Open source : flapjack.io
5. statuspage.io
6. Command dispatcher tools : RunDeck, 

In addition to Monitoring, Metrics and Logging, few more tools for feedback. Incident command system, blameless postmortem and transparent uptime.

2018 : Looking back with smile


Happy New Year 2019

Let me continue the tradition, I started last year about reviewing the previous year. Year 2018. 


Spiritual/Religious: 

  • HDH Mahant Swmai Maharaj visited Banglaore. I attended divine PUJA at early morning and Satsang in His presence at evening. 
  • Attended Happiness programme (Part 1) by the Art of Living for the 6th time. It was part of Kannada Festival (RAJYOTSAV) in the month of  November. We all participants met Sri Sri Ravishankar at Ashram
  • Attended HDH Mahant Swami Maharaj's birthday celebration at Bhavnagar, in His divine presence. We also had a short trip to Pune.
  • I attended SHAKTI KRIYA. It is an ultimate process to start blossoming. Enjoyed :-)
  • I am now regular at daily SUDARSHAN KRIYA for last couple of months. 
  • I started again sending regular SMS about EKAADASHI fast, to interested people. 
  • I wrote a short blog about my DIKSHA Guru Pu. Go. Shri. Vallabharaiji Bava Shri (Jamnagar - Chopasani)
  • Attended an event on complete Bhagavad Gita chanting. 
Learning:
  • Participated at HackIV event in office and demonstrated Machine Learning based, image to HTML converter.  
  • Participated one more Hackathon event and worked on propitiatory edge computing IoT boards to integrate them with Alexa Skill Kit.  
  • Continue attending meetups about advanced technologies and captured, shared my notes at my personal blog.
  • I attended and enjoyed Python session by Luciano Ramalho. 
  • I also developed a small application along with my friend using Python to process Uni-code text. We were trying to analyze text from Bhagavad Gita. 
  • I continued some more experiments with development of Skills using Amazon Alexa Skill kit.
  • Also attended "Bengaluru Tech Summit" and writing key take away points at my blog. Somehow I missed Open Source event. 
  • This year I got opportunity to work on interesting projects in SON domain. I learnt more and enjoyed. 
Sanskrit
  • Started watching YouTube series by Dr. Baladevanand Sagar on topic "Learn Sanskrit, Be Modern !". 
  • We planned to present one translated BHAJAN/KIRTAN in Sanskrit to HDH Mahant Swmai Maharaj
  • Attended a competition on Gita Sloka chanting competition orgnaized by Samskrita Bharati. One Sanskrit dialogue from that programme : Watch here. 
  • Encouraged school students for Sanskrit by solving their doubts 
Entertainment: 
  • Enjoyed few more videos CDs 
  • Enjoyed Gazals from almost all A to Z albums of Jagjit Singh. Also filtered my favorite Gazals. 
  • Watched a documentary movie on Jagjit Singh by name "Woh Kagaz Ki Kasti"
  • Enjoyed one more movie "104 not out"
  • Enjoyed one Dance-cum-drama-cum-RamLila by name "Sri Ram"
  • As a birthday gift to myself, purchased Amazon Alexa Echo Dot. Explore it further for customized skill development. 
  • Also purchased and enjoyed my new Bluetooth speaker. 
Social Media
  • My blog is active now. Majority post are technical in nature. E.g. Node.JS,  OpenStack, Kubernetes, 5G NR, AWS, Python etc. I started series of blogs on DevOps. 
  • Updated my Linked-in profile
  • At WhatsApp now, I learnt and used to see status update of friends. 
  • As I learnt from Sri Sri Ravishankar at Happiness programme : One should utilize social media to spread more positive news. 
I made many new friends using QuickRide car pooling Android apps. Many interesting talks. I came to know about different projects, different companies, different culture in IT industry. I could utilize my travel time to listen to music, to listen to lectures and to meditate. 

Next Year 2019:
  • Spread positive news using Social Media. 
  • Contribute more at Github.
  • Re-Start conducting Free Spoken Sanskrit classes. 
  • Continue enriching my personal blog. The "DevOps" series will continue at the blog. A new series will be started about my personal notes on  "Machine Learning" course at coursera by Andrew NG. 
  • Participate at competitive platform like Kaggle. 
New hope, confidence to move forward, progress in spiritual front, personal front and professional front in this digital era. 

You may also like: 

2017: Looking back with smile