Reliability Engineering : 3rd Pillar of DevOps
Reliability : system or component function under stated condition for specified period of time. It includes :
- Availability
- Performance
- Security
These should not be part of non-functional requirements.
Key Areas of DevOps
1. Extending delivery to production
2. Extending feedback from operations to development (operate for design)
3. Embedding development into operations
4. Embedding operations into development
Dev comes from the school and Ops comes from the street. Reliability engineering = design for operate + operate for design
As per "Site Reliability Engineering" book by Google, development team handles 100% and after the service reach maturity and stability 5% of operational workload
Design for operation
Monitoring
Monitoring metrics
1. Legacy tools : Nagios, Savics
Metrics
Avoid so many metrics.
Logging
5 principles of logging
1. Do not collect log, that you will not use
2. Retain log data as long as it needed to retain by regulatory authority
3. Log all you can, but alert only when actions needed. Define log levels
4. application availability and security is more needed, compare to logging availability and security
5. Logs change. Format, content. Let's all take ownership of their log and keep at centralized system.
Books related to Logging
1. Logging and Log Management
2. The practice of cloud system administration
3. Web Operations.
Log Management Tools
1. Legacy tool : Splunk
2. Open source ELK stack = Elastic search + log stashin + kibana
3. SaaS incident management tool : Pagerduty and VictorOps
4. Open source : flapjack.io
5. statuspage.io
6. Command dispatcher tools : RunDeck,
In addition to Monitoring, Metrics and Logging, few more tools for feedback. Incident command system, blameless postmortem and transparent uptime.
- Availability
- Performance
- Security
These should not be part of non-functional requirements.
Key Areas of DevOps
1. Extending delivery to production
2. Extending feedback from operations to development (operate for design)
3. Embedding development into operations
4. Embedding operations into development
Dev comes from the school and Ops comes from the street. Reliability engineering = design for operate + operate for design
As per "Site Reliability Engineering" book by Google, development team handles 100% and after the service reach maturity and stability 5% of operational workload
Design for operation
- "Design Pattern" by Gang of Four is for software design, design pattern and architecture. Similar design pattern book for stability is "Release it" by Michale Nygard. Another good book is "The Twelve Factor App" https://12factor.net/
- Failure at integration point : Hystrix is open source library by Netflix that wrap a call to integration point with circuit breaker.
- config state should be separate from app code and store in environment variable
- Factorish github project to move legacy app to 12 factor app.
- Follow https://martinfowler.com/
- Replication avoid single point of failures
- Performance testing can be part of build pipeline.
- User profiler tool and APM (Application Performance Management) tools to locate performance bottleneck.
Monitoring
Monitoring metrics
- service performance & uptime
- software components metrics
- system metrics (time series metrics about host)
- app metrics
- performance
- security
- System security : Bad TLS/SSL settings, open ports, system configuration probelms
- Application security : XSS/SQL injection, custom events like password reset, invalid logins, new account creation.
- Anomalies
Monitoring Tools
2. Simple endpoint monitoring : Pingdom,
3. system and metric monitoring : Datadog, Netuitive, Ruxit, and Librato,
4. full application performance management tools: New Relic and AppDynamics.
5. Open Source tools : graphite, grafana, statsd, gangila, InfluxDB, OpenTSDB, mitrics.dropwizard.io
6. Open source solutions : icinga, sensu similar to nagios
7. container monitoring open source tools: prometheus, sysdig
8. security monitoring tools
3. system and metric monitoring : Datadog, Netuitive, Ruxit, and Librato,
4. full application performance management tools: New Relic and AppDynamics.
5. Open Source tools : graphite, grafana, statsd, gangila, InfluxDB, OpenTSDB, mitrics.dropwizard.io
6. Open source solutions : icinga, sensu similar to nagios
7. container monitoring open source tools: prometheus, sysdig
8. security monitoring tools
Avoid so many metrics.
Logging
5 principles of logging
1. Do not collect log, that you will not use
2. Retain log data as long as it needed to retain by regulatory authority
3. Log all you can, but alert only when actions needed. Define log levels
4. application availability and security is more needed, compare to logging availability and security
5. Logs change. Format, content. Let's all take ownership of their log and keep at centralized system.
Books related to Logging
1. Logging and Log Management
2. The practice of cloud system administration
3. Web Operations.
Log Management Tools
1. Legacy tool : Splunk
2. Open source ELK stack = Elastic search + log stashin + kibana
3. SaaS incident management tool : Pagerduty and VictorOps
4. Open source : flapjack.io
5. statuspage.io
6. Command dispatcher tools : RunDeck,
In addition to Monitoring, Metrics and Logging, few more tools for feedback. Incident command system, blameless postmortem and transparent uptime.
2018 : Looking back with smile
Happy New Year 2019
Let me continue the tradition, I started last year about reviewing the previous year. Year 2018.
Spiritual/Religious:
Entertainment:
Next Year 2019:
You may also like:
2017: Looking back with smile
Let me continue the tradition, I started last year about reviewing the previous year. Year 2018.
Spiritual/Religious:
- HDH Mahant Swmai Maharaj visited Banglaore. I attended divine PUJA at early morning and Satsang in His presence at evening.
- Attended Happiness programme (Part 1) by the Art of Living for the 6th time. It was part of Kannada Festival (RAJYOTSAV) in the month of November. We all participants met Sri Sri Ravishankar at Ashram
- Attended HDH Mahant Swami Maharaj's birthday celebration at Bhavnagar, in His divine presence. We also had a short trip to Pune.
- I attended SHAKTI KRIYA. It is an ultimate process to start blossoming. Enjoyed :-)
- I am now regular at daily SUDARSHAN KRIYA for last couple of months.
- I started again sending regular SMS about EKAADASHI fast, to interested people.
- I wrote a short blog about my DIKSHA Guru Pu. Go. Shri. Vallabharaiji Bava Shri (Jamnagar - Chopasani)
- Attended an event on complete Bhagavad Gita chanting.
Learning:
- Participated at HackIV event in office and demonstrated Machine Learning based, image to HTML converter.
- Participated one more Hackathon event and worked on propitiatory edge computing IoT boards to integrate them with Alexa Skill Kit.
- Continue attending meetups about advanced technologies and captured, shared my notes at my personal blog.
- I attended and enjoyed Python session by Luciano Ramalho.
- I also developed a small application along with my friend using Python to process Uni-code text. We were trying to analyze text from Bhagavad Gita.
- I continued some more experiments with development of Skills using Amazon Alexa Skill kit.
- Also attended "Bengaluru Tech Summit" and writing key take away points at my blog. Somehow I missed Open Source event.
- This year I got opportunity to work on interesting projects in SON domain. I learnt more and enjoyed.
Sanskrit
- Started watching YouTube series by Dr. Baladevanand Sagar on topic "Learn Sanskrit, Be Modern !".
- We planned to present one translated BHAJAN/KIRTAN in Sanskrit to HDH Mahant Swmai Maharaj
- Attended a competition on Gita Sloka chanting competition orgnaized by Samskrita Bharati. One Sanskrit dialogue from that programme : Watch here.
- Encouraged school students for Sanskrit by solving their doubts
- Enjoyed few more videos CDs
- Enjoyed Gazals from almost all A to Z albums of Jagjit Singh. Also filtered my favorite Gazals.
- Watched a documentary movie on Jagjit Singh by name "Woh Kagaz Ki Kasti"
- Enjoyed one more movie "104 not out"
- Enjoyed one Dance-cum-drama-cum-RamLila by name "Sri Ram"
- As a birthday gift to myself, purchased Amazon Alexa Echo Dot. Explore it further for customized skill development.
- Also purchased and enjoyed my new Bluetooth speaker.
- My blog is active now. Majority post are technical in nature. E.g. Node.JS, OpenStack, Kubernetes, 5G NR, AWS, Python etc. I started series of blogs on DevOps.
- Updated my Linked-in profile
- At WhatsApp now, I learnt and used to see status update of friends.
- As I learnt from Sri Sri Ravishankar at Happiness programme : One should utilize social media to spread more positive news.
I made many new friends using QuickRide car pooling Android apps. Many interesting talks. I came to know about different projects, different companies, different culture in IT industry. I could utilize my travel time to listen to music, to listen to lectures and to meditate.
Next Year 2019:
- Spread positive news using Social Media.
- Contribute more at Github.
- Re-Start conducting Free Spoken Sanskrit classes.
- Continue enriching my personal blog. The "DevOps" series will continue at the blog. A new series will be started about my personal notes on "Machine Learning" course at coursera by Andrew NG.
- Participate at competitive platform like Kaggle.
You may also like:
2017: Looking back with smile
Effective PPT
The goal for any presentation can be to create slides that lead to yes !
Yes for question about Why, How and What
Design Aspect
Content of PPT
Data
1. Relationship
Bubble chart (2 variables)
Scatter chart (3 variables)
2. Composition
2.1 Changing over time
Stacked 100% column chart
Stacked column chart
Stacked 100% area chart
Stacked area chart
2.2 static
Pie chart
Waterfall chart
Stacked 100% column chart with subcomponents
3. Distribution
Histogram (1 variable few items)
Line Histogram (1 variable many items)
Scatter Chart (2 variable)
3D Area chart (3 variable)
4. Comparison
Cyclic Area Chart
Line Chart
Column Chart
Single Bar Chart
Variable Width Column Chart
Reference Books
1. "brain rules" by John Medina
2. slide:ology
3. "Presentation Zen" by Garr Reynolds
4. "PresentationZen Design" by Garr Reynolds
5. "The Wall Street Journal Guide to Information Graphics" by Wong, Dona M.
Yes for question about Why, How and What
- The audience can retain limited information. The audience want to listen to you, instead of too much content on PPT slides.
- Show important points with image. Human process 60,000 x faster images compare to text. We can also recall information from image 6x time more easily compare to test.
Design Aspect
- Consistent font, font size, line length, spacing, margin, text alignment, kerning. For consistent arrangements of components, grid functionality can be used.
- Arial and Verdana fonts are better compare to Times New Roman. Avoid ALL CAPS
- Use any and only one highlighting method
- Simple and visually appealing slide is better compare to a cluttered slide to avoid distract and confusion. Avoid picture for decoration. Avoid too much of 3D, clip arts, word art, photos, complex visuals, lengthy text and animations.
- Add logo, slide number and copyright notice
- Select picture that support your message and story line in graphs.
Content of PPT
- Organize points in groups and use SmartArt
- Use concise slide title
- Use brand color as primary color and avoid using so many different colors. Maintain contrast between foreground and background color.
Data
- Use data to highlight important points
- Use table with detail view and summary view (total) both. Highlight key values.
- Mention the source of data
1. Relationship
Bubble chart (2 variables)
Scatter chart (3 variables)
2. Composition
2.1 Changing over time
Stacked 100% column chart
Stacked column chart
Stacked 100% area chart
Stacked area chart
2.2 static
Pie chart
Waterfall chart
Stacked 100% column chart with subcomponents
3. Distribution
Histogram (1 variable few items)
Line Histogram (1 variable many items)
Scatter Chart (2 variable)
3D Area chart (3 variable)
4. Comparison
Cyclic Area Chart
Line Chart
Column Chart
Single Bar Chart
Variable Width Column Chart
Reference Books
1. "brain rules" by John Medina
2. slide:ology
3. "Presentation Zen" by Garr Reynolds
4. "PresentationZen Design" by Garr Reynolds
5. "The Wall Street Journal Guide to Information Graphics" by Wong, Dona M.
Bengaluru Tech Summit : Day 3 - Part 1
We had interesting panel discussion on the topic Innovations driving next generation enterprise
Trends
* Now, the personal computing is more affordable compare to 1960s and 1980s due to to reduction is computing cost, storage cose and emergence of open source culture.
* In 1990s all the computers are interconnected and the Internet emerged along with new business models for finance/banking, retail e-commerce and e-mail. New enterprises also got emerged. Today we are living in digital age or mobile age.
* The people who born around 1995, have entered as work force around 2015. Generation Z, how to influence these 2 billion people, with different set of expectations.
* IoT
* Emergence of rural area. Now rural people also have access to information.
* Now people feel pride to work at startups.
* A new startup can be started just using WhatsApp groups.
* Supportive governments and Law. Few examples of few funny laws, that exist even today. Laws are reaction to 150 years of industrial revolution. The government should not stop the growth. So Karnatak is first state in India with regulatory sandbox, where you are allowed to break (only state) law for duration ranges from 3 months to 2 years, powered by Karnataka Innovation Authority. More can be achieve on startup front with de-regularization like allowing crypto currency.
* All the new innovations are becoming necessities.
* Big giant companies like IBM are supporting startups by finance, by connecting them to large customer base, scaling up. They cannot continue with their own R&D center so they come to startups. Block chain based virtual idea market may emerged in next few months.
* Online transactions will ultimately eliminate the need for banks.
* Education: 500 Mbps bandwidth at each home, will eliminate universities. all students will study online. Then even 6th standard students will talk about Artificial Intelligence, Machine Learning, IoT, Block Chain etc.
* The rule based and law based audit will change. Already after GST and demonetization the B.Com degree has more value compare to BBA or BBM. Now working people need to play different roles to remain in job market. Innovations at large enterprises are reducing jobs. On the other hand, same innovation increases the jobs at startups.
The factors that drives innovations are changed based on above trends. The generation z is consumer as well as is at startups, so disruption is bound to happen. Now all enterprise needs innovation to survive or to make more profit in areas of their products, service and/or process.
Discovery v/s Invention v/s Innovation
We know discovery is about something that already exists. Invention is one step further, by combining multiple few discoveries. Innovation goes even further by combining multiple inventions and build a use case to satisfy needs. The startups are all about building faster, better, cheaper solutions with help of inventions. Today major innovations are around (1) Artificial Intelligence, (2) data science and (3) block chain. Again startups need to focus on usage rather than the technology. 90% of startups fail due to lecuna about how to take it forward. All entrepreneurs cannot scale up their venture. Startups are risky, but design driven approach, reduces the risk.
It is not important, what you study (in college/school to get your degree). The important aspect is what you continue to study. Today everyone should know, how to learn new topics, unlearn old topics and re-learn as per emerging trends.
Bengaluru Tech Summit : Day 2
There was an interesting experiment on a patient, who daily forgets her past life. Everyday, she introduce herself, shake hands like meeting for the first time. The scientist doctor, started hurting her with small pin, during shake hand. Just in few days, she stopped shake hands with the scientist, even though she could not recover any memory associated about him. There are many research on topics of fear, fact, emotions, memory etc.
Brain Research in the age of information technology
As per Moravec's paradox, what we human beings find very easy is difficult for computers and what is easy for computers, that we find the most difficult. Like computer compare two images, pixel by pixel and so it cannot identify two objects are same, if the images are from different angle. On other way, two images of up-side-down faces, we human beings find identical, but when we rotate 180 degree, we realize they are different. It is easy for computer to tell, that they are different, as it compare pixel by pixel. Today 80 % of object detection is done by AI/ML.The 40% of our brain are is involved in visual processing. The brains does not look at brightness of pixel. We also discussed about famous image, about a dress, where a group people says white and blue, while others say gold and black. The color depends on individual perception about color of light falling on that dress white or yellow. Symmetry in deep learning network, increases accuracy from 1 to 10 %. Few initiatives:
* OpenAI is a non-profit AI research company, discovering and enacting the path to safe artificial general intelligence. Many projects by OpenAI on github.
* VisionLab@IISc, Bangalore. Website and Facebook page
Ashwini Godbole emphasize on trans-disciplinary approach of brain health as per Ayurveda and modern biomedicine for brain. She also explained about clinic to lab approach. She mentioend about meaningful eating.
Brain Research in the age of information technology
As per Moravec's paradox, what we human beings find very easy is difficult for computers and what is easy for computers, that we find the most difficult. Like computer compare two images, pixel by pixel and so it cannot identify two objects are same, if the images are from different angle. On other way, two images of up-side-down faces, we human beings find identical, but when we rotate 180 degree, we realize they are different. It is easy for computer to tell, that they are different, as it compare pixel by pixel. Today 80 % of object detection is done by AI/ML.The 40% of our brain are is involved in visual processing. The brains does not look at brightness of pixel. We also discussed about famous image, about a dress, where a group people says white and blue, while others say gold and black. The color depends on individual perception about color of light falling on that dress white or yellow. Symmetry in deep learning network, increases accuracy from 1 to 10 %. Few initiatives:
* OpenAI is a non-profit AI research company, discovering and enacting the path to safe artificial general intelligence. Many projects by OpenAI on github.
* VisionLab@IISc, Bangalore. Website and Facebook page
Ashwini Godbole emphasize on trans-disciplinary approach of brain health as per Ayurveda and modern biomedicine for brain. She also explained about clinic to lab approach. She mentioend about meaningful eating.
