Real-Time Kubernetes Debugging, Monitoring and Alerting with BotKube
Ref: https://www.meetup.com/k8s-cloudnative-online/events/269706847
My take away points
Real-Time Kubernetes Debugging
1 Describe
* describe pod and see event
* describe service and see endpoint
2 Events
* kubectl get events
gives events in that namespace
* kubectl get events --all-namespace
3 exec with -it (interactive terminal) for pod. Use -c to run inside specific container.
4. kubectl debug is K8s 1.18 alpha feature. First enable alpha feature. We can dynamically insert container in pod, for debug purpose.
5. Logs
* kubectl logs,
Container shall expose its logs using stdout
* kubectl logs --previous
provides log for when it run last time. It is useful if any pod is keep restating.
Grafana Dashboard: https://grafana.com/grafana/dashboards?orderBy=name&direction=asc
For K8s + Prometheus setup, relevant dashboards are : https://grafana.com/grafana/dashboards?dataSource=prometheus&direction=asc&orderBy=name&search=kubernetes
Select one dashboard. import in you Grafana with its ID. OR one can download JSON file and import the JSON file
BotKube
BotKube is for debugging and monitoring
it can run kubectl also
kubectl get clusterroles
give all who have cluster role
BotKube can work with multiple cluster
We can add custom filter in GoLang to BotKube: https://www.botkube.io/filters/
BotKube next release
- It will be integrated with Microsoft Team
- It will have json path based status:field monitoring
Reference:
https://github.com/infracloudio/botkube
https://www.botkube.io
From Live Chat
* if service has two label and pod has only 1 out of 2 labels then also that pod is part of service due to OR condition.
* deployments are not associated with any node, so one cannot get node IP address from deployment
* Better go with Prometheus Operator Model that is a part of Kube-Prometheus project. It'll install node-exporter that is a daemonset. It'll act as a agent for nodes
* Prometheus can work with docker container also, without K8s
* splunk and kibana both can be combined.
* https://blogs.halodoc.io/production-grade-eks-logging-with-efk-via-sidecar/
* how scaling is designed for botkube ? once is a while, there can be many events from K8s cluster . Most of the time, there are no events from k8s cluster.
Scaling is using K8s HPA only. Even we have so many events, each event processing needs little CPU.
Meetup Recording : https://www.youtube.com/watch?v=bGnQep5bY6c
My take away points
Real-Time Kubernetes Debugging
1 Describe
* describe pod and see event
* describe service and see endpoint
2 Events
* kubectl get events
gives events in that namespace
* kubectl get events --all-namespace
3 exec with -it (interactive terminal) for pod. Use -c to run inside specific container.
4. kubectl debug is K8s 1.18 alpha feature. First enable alpha feature. We can dynamically insert container in pod, for debug purpose.
5. Logs
* kubectl logs,
Container shall expose its logs using stdout
* kubectl logs --previous
provides log for when it run last time. It is useful if any pod is keep restating.
Grafana Dashboard: https://grafana.com/grafana/dashboards?orderBy=name&direction=asc
For K8s + Prometheus setup, relevant dashboards are : https://grafana.com/grafana/dashboards?dataSource=prometheus&direction=asc&orderBy=name&search=kubernetes
Select one dashboard. import in you Grafana with its ID. OR one can download JSON file and import the JSON file
BotKube
BotKube is for debugging and monitoring
it can run kubectl also
kubectl get clusterroles
give all who have cluster role
BotKube can work with multiple cluster
We can add custom filter in GoLang to BotKube: https://www.botkube.io/filters/
BotKube next release
- It will be integrated with Microsoft Team
- It will have json path based status:field monitoring
Reference:
https://github.com/infracloudio/botkube
https://www.botkube.io
From Live Chat
* if service has two label and pod has only 1 out of 2 labels then also that pod is part of service due to OR condition.
* deployments are not associated with any node, so one cannot get node IP address from deployment
* Better go with Prometheus Operator Model that is a part of Kube-Prometheus project. It'll install node-exporter that is a daemonset. It'll act as a agent for nodes
* Prometheus can work with docker container also, without K8s
* splunk and kibana both can be combined.
* https://blogs.halodoc.io/production-grade-eks-logging-with-efk-via-sidecar/
* how scaling is designed for botkube ? once is a while, there can be many events from K8s cluster . Most of the time, there are no events from k8s cluster.
Scaling is using K8s HPA only. Even we have so many events, each event processing needs little CPU.
Meetup Recording : https://www.youtube.com/watch?v=bGnQep5bY6c
0 comments:
Post a Comment