Showing posts with label istio. Show all posts
Showing posts with label istio. Show all posts
Distributed Tracing FAQ
** Is tracing only for request-response pair? How tracing happens for
indication where response is not expected.
Zipkin
This is: One-way RPC Tracing. One-way is the same as normal RPC
tracing, except there is no response anticipated.
In normal RPC tracing 4 annotations are used: “cs” “sr” (request) then
“ss” “cr” (response). In one-way tracing, the first two are used “cs” “sr” as
there is no response returned to the caller.
So, the client adds “cs” to a span and reports it to zipkin. Then, the
server adds “sr” to the same span and reports it. Neither side add
Span.timestamp or duration because neither side know both when the span started
and finished.
CS = Client Send
SR = Server Receive
SS = Server Sent
CR = Client Receive
They are annotations for spans in Zipkin format
Jaeger
For asynchronous messaging communications: “span.kind” tag values are:
Producer and Consumer
For synchronous RPC communications: “span.kind” tag values are: Client
and Server
The span.kind=server tag denotes an entry span, e.g. a span created in
the local code in response to an external request. Likewise, span.kind=client
denotes an exit span, e.g. a call made from the local code to another server.
** At ‘Istio service mesh’, does ‘Envoy sidecar proxy’ add ‘B3 headers’
for incoming message or outgoing message,
For incoming message.
** What if some of the microservices use Zipkin format and new
microservices use Jaeger format at K8s cluster?
Jaeger provides backward compatibility. So, just need to route ‘Zipkin
format data’ from ‘legacy services’ to ‘Jaeger backend’ using ‘zipkin
collector’
Backwards compatibility with Zipkin
Although we recommend instrumenting applications with OpenTracing API and
binding to Jaeger client libraries to benefit from advanced features not
available elsewhere, if your organization has already invested in the
instrumentation using Zipkin libraries, you do not have to rewrite all that
code. Jaeger provides backwards compatibility with Zipkin by accepting spans in
Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend
is just a matter of routing the traffic from Zipkin libraries to the Jaeger
backend.
** The ‘Envoy sidecar proxy’ at ‘Istio Service Mesh’ sends collected
span data (JSON format) to Jaeger Collector. It it for incoming message or
outgoing message?
The ‘Envoy sidecar proxy’ sends a single span (JSON format), asynchronously,
anytime, after receiving response for the request which was originated by
application.
** Can we bypass Jaeger Agent?
Yes for C++ Jaeger ClientLib.
** Can Jaeger Collector work on UDP?
No.
** How to use Baggage in Jaeger headers?
Key: uberctx-{baggage-key}
Value: url-encoded string
** How to use Baggage in Zipkin B3 header?
baggage-:
The support is only for Java and Go languages
** What is thrift
https://en.wikipedia.org/wiki/Apache_Thrift. It is for RPC with format
like: JSON, Binary etc.
** what is TChannel?
It is a protocol over TCP for RPC
** Does Zipkin supports binary/compress format?
Zipkin can work on TChannel. It also supports compact Thrift format.
However, Jaeger is better as being CNCF project.
** TChannel, Thrift, or its combination, which one is better?
Now, Jaeger collector supports gRPC as well. gRPC can be even better
choice.
** Wild card query is supported at Jaeger UI?
No, due to limitations of other storage implementations. However, one
can use Kibana + Elastic Search for the same.
Istio Practical - 1
Installation
Istio version istio-1.3.0-rc.1 at path Downloads/istio-1.3.0-rc.1
Helm version helm-v2.14.3
sudo apt-get install socat
kubectl create serviceaccount tiller --namespace kube-system
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --wait --service-account tiller
kubectl create namespace istio-system
helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.2.5/charts/
helm repo update
helm template Downloads/istio-1.3.0-rc.1/install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -
helm install --wait --name istio --namespace istio-system Downloads/istio-1.3.0-rc.1/install/kubernetes/helm/istio \
--set gateways.istio-ingressgateway.type=NodePort \
--set gateways.istio-egressgateway.type=NodePort \
--set grafana.enabled=true \
--set kiali.enabled=true \
--set kiali.dashboard.grafanaURL=http://localhost:3000 \
--set kiali.dashboard.jaegerURL=http://localhost:16686 \
--set servicegraph.enabled=true \
--set telemetry-gateway.grafanaEnabled=true \
--set telemetry-gateway.prometheusEnabled=true \
--set tracing.enabled=true \
--set sidecarInjectorWebhook.enabled=true \
--set global.mtls.enabled=false
Istio version istio-1.3.0-rc.1 at path Downloads/istio-1.3.0-rc.1
Helm version helm-v2.14.3
sudo apt-get install socat
kubectl create serviceaccount tiller --namespace kube-system
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --wait --service-account tiller
kubectl create namespace istio-system
helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.2.5/charts/
helm repo update
helm template Downloads/istio-1.3.0-rc.1/install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -
helm install --wait --name istio --namespace istio-system Downloads/istio-1.3.0-rc.1/install/kubernetes/helm/istio \
--set gateways.istio-ingressgateway.type=NodePort \
--set gateways.istio-egressgateway.type=NodePort \
--set grafana.enabled=true \
--set kiali.enabled=true \
--set kiali.dashboard.grafanaURL=http://localhost:3000 \
--set kiali.dashboard.jaegerURL=http://localhost:16686 \
--set servicegraph.enabled=true \
--set telemetry-gateway.grafanaEnabled=true \
--set telemetry-gateway.prometheusEnabled=true \
--set tracing.enabled=true \
--set sidecarInjectorWebhook.enabled=true \
--set global.mtls.enabled=false
Istio HandsOn
Here is list of URLs, for hands on with Istio
•https://blog.nebrass.fr/playing-with-istio-service-mesh-on-kubernetes/
•https://blog.nebrass.fr/playing-with-istio-service-mesh-on-kubernetes/
Istio 101
Istio 101
Meetup event by
Kubernetes & Openshift India Community
https://www.meetup.com/kubernetes-openshift-India-Meetup/events/263328152/
=================
Challenge with microservice
- Service Discovery
- Load Balancing
- Monitoring and Observability
- Network resiliency
- Latency
- Security
- ACL
Istio has evolved.
When people realized the challenges with micro services, Netflix OSS has developed following tools
Hystrix: Circuit Breaking
Zuul: Edge Router
Ribbon: Service Discovery, LB
Eureka: Service Registry
Brave / Zipkin: Tracing
Spectator / Atlas : Metrics
=================
Few analogy between Open Shift and Kubernetes.
IstioAuth module is not Citadel
================= We had interesting question about Mirroing / Shadowing the incoming request. How even a new TCP session will be created?
Add
- name: tcpdump
image: corfr/tcpdump
command:
- /bin/sleep
- infinity
at Deployment.yaml
under spec: containers:
https://developers.redhat.com/blog/2019/02/27/sidecars-analyze-debug-network-traffic-kubernetes-pod/
=================
Cross cluster federation is also present at Istio, in case if the application is deployed on two different clusters hosted by two different cloud service provider.
Envoy proxy is light weight, efficient and very powerful. It has lots of configuration options. One should avoid play around with them, at beginner stage.
=================
Istio can be installed using Helm chart. Another option is to use Maistra Istio-operator. It is wrapper around Helm chart.
=================
Redhat offers Istio as "OpenShift Service Mesh"
=================
Reference :
https://github.com/redhat-developer-demos/istio-tutorial
https://redhat-developer-demos.github.io/istio-tutorial/istio-tutorial/1.1.x/index.html
https://developers.redhat.com/topics/service-mesh/
For cartoons : http://turnoff.us
Books
=================
Meetup event by
Kubernetes & Openshift India Community
https://www.meetup.com/kubernetes-openshift-India-Meetup/events/263328152/
=================
Challenge with microservice
- Service Discovery
- Load Balancing
- Monitoring and Observability
- Network resiliency
- Latency
- Security
- ACL
Istio : Connect, Manage, Secure microservices.
Istio has rich policy driven ops IFTTT
Istio has rich policy driven ops IFTTT
Istio has evolved.
When people realized the challenges with micro services, Netflix OSS has developed following tools
Hystrix: Circuit Breaking
Zuul: Edge Router
Ribbon: Service Discovery, LB
Eureka: Service Registry
Brave / Zipkin: Tracing
Spectator / Atlas : Metrics
However they are specific to Java. Addition code was added existing Java application code.
In case of Istio, side car proxy container is added to each pod. The existing application code is not modified. Istio can be used for application developed in any language and polyglot applications.
Early version of Istio was not optimized. Industry was skeptical and reluctant to adopt Istio. For each request, Envoy Sidecar proxy contacts Mixer module for policy check. After the request is processed, it updates the metrics to Mixer. Later on Caching was added. The early adopters of Istio, themselves contribute back to Istio. Lately many many performance optimization happened in Istio. Now more and more micro service based applications are using Istio.
Istio : Production deployment
Success : eBay, IBM
Failure : BigBasket https://tech.bigbasket.com/bigbaskets-experience-with-istio/
Success : eBay, IBM
Failure : BigBasket https://tech.bigbasket.com/bigbaskets-experience-with-istio/
Few analogy between Open Shift and Kubernetes.
* project = namespace
* oc = kubectl
* oc expose service = ingress in k8s
=================
Side Car proxy can be injected by two ways
Side Car proxy can be injected by two ways
1. mannual injection with istioctl command
2. automatic injection: by annotation for mutation webhook
=================
istioctl modules talks with istio's control plane component by name Pilot
=================istioctl modules talks with istio's control plane component by name Pilot
IstioAuth module is not Citadel
================= We had interesting question about Mirroing / Shadowing the incoming request. How even a new TCP session will be created?
Add
- name: tcpdump
image: corfr/tcpdump
command:
- /bin/sleep
- infinity
at Deployment.yaml
under spec: containers:
https://developers.redhat.com/blog/2019/02/27/sidecars-analyze-debug-network-traffic-kubernetes-pod/
=================
Cross cluster federation is also present at Istio, in case if the application is deployed on two different clusters hosted by two different cloud service provider.
=================
There are set of istio-ctl commands for debugging the application deployment. I found this URL : https://istio.io/docs/ops/component-debugging/
=================Envoy proxy is light weight, efficient and very powerful. It has lots of configuration options. One should avoid play around with them, at beginner stage.
=================
Istio can be installed using Helm chart. Another option is to use Maistra Istio-operator. It is wrapper around Helm chart.
=================
Redhat offers Istio as "OpenShift Service Mesh"
=================
Reference :
https://github.com/redhat-developer-demos/istio-tutorial
https://redhat-developer-demos.github.io/istio-tutorial/istio-tutorial/1.1.x/index.html
https://developers.redhat.com/topics/service-mesh/
For cartoons : http://turnoff.us
Books
- Introducing Istio Service Mesh MicroServices
- Migrating from monolithic to microservice databases
- Designing Distributed Systems : Chapter 2 about side car proxy
=================
Disclaimer : This blog is just my note from an event, that I attended. It is not verbatim of any speech. This blog may not indicate the exact expression/opinion of speakers of the event, due to my possible mistake in taking note. Any corrections/suggestions are welcome.
Service Meshes + Kubernetes: Solving Service-Service Issues
Saturday 23rd March Kubernetes Day India 2019 event is happening in Bangalore. Mr. Ben Hall has arrived here, in Bangalore as speaker for this event. He is founder of https://katacoda.com/ The "DigitalOcean Bangalore" Meetup group, grabbed this opportunity and organized meetup event "Service Meshes + Kubernetes: Solving Service-Service Issues" today evening, where MR. Ben Hall shared his knowledge. Here is my notes exclusively for readers of this blog "Express YourSelf !"
Digital Ocean announced about another event do.co/tide2019 on April 4th and encourage all to participate. They also promote new offering about managed Kubernetes on Digital Ocean.
Kubernetes is great tool. However we need service mesh for security, scaling, communication among pods. It is about TLS, certification roations Auth/Security, Rate Limit, Retry Logic, Fail Over etc. It provides more control for A/B testing, Canary releases, collecting system metrics and to verify, is this trusted caller pod? The service mesh can be implemented using:
istio is all about just configuring Envoy proxy. Istio uses three major components. 1. Pilot 2. Mixer and 3. Citadel
He demonstrated istio on his own website / learning platform katacoda. He testing using curl to generate ingress traffic. He also mentioned and demonstrated "scope" tool. It is for monitoring, visualization and management for Docker and k8s. It is not part of istio.
We had interesting QA sessions.
* Yes, istio is adding little latency, when we use HTTP 1.0 based RESTful APIs. However, we get performance gain, when we use gRPC or HTTP 2.0 based RESTful APIs. It is tradeoff between, performance at production environment for given hardware, or gain in terms of developer's productivity.
* Prometheus is used to store all matrices of cluster.
* How to configure different time out values for 2 different consumers, who consumes the same service? Well, we can duplicate service with different names, or modify some application level logic.
* for pod communication, one can use either RPC (gRPC, RESTful API) based or enterprise service bus based (message queue and kafka). If one takes the second approach, then istio may or may not provide additional values. It depends.
* These was a quick comparison between SDN based routing and istio based approach.
* during informal QA, we discussed about Gloo. Gloo is a feature-rich, Kubernetes-native ingress controller, and next-generation API gateway, powered by Envoy.
At the end we had tea and light snacks of cookies and Samosa.
Reference
https://www.meetup.com/DigitalOceanBangalore/events/259864782/
https://www.slideshare.net/BenHalluk/presentations
https://medium.com/google-cloud/understanding-kubernetes-networking-ingress-1bc341c84078
https://blog.aquasec.com/istio-service-mesh-traffic-control
https://gloo.solo.io/
https://github.com/solo-io/gloo
https://layers7.blogspot.com/2019/03/kafka-communication-among-micro-services.html
https://layers7.blogspot.com/2017/12/istio.html
Disclaimer
This blog post is NOT verbatim of the speech. I captured this note as per my understanding. It may not necessarily indicate the speaker's intention. So corrections/suggestions/comments are welcome.
Digital Ocean announced about another event do.co/tide2019 on April 4th and encourage all to participate. They also promote new offering about managed Kubernetes on Digital Ocean.
Kubernetes is great tool. However we need service mesh for security, scaling, communication among pods. It is about TLS, certification roations Auth/Security, Rate Limit, Retry Logic, Fail Over etc. It provides more control for A/B testing, Canary releases, collecting system metrics and to verify, is this trusted caller pod? The service mesh can be implemented using:
- ASPEN MESH
- linkerd
- HasiCorp Consul
- istio
Istio provides four key capabilities
1 connect. service discovery, load balancing
2 secure. e.g protect the system against fake payment service. encryption, authentication, authorization
3 control
4 observe
Istio adds/extends some more capabilities to kubernetes APIs, by adding YAML files. Grafana and Prometheus installation is part of istio installation.
istio is all about just configuring Envoy proxy. Istio uses three major components. 1. Pilot 2. Mixer and 3. Citadel
He demonstrated istio on his own website / learning platform katacoda. He testing using curl to generate ingress traffic. He also mentioned and demonstrated "scope" tool. It is for monitoring, visualization and management for Docker and k8s. It is not part of istio.
We had interesting QA sessions.
* Yes, istio is adding little latency, when we use HTTP 1.0 based RESTful APIs. However, we get performance gain, when we use gRPC or HTTP 2.0 based RESTful APIs. It is tradeoff between, performance at production environment for given hardware, or gain in terms of developer's productivity.
* Prometheus is used to store all matrices of cluster.
* How to configure different time out values for 2 different consumers, who consumes the same service? Well, we can duplicate service with different names, or modify some application level logic.
* for pod communication, one can use either RPC (gRPC, RESTful API) based or enterprise service bus based (message queue and kafka). If one takes the second approach, then istio may or may not provide additional values. It depends.
* These was a quick comparison between SDN based routing and istio based approach.
* during informal QA, we discussed about Gloo. Gloo is a feature-rich, Kubernetes-native ingress controller, and next-generation API gateway, powered by Envoy.
At the end we had tea and light snacks of cookies and Samosa.
Reference
https://www.meetup.com/DigitalOceanBangalore/events/259864782/
https://www.slideshare.net/BenHalluk/presentations
https://medium.com/google-cloud/understanding-kubernetes-networking-ingress-1bc341c84078
https://blog.aquasec.com/istio-service-mesh-traffic-control
https://gloo.solo.io/
https://github.com/solo-io/gloo
https://layers7.blogspot.com/2019/03/kafka-communication-among-micro-services.html
https://layers7.blogspot.com/2017/12/istio.html
Disclaimer
This blog post is NOT verbatim of the speech. I captured this note as per my understanding. It may not necessarily indicate the speaker's intention. So corrections/suggestions/comments are welcome.
istio
istio
Micro-service mesh management framework
It provides a uniform way to connect, manage, and secure microservices. It supports managing traffic flows between microservices, enforcing access policies, and aggregating telemetry data, all without requiring changes to the microservice code.
Benifit
* A/B testing,
* canary releases,
* failure recovery,
* metrics,
Key Capability
* Traffic Management
- load balancing,
- rate limiting,
* Observability
- monitoring
* Policy Enforcement
- access control,
- load balancing,
* Service identity and security
- service-to-service authentication,
- discovery of services,
- end-to-end authentication.
* Platform Support
- Cloud,
- on-premise,
- Kubernetes,
- Mesos
* Integration and Customization : integrate with existing solutions for
- ACLs,
- logging,
- monitoring,
- quotas,
- auditing
- etc.
Istio pre-configured add-ons
* Grafana : dashboard to visualize service mesh traffic data
* Prometheus : to query istio metrics
* ServiceGraph : generating and visualizing a graph of services within a mesh
* Zipkin : distributed tracing system
Architecture
1. Data plane :
set of intelligent proxy (Envoy)
2. Control plane :
manage and configure proxy
- to route traffic
- to enforce policy runtime.
1. Envoy : sidecar proxy in same pod with features :
dynamic service discovery,
load balancing,
TLS termination,
HTTP & gRPC proxying,
circuit breakers,
health checks,
staged roll-outs with percentage-based traffic split,
fault injection,
rich metrics.
rich L7 routing
2. Mixer:
platform independent
flexible plugin model
with a variety of host environments and infrastructure back end
Tasks:
- enforce access control
- enforce usage policies such as authorization, rate limits, quotas, authentication etc.
- collect telemetry data from envoy
- request tracing
Mixer configuration for
- attribute extraction
- policy evaluation
Adapter
Go Package.
Guide to develop new adapter : https://github.com/istio/istio/wiki/Mixer-Compiled-In-Adapter-Dev-Guide
https://istio.io/blog/2017/adapter-model/
3. Pilot
Tasks:
- converts high level routing rules that control traffic behavior into Envoy-specific configurations
- propagates Envoy-specific configurations to the sidecars at runtime
- abstracts platform-specific service discovery mechanisms
- translate service discovery to Envoy data plane API
Benefits
* service discovery
* traffic management
* intelligent routing
- A/B tests,
- canary deployments
* resiliency
- timeouts,
- retries,
- circuit breakers,
- etc.
* multiple environments
- Kubernetes,
- Consul/Nomad
4. istio-Auth
Authentication using mutua TLS
Built-in identity + credentials management
enforce policy based on service identity
5. Citadel
A centralized component responsible for certificate issuance and rotation.
6. Node Agent
A per-node component responsible for certificate issuance and rotation.
7. Galley
Central component for validating, ingesting, aggregating, transforming and distributing config within Istio.
In nut-shell istio is all about just configuring Envoy proxy
Micro-service mesh management framework
It provides a uniform way to connect, manage, and secure microservices. It supports managing traffic flows between microservices, enforcing access policies, and aggregating telemetry data, all without requiring changes to the microservice code.
Benifit
* A/B testing,
* canary releases,
* failure recovery,
* metrics,
Key Capability
* Traffic Management
- load balancing,
- rate limiting,
* Observability
- monitoring
* Policy Enforcement
- access control,
- load balancing,
* Service identity and security
- service-to-service authentication,
- discovery of services,
- end-to-end authentication.
* Platform Support
- Cloud,
- on-premise,
- Kubernetes,
- Mesos
* Integration and Customization : integrate with existing solutions for
- ACLs,
- logging,
- monitoring,
- quotas,
- auditing
- etc.
Istio pre-configured add-ons
* Grafana : dashboard to visualize service mesh traffic data
* Prometheus : to query istio metrics
* ServiceGraph : generating and visualizing a graph of services within a mesh
* Zipkin : distributed tracing system
Architecture
1. Data plane :
set of intelligent proxy (Envoy)
2. Control plane :
manage and configure proxy
- to route traffic
- to enforce policy runtime.
1. Envoy : sidecar proxy in same pod with features :
dynamic service discovery,
load balancing,
TLS termination,
HTTP & gRPC proxying,
circuit breakers,
health checks,
staged roll-outs with percentage-based traffic split,
fault injection,
rich metrics.
rich L7 routing
2. Mixer:
platform independent
flexible plugin model
with a variety of host environments and infrastructure back end
Tasks:
- enforce access control
- enforce usage policies such as authorization, rate limits, quotas, authentication etc.
- collect telemetry data from envoy
- request tracing
Mixer configuration for
- attribute extraction
- policy evaluation
Adapter
Go Package.
Guide to develop new adapter : https://github.com/istio/istio/wiki/Mixer-Compiled-In-Adapter-Dev-Guide
https://istio.io/blog/2017/adapter-model/
3. Pilot
Tasks:
- converts high level routing rules that control traffic behavior into Envoy-specific configurations
- propagates Envoy-specific configurations to the sidecars at runtime
- abstracts platform-specific service discovery mechanisms
- translate service discovery to Envoy data plane API
Benefits
* service discovery
* traffic management
* intelligent routing
- A/B tests,
- canary deployments
* resiliency
- timeouts,
- retries,
- circuit breakers,
- etc.
* multiple environments
- Kubernetes,
- Consul/Nomad
4. istio-Auth
Authentication using mutua TLS
Built-in identity + credentials management
enforce policy based on service identity
5. Citadel
A centralized component responsible for certificate issuance and rotation.
6. Node Agent
A per-node component responsible for certificate issuance and rotation.
7. Galley
Central component for validating, ingesting, aggregating, transforming and distributing config within Istio.
In nut-shell istio is all about just configuring Envoy proxy
