Jaeger + Elastic Search: Handson


docker run --rm -it --name=elasticsearch -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:6.8.0

docker run --rm -it --link=elasticsearch --name=kibana -p 5601:5601 docker.elastic.co/kibana/kibana:6.8.0

docker run --rm -it --link=elasticsearch --name=jaeger -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://elasticsearch:9200 -e ES_TAGS_AS_FIELDS_ALL=true -p 16686:16686 jaegertracing/all-in-one:1.12

docker run --rm --link jaeger --env JAEGER_AGENT_HOST=jaeger --env JAEGER_AGENT_PORT=6831 -p8080-8083:8080-8083 jaegertracing/example-hotrod:latest all 

Create a file JSON.txt with below conntent

    "query" : {
        "match_all" : {}


curl -XGET "localhost:9200/jaeger-span-YYYY-MM-DD/_search" -H "Content-Type: application/json" -d @json.txt

curl -XGET "localhost:9200/jaeger-span-YYYY-MM-DD/_search?size=10000" -H "Content-Type: application/json" -d @json.txt



JSON Output: 

  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  "hits": {
    "total": 441,
    "max_score": 1,
    "hits": [
        "_index": "jaeger-span-2019-11-12",
        "_type": "span",
        "_id": "mQ2cXm4BVYoOLxBFNHpP",
        "_score": 1,
        "_source": {
          "traceID": "553c8316169ab970",
          "spanID": "2cd1c33438cb79f5",
          "flags": 1,
          "operationName": "HTTP GET /route",
          "references": [
              "refType": "CHILD_OF",
              "traceID": "553c8316169ab970",
              "spanID": "44fa89467cc873aa"
          "startTime": 1573545324720315,
          "startTimeMillis": 1573545324720,
          "duration": 53302,
          "tags": [],
          "tag": {
            "component": "net/http",
            "http@method": "GET",
            "http@status_code": 200,
            "http@url": "/route?dropoff=728%2C326&pickup=106%2C311",
            "internal@span@format": "proto",
            "span@kind": "server"
          "logs": [
              "timestamp": 1573545324720369,
              "fields": [
                  "key": "event",
                  "type": "string",
                  "value": "HTTP request received"
                  "key": "level",
                  "type": "string",
                  "value": "info"
                  "key": "method",
                  "type": "string",
                  "value": "GET"
                  "key": "url",
                  "type": "string",
                  "value": "/route?dropoff=728%2C326&pickup=106%2C311"
          "process": {
            "serviceName": "route",
            "tags": [],
            "tag": {
              "client-uuid": "761a24cbd892ada8",
              "hostname": "367b2536b5f9",
              "ip": "",
              "jaeger@version": "Go-2.20.0"

Trace JSON at Kibana APM

    "events": [
            "@timestamp": "2018-07-30T18:53:42.281Z",
            "agent": {
                "name": "elastic-node",
                "version": "3.14.0"
            "parent": {
                "id": "abcdef0123456789"
            "processor": {
                "event": "span",
                "name": "transaction"
            "service": {
                "environment": "staging",
                "name": "backendspans"
            "span": {
                "action": "query.custom",
                "duration": {
                    "us": 141581
                "id": "abcdef01234567",
                "name": "GET /api/types",
                "subtype": "postgresql",
                "type": "db"
            "timestamp": {
                "us": 1532976822281000
            "trace": {
                "id": "fdedef0123456789abcdef9876543210"
            "transaction": {
                "id": "01af25874dec69dd"




Distributed Tracing FAQ

** Is tracing only for request-response pair? How tracing happens for indication where response is not expected.

This is: One-way RPC Tracing. One-way is the same as normal RPC tracing, except there is no response anticipated.
In normal RPC tracing 4 annotations are used: “cs” “sr” (request) then “ss” “cr” (response). In one-way tracing, the first two are used “cs” “sr” as there is no response returned to the caller.
So, the client adds “cs” to a span and reports it to zipkin. Then, the server adds “sr” to the same span and reports it. Neither side add Span.timestamp or duration because neither side know both when the span started and finished.
CS = Client Send
SR = Server Receive
SS = Server Sent
CR = Client Receive
They are annotations for spans in Zipkin format


For asynchronous messaging communications: “span.kind” tag values are: Producer and Consumer
For synchronous RPC communications: “span.kind” tag values are: Client and Server
The span.kind=server tag denotes an entry span, e.g. a span created in the local code in response to an external request. Likewise, span.kind=client denotes an exit span, e.g. a call made from the local code to another server.

** At ‘Istio service mesh’, does ‘Envoy sidecar proxy’ add ‘B3 headers’ for incoming message or outgoing message,
For incoming message.

** What if some of the microservices use Zipkin format and new microservices use Jaeger format at K8s cluster?
Jaeger provides backward compatibility. So, just need to route ‘Zipkin format data’ from ‘legacy services’ to ‘Jaeger backend’ using ‘zipkin collector’

Backwards compatibility with Zipkin
Although we recommend instrumenting applications with OpenTracing API and binding to Jaeger client libraries to benefit from advanced features not available elsewhere, if your organization has already invested in the instrumentation using Zipkin libraries, you do not have to rewrite all that code. Jaeger provides backwards compatibility with Zipkin by accepting spans in Zipkin formats (Thrift or JSON v1/v2) over HTTP. Switching from Zipkin backend is just a matter of routing the traffic from Zipkin libraries to the Jaeger backend.

** The ‘Envoy sidecar proxy’ at ‘Istio Service Mesh’ sends collected span data (JSON format) to Jaeger Collector. It it for incoming message or outgoing message?
The ‘Envoy sidecar proxy’ sends a single span (JSON format), asynchronously, anytime, after receiving response for the request which was originated by application.

** Can we bypass Jaeger Agent?
Yes for C++ Jaeger ClientLib.

** Can Jaeger Collector work on UDP?

** How to use Baggage in Jaeger headers?
Key: uberctx-{baggage-key}
Value: url-encoded string

** How to use Baggage in Zipkin B3 header?
The support is only for Java and Go languages

** What is thrift
https://en.wikipedia.org/wiki/Apache_Thrift. It is for RPC with format like: JSON, Binary etc.

** what is TChannel?
It is a protocol over TCP for RPC

** Does Zipkin supports binary/compress format?
Zipkin can work on TChannel. It also supports compact Thrift format. However, Jaeger is better as being CNCF project.

** TChannel, Thrift, or its combination, which one is better?
Now, Jaeger collector supports gRPC as well. gRPC can be even better choice.

** Wild card query is supported at Jaeger UI?
No, due to limitations of other storage implementations. However, one can use Kibana + Elastic Search for the same.

Programming Kubernetes - I

Meetup Event : https://www.meetup.com/kubernetes-openshift-India-Meetup/events/264123372/

on 24th August 2019

* Only API server interact with etcd

* Extend K8s

1. Extend life cycle of API server by adding new controller
2. add more APIs : by adding new API resource types and by adding CRDs
3. cluster daemons for automation. E.g. collecting metrics
4. cluster assist to extend functionality. E.g. SSL certificate manager. 

* Action or commands on K8s resources

1 create
2 read
- get
- list
- watch
3 update
- patch
- replace
4 delete

Here are all such actions/command on Deployment resource



Short form : GVR = group, version, resource 

Possible versions: 
Alpha (v1alpha1)
Beta (v1beata1)
Stable (v1)

* Few useful commands

kubectl api version
kubectl api-resource

we can get short names
kubectl get --raw 

* KIND used at YAML file is corrosponding to golang type. For any built-in K8s object:
1. https://github.com/kubernetes/api/blob/master///types.go has multiple struct in Go langugae corrpospond to each K8s object at YAML file.
2. https://github.com/kubernetes/client-go/tree/master/kubernetes/typed// folder has multiple files. Each file is corrsoponding to one K8s object. 
* K8s support for client library implemented in many languages: 
Official support for : Go, Python, Java, DotNet, JavaScript, Haskell
Many more language supported by community implementaiton
Please refer: https://kubernetes.io/docs/reference/using-api/client-libraries/ The source code is available at https://github.com/kubernetes-client
Out of all of them: client-go has cache and many more features. Kubectl and prometheus-operator also internally use client-go library. Important packages / utilities of client-go library are described at blog https://itnext.io/how-to-create-a-kubernetes-custom-controller-using-client-go-f36a7a7536cc E.g. Kubernetes, Discovery, Dynamic, Transport, Plugin, Scale, Util, tool/cache etc. tools/clientcmd package is useful to setup client from kubeconfig file. 

* Flatcar Linux: 

- an immutable Linux distribution for containers. 
- a friendly fork of CoreOS’s Container Linux and as such, compatible with it. 
- a self-updating operating system 
- It is designed for containers, making it ideal for running Kubernetes and other container platforms. 
- It aims to be an independently built, distributed, and supported Linux distribution designed for container workloads.
Reference : https://www.flatcar-linux.org/

* apimachienry
- Scheme, typing, encoding, decoding, and conversion packages for Kubernetes and Kubernetes-like API objects.
- This library is a shared dependency for servers and clients to work with Kubernetes API infrastructure without direct type dependencies. 
- Its first consumers are k8s.io/kubernetes, k8s.io/client-go, and k8s.io/apiserver.

* "~/.kube/config" path has all the security related file to interface with K8s master node (or minikybe) kubeconfig has context

* controller uses service account. secreat is mounted inside pod

* jq is a lightweight and flexible command-line JSON processor. It is like sed for JSON. Reference : https://stedolan.github.io/jq/

* YAML file consists 

1. api version
2. kind
3. matadata
4. spec
5. status
everything under metadata is in type of go language

Read Annotations

Unit testing of controller with fake client


* Join K8s slack channel "k8sBLR"


Go Client to communicate with K8s cluster: https://github.com/kubernetes/client-go

Schema of the external API types that are served by the Kubernetes API server: https://github.com/kubernetes/api 

All relevant blogs: https://kubernauts.dev/ 

PPT slide deck: https://docs.google.com/presentation/d/1NgAxNrUxcOnodm9VA4plidrKradQM_kAbrTrF844gec/edit#slide=id.g5dcf860567_4_3

Github accounts

Relevant blogs




Alternatives of tcpdump

There are many tools similar to tcpdump, as per https://en.wikipedia.org/wiki/Comparison_of_packet_analyzers

Here, I choose only Free and Open Source tools, whose docker image is available and tool is lightweight.

  1. Ngrep is best, for capture only those packets, whose payload has certain pattern.

  1. Packetbeat is lightweight open source packet analyzer. It sends data to Elastic Search OR Logstash. It is not inline to datapath. So no impact on latency. It consumes high CPU. Packetbeat can run as sidecar Docker container: https://www.elastic.co/guide/en/beats/packetbeat/current/running-on-docker.html
It can capture all HTTP headers from request and response https://www.elastic.co/guide/en/beats/packetbeat/current/exported-fields-http.html

  1. Tranalyzer is Lightweight open-source flow generator and packet analyzer for practitioners and researchers

  1. Justniffer is like tcpdump. Tcpdump is for TCP, while Justniffer for HTTP. Useful to debug webserver.

Digital Certificate and SSL

1. cryptographic algorithms, 
1.1 Conventional cryptography (symmetric key)
1.2 Public Key cryptography 
2. message digest functions, = one-way hash
3. digital signatures
Encrypt 1. digest 2. seq number 3. etc. using private key

DER is based on BER. Digital certificate is converted to binary format using DER. Then Base64 convert + add prefix BEGIN + add suffix END = PEM format.

Cipher suite
1. Key Exchange Method : RSA, DH. with / without signature
2. Cipher for data transfer
2.1 No encryption
2.2 Steam cipher
2.3 Block cipher
3. Message Digest for creating MAC.
3.1 no digest
3.2 MD5
3.3 SHA

SSL Record Protocol between TCP and HTTP layers
1. Input from HTTP goes to RPU (Record Protocol Unit)
2. Compress input
3. add MAC
4. encrypt
5. output as TCP payload