Kafka : Communication among Micro Services

Microservice Communications has two Approaches

1. RPC based and
2. Enterprise Service Bus which has root from SOA. 


RPC based is implemented using 

It needs: 
-load balancer
-service discovery
2. gRPC

Enterprise Service Bus is implemented using

1. Messaging Queue
1.1 RabbitMQ
1.2 ActiveMQ
1.3 ZeroMQ
2. Kafka

1. Synchronous protocol using RESTful API over HTTP
2. Asynchronous protocol AMQP. smart endpoints and dumb pipe pattern. 

Kafka is distributed stream processing platform with high resilience and fault tolerance. Kafka replaces  Java Message Service (JMS), Advanced Message Queuing Protocol (AMQP), etc. Kafka supports both Pub-sub and queuing feature

Streaming platform capabilities

1. Publish and subscribe
2. Store stream of record in fault-tolerant way
3. Process stream of records

Kafka Features

-  low latency, 
-  auto-scaling, 
-  centralized management, 
-  proven high availability 
-  unified platform, 
-  high-throughput, 
for handling real-time data feeds

Apache Kafka Architecture

Kafka is combination of messaging, storage, and stream processing 

Messages are written to a log-style stream called a topic. It is like a list of chronological events broken up into multiple streams, known as “topics”.

Two types of topics
1. Regular topic
- time or space bound
- default 7 days.  
2. compacted
- never expire
- can be updated
- to delete write : tombstone message with null-value for a specific key.

Topic has records
Record has key, value and timestamp

Kafka topics are divided into various partitions. Partitions enable parallelization of topics. As many consumer that many partitions. Partitions are distributed and replicated across multiple brokers (servers). The flag "replication-factor" determines how many copies of the topic partition has to be made. This is how fault tolerance is achieved. 

Broker has partitions, that can be leader or replica for given topic. 

JSON, AVRO, or Protobufs as their serialization format. For effective use of n/w bandwidth, Kafka supports GZIP, Snappy, LZ4 and ZStandard compression protocol

ACL for read and write topics

Kafka APIs types

1. consumer
2. producer
3. connector : for import and export
4. Streams : To develop, stateful, scalable stream processing app. Kafka offers the Streams API that allows writing Java applications that consume data from Kafka and write results back to Kafka. Apache Kafka also works with external stream processing systems such as 
- Apache Apex, 
- Apache Flink, 
- Apache Spark, and 
- Apache Storm, 
- Samza, 
- Spout, 
- SpartStreaming, 
- IBM Streams, 
- Spring Cloud Stream. 
5. Admin Client API

Two types of Stream APIs
4.1 DSL API. Stream processing DSL (Domain specific language) offers filter, map, grouping, windowing, aggregation, joins, and the notion of tables
4.2 Processor API. 


Controller is also broker with more responsibilities of partition management, that includes

* Leader selection
* Leader switch
* New topic and partition
* New broker

Kafka Monitoring tools

1. Burrow
2. Datadog

Advantage with micro-services

* Messages are ordered chronologically and delivery is guaranteed
* Strong durability, resilience and performance


1. Add Kafka producer code to existing service in monolith
2. Develop new service with Kafka consume code and store everything in DB
3. Test. How far the new consumer service is time lagging. 
4. Enhance newly added service with some relevant code from existing monolith. It should fetch data from DB. Temporarily disable some external calls. 
5. Now filter events in producer at monolith. Send only actionable events to consumer new service, via Kafka.
6. Enhance the new service further. Send events back to monolith via Kafka. 
7. Test
8. Remove the code from monolith that was added to new service. 
9. Repeat. 


1. Gradel build tool
2. Java

Python and Kafka

Three alternatives

1. kafka-python : https://github.com/dpkp/kafka-python
from time import sleep
from json import dumps, loads
from kafka import KafkaProducer, KafkaConsumer
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],
                         value_serializer=lambda x: 
for e in range(1000):
    data = {'number' : e}
    producer.send('numtest', value=data)
consumer = KafkaConsumer(
     value_deserializer=lambda x: loads(x.decode('utf-8'))
for message in consumer:
    message = message.value
    print('{} added to {}'.format(message, collection))

2. pykafka https://github.com/Parsely/pykafka
3. confluent-kafka-python https://github.com/confluentinc/confluent-kafka-python


Kafka eco system : https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

Kafka Documentation

Python and Kafka



Post a Comment