Kafka tutorial : My First kafka Program

My First Kafka program to start as beginner ,Before start you need to setup Kafka server and Start it resources.

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming and creating a data pipelines and applications. A Kafka consumer is a client that reads records from a Kafka cluster.A Kafka Producer is store data and send to Topic.

We can see kafka consumer and producer in action.In this example we are using console .we have csv file and send to Kafka consumer .

Step 1 : Creating Kafka Topic

Apache Kafka producers are clients that send records to Kafka topic,Here we are seeing two factor which affect a creation of topic as follow

Replication factor : for backup purpose
Partition : depend on parellel processing and storage

Below is command you need to run on console after starting of Kafka server.

bin\windows\kafka-topics.bat - create - topic test - partitions 1 
- replication-factor 1 - bootstrap-server localhost:9092

Step 2 : Sending data to Kafka producer using Kafka console

We need to define broker list and path of file , where our data is store (MysampleFile.csv).Here we can see Producer console and run below command

bin\windows\kafka-console-producer.bat - topic test - broker-list 
localhost:9092 < C:\Users\Admin\Documents\kafka\data\data\MysampleFile.csv

Step 3: Reading Kafka topic using Consumer using kafka console

Now we can read topic from consumer and we need define parameter as beginning

bin\windows\kafka-console-consumer.bat - topic test - bootstrap-server
 localhost:9092 - from-beginning

We can say ” A group of consumers that work together to consume records from a topic. Each consumer in the group is assigned a subset of the partitions”.