Java World Knowledge Sharing Portal: Apache KAFKA CLI

To understand and use initial level Apache kafka we need to install it on our machine. As i am using window i had downloaded kafka_2.13-2.6.0.tgz the same from the belwo site.

https://kafka.apache.org/downloads

extract it and set the required environment variable so that you can directly use it when you open any cmd prompt.
i.e.
KAFKA_HOME C:\kafka_2.13-2.6.0\bin

However you can also take he commercial version from confluent.io or also used the managed service from cloud system like confluent, amazone etc.
benefit of using commercial version is that they provided a bunch of tool and support that is needed to you. most likely in prod env we are going to use the confluent.io.Additional for managed approach from cloude we dont need to think of anything all infra needs are taken care by them we only get the instance and we are good to go with it.

Now there is sequence of step that we need to for starting our Apache cluster in local machine.

Step 1 :-
‘- Zookeper starting

Zookeper is bundled by default with Apache kafka it is used to monitor kafka. To understand it visit this site.
https://zookeeper.apache.org/index.html. Zookeeper keeps track of status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions etc. Zookeeper it self is allowing multiple clients to perform simultaneous reads and writes and acts as a shared configuration service within the system.

C:\kafka_2.13-2.6.0\bin\windows>zookeeper-server-start.bat ../../config/zookeeper.properties

Step 2:-
‘- kafka broker
Then we will start the broker using below command.

C:\kafka_2.13-2.6.0\bin\windows>kafka-server-start.bat C:\kafka_2.13-2.6.0\config\server.properties

Note:- We can create many broker as we want only the changes we want to in the C:\kafka_2.13-2.6.0\config\server.properties are

1- broker.id=0
2- log.dirs=/tmp/kafka-logs
3- listeners=PLAINTEXT://:9092

Step 3:-

‘- Create Topics
We will create a topic where we are going to store our message send from the producer.

C:\kafka_2.13-2.6.0\bin\windows>kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test-siddhu

Ste 4:-
‘- consumer

We will create a consumer who will send a message to above created topics.

C:\kafka_2.13-2.6.0\bin\windows>kafka-console-consumer.bat –bootstrap-server localhost:9092 –topic test-siddhu –from-beginning

Step 5:-
‘- producer

We will create a producer which will read the data entered in to the Topic by Producer.

C:\kafka_2.13-2.6.0\bin\windows>kafka-console-producer.bat –broker-list localhost:9092 –topic test-siddhu

Step 6:-
‘- Check topic is created and list of the topics.

C:\kafka_2.13-2.6.0\bin\windows>kafka-topics.bat –list –zookeeper localhost:2181
test-siddhu

‘- Topic descriptions
C:\kafka_2.13-2.6.0\bin\windows>kafka-topics.bat –describe -zookeeper localhost:2181 –topic test-siddhu
Topic: test-siddhu PartitionCount: 1 ReplicationFactor: 1 Configs:
Topic: test-siddhu Partition: 0 Leader: 0 Replicas: 0 Isr: 0

Now lets send the data from the producer and check it is coming to consumer. We had created/setup the apache kafka with one producer and one consumer

Now lets send the data from the producer and check it is coming to consumer. We had created/setup the apache kafka with one producer and one consumer

For this we will three broker lets create it. This can be done simply by coping the C:\kafka_2.13-2.6.0\config\server.properties to differ name properties file i.e. C:\kafka_2.13-2.6.0\config\server1.properties and C:\kafka_2.13-2.6.0\config\server2.properties

Additional we also need to change the following below given three things in each properties files

server.properties- Will keep its value default

server1.properties-

broker.id=1

port=9093

log.dirs=/tmp/kafka-logs1

server2.properties

broker.id=2

port=9094

log.dirs=/tmp/kafka-logs2

Now start zookeeper using command

C:\kafka_2.13-2.6.0\bin\windows>zookeeper-server-start.bat ../../config/zookeeper.properties

Start three broker using command

C:\kafka_2.13-2.6.0\bin\windows>kafka-server-start.bat C:\kafka_2.13-2.6.0\config\server.properties

C:\kafka_2.13-2.6.0\bin\windows>kafka-server-start.bat C:\kafka_2.13-2.6.0\config\server1.properties

C:\kafka_2.13-2.6.0\bin\windows>kafka-server-start.bat C:\kafka_2.13-2.6.0\config\server2.properties

Now create a topic that will have 3 partition using below command

C:\kafka_2.13-2.6.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic test-consumer-group

Test Topic is created using below command

C:\kafka_2.13-2.6.0\bin\windows>kafka-topics.bat --describe -zookeeper localhost:2181 --topic test-consumer-group

Now lets create a two consumer with same consumer group for that execute below command twice

C:\kafka_2.13-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-consumer-group --from-beginning --group Group1

Now lets produce the data using below command

C:\kafka_2.13-2.6.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic test-consumer-group < C:\to_delete\Apache_Kafka\data.xlsx

Once you hit the above command our data will be inserted into the two consumer which indicate that work is shared with both client/consumer as they belog to same consumer group.