In recent times the Kafka has become a buzzword in the industry, it all started with Linkedin trying to manage the large amount of the data across different applications and data store.
Although earlier systems did provide methods of replication(mostly at database level), high throughput and reliability. One of the major challenge was to connect these technologies into a seamless integration to build an enterprise level application. Also how to handle failures, what if there was a bug in the consumer, you processed the message, now the data is corrupt. Can we correct the data, once you have fixed the consumer.
Kafka has been designed as a high throughput messaging system, with capabilities of high scalability. It is highly reliable and durable. The api’s have been developed keeping loose coupling between the producer and consumer in mind. It provides a flexible semantics to consume and publish messages.
Kafka bases itself on the publisher/subscriber model. In kafka world, publisher is called as Producer and subscriber is referred as Consumer. To simply put producer publishes message to central location called as Topic, the consumer receives the message and process them.
The Broker manages all the topics, and the broker can reside on a single/multiple host. Multiple brokers’ comes together to form a cluster, which is managed by Apache Zookeeper.
There are lots of concepts, but for now let proceed with getting the hands dirty.
The first step is to download the Kafka on your machine, I am using a mac(steps will be similar for other OS as well).
Extract the zip to your local machine
$ tar -xvf kafka_2.12-2.5.0.tgz $ cd kafka_2.12-2.5.0
Before we start, we need to configure the zookeeper. We need to modify
$ vi config/zookeeper.properties
By default, the zookeeper stores the data at
/tmp/zookeeper, I will modify it to use a custom directory
zookeeper-data. To do that, we need to change the dataDir in the zookeeper.properties file.
$ mkdir zookeeper-data
My modified zookeeper.properties file look like this:
# Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # the directory where the snapshot is stored. dataDir=/zookeeper-data # the port at which the clients will connect clientPort=2181 # disable the per-ip limit on the number of connections since this is a non-production config maxClientCnxns=0 # Disable the adminserver by default to avoid port conflicts. # Set the port to something non-conflicting if choosing to enable this admin.enableServer=false # admin.serverPort=8080
To run the zookeeper we need to goto bin folder and run the following command.
$ cd bin $ ./zookeeper-server-start.sh ../config/zookeeper.properties
This should start the zookeeper, now let’s move to kafka.
Similarly, we are going to modify kafka configuration. To do that we need to modify
config/server.properties to point to custom directory. By default, it points to
Now lets start the server:
$ cd bin $ ./kafka-server-start.sh ../config/server.properties
Kafka should be up and running.
Create Kafka Topic
As per the documentation: A topic is a category or feed name to which records are published. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it For now we are using commandline utility provided by kafka to create a topic. We will create a topic with single replication factor since we are running a single broker instance.
$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test Created topic test. $
The above command creates a topic named test. we will use this topic to publish our messages.
We will again use the commandline utility to publish the message to Kafka topic../
$ ./kafka-console-producer.sh --broker-list localhost:9092 --topic test >message 1 >message 2 >message 3
We will use the kafka console consumer to connect to Kafka at port 9092 and receive message from test topic.
$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning message 1 message 2 message 3
We have successfully started single broker of Kafka with the zookeeper, published and consumed the messages.