In this step-by-step guide, I will show you how to use Apache Kafka with Spring Boot and Kotlin.

When you finish this article, you will know precisely:

• how to Produce and Consume String messages,
• how to Publish and Consume JSON messages,
• how to fix an infinite-loop problem with invalid JSON messages

Please keep in mind, that you should have Apache Kafka up and running on your local machine. If you would like to learn how to set it up, then check out my other articles under the Kafka tag.

2. Create New Project

Let’s start with creating a new Spring Boot project. As always, I highly encourage you to use the Spring Initializr page, when creating a new one:

To be on the same page, I’ve selected Spring Boot version 2.7.4 with Jar packaging and Java 17. Additionally, we will need two more dependencies:

• Spring for Apache Kafka– necessary to work with our Kafka node,
• Spring Web– although this one is not necessary, I’ve added it so that we can expose a test REST API endpoint.

[elementor-template id=”9007393″]

3. Publish/Consume String Messages

As the first example, let’s take a look at how to publish and consume simple String messages.

3.1. application.yaml Configuration

After we import our Kafka Spring Boot Kotlin project into our favorite IDE (like IntelliJ, for instance), let’s navigate to the application.yaml file and put the following:

spring:
  kafka:
    consumer:
      bootstrap-servers: localhost:8098
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    producer:
      bootstrap-servers: localhost:8098
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

As we can see, these settings are responsible for setting both Kafka consumer and producer:

• With bootstrap-servers, we point to the host and port, where we can reach out to our Kafka listener. Of course, depending on your node configuration this value may be different.
• When it comes to the auto-offset-reset, it lets us specify consumer behavior when there is no initial offset in Kafka or if the current offset does not exist
• With key-deserializer and value-deserializer, we specify the full qualifier to the deserializer we want to use. As the name suggests, we want to use the StringDeserializer for bot key and value deserialization.
• Finally, the key-serializer and value-serializer let us do almost exactly the same, but for the serialization process.

3.2. Create Constants.kt File

As the next step, let’s create Constants.kt file inside the config package:

const val EXAMPLE_TOPIC_NAME = "someTopicOne"

const val GROUP_ID = "groupId"

As we can see, we put two String constants here with the example topic name and group ID.

Although this is not necessary, it will help us maintain our code a bit cleaner.

3.3. Implement String Messages Producer

Nextly, let’s implement an ExampleStringProducer class inside the producer package:

@Component
class ExampleStringProducer(
  private val kafkaTemplate: KafkaTemplate<String, String>
) {

  fun sendStringMessage(message: String) {
    kafkaTemplate.send(EXAMPLE_TOPIC_NAME, message)
  }

}

As can be seen, in order to send messages we have to inject a KafkaTemplate class. This generic class exposes methods for executing high-level operations and as type parameters (<String, String> in our case), we have to specify the desired key and the value types.

When it comes to the sendStringMessage function, it simply takes a String argument called message and sends it to the topic- called someTopicOne in our case.

3.4 Create Messages Consumer

With that being done, let’s implement a consumer called ExampleConsumer:

@Component
class ExampleConsumer {

  private val logger = LoggerFactory.getLogger(this.javaClass)

  @KafkaListener(topics = [EXAMPLE_TOPIC_NAME], groupId = GROUP_ID)
  fun firstListener(message: String) {
    logger.info("Message received: [$message]")
  }

}

This time, in order to make our firstListener function a target of a Kafka message listener on the specified topics, we have to mark it with the @KafkaListener annotation. Moreover, if we would like to handle multiple topics with this function, we could simply specify their names in the topics array.

It’s worth mentioning that with the groupId, we override the group ID property for the consumer factory, but for this particular listener only.

And to summarize this code snippet- the incoming message will be simply logged into the output.

3.5 Expose Test Controller

Finally, let’s expose a test REST endpoint, which we will use to send messages:

@RestController
class ExampleController(
  private val exampleStringProducer: ExampleStringProducer
) {

  @PostMapping("/test")
  @ResponseStatus(HttpStatus.NO_CONTENT)
  fun sendTestMessage(
    @RequestBody requestBody: RequestBodyDto
  ) {
    exampleStringProducer.sendStringMessage(
      message = requestBody.message
    )
  }

  data class RequestBodyDto(val message: String)
}

As we can see, this handler will be responding to the POST requests to the /test endpoint.

As a request body, we will send the message, which will be then passed to the sendStringMessage function and if everything works fine, we should get a 204 No Content response.

3.6 Test The Endpoint

In order to test our functionality, let’s run a couple of POST requests:

curl --location --request POST 'localhost:8080/test' \
--header 'Content-Type: application/json' \
--data-raw '{
    "message": "SomeMessage"
}'

As a result, we should see our messages printed to the output, like these:

Message received: [Some Example Message]
Message received: [Another one]

// ExampleDto.kt file:
data class ExampleDto(val someMessage: String)

// UserDto.kt file:
data class UserDto(val id: Long, val name: String)

const val EXAMPLE_TOPIC_NAME_TWO = "someTopicTwo"
const val EXAMPLE_TOPIC_NAME_THREE = "someTopicThree"

@Component
class ExampleJsonProducer(
  private val exampleDtoKafkaTemplate: KafkaTemplate<String, ExampleDto>,
  private val userDtoKafkaTemplate: KafkaTemplate<String, UserDto>
) {

  fun sendExampleDtoMessage(dto: ExampleDto) {
    exampleDtoKafkaTemplate.send(EXAMPLE_TOPIC_NAME_TWO, dto)
  }

  fun sendUserDtoMessage(dto: UserDto) {
    userDtoKafkaTemplate.send(EXAMPLE_TOPIC_NAME_THREE, dto)
  }
}

@KafkaListener(topics = [EXAMPLE_TOPIC_NAME_TWO], groupId = GROUP_ID)
fun secondListener(message: ExampleDto) {
  logger.info("Message received: [$message]")
}

@KafkaListener(topics = [EXAMPLE_TOPIC_NAME_THREE], groupId = GROUP_ID)
fun secondListener(message: UserDto) {
  logger.info("Message received: [$message]")
}

@RestController
class ExampleController(
  private val exampleStringProducer: ExampleStringProducer,
  private val exampleJsonProducer: ExampleJsonProducer
)

exampleJsonProducer.sendExampleDtoMessage(
  dto = ExampleDto(requestBody.message)
)
exampleJsonProducer.sendUserDtoMessage(
  dto = UserDto(
    id = Random.nextLong(0, 100),
    name = requestBody.message
  )
)

spring:
  kafka:
    consumer:
      bootstrap-servers: localhost:8098
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
    producer:
      bootstrap-servers: localhost:8098
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

curl --location --request POST 'localhost:8080/test' \
--header 'Content-Type: application/json' \
--data-raw '{
    "message": "SomeMessage"
}'

spring:
  kafka:
    consumer:
      bootstrap-servers: localhost:8098
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring:
          deserializer:
            value:
              delegate:
                class: org.springframework.kafka.support.serializer.JsonDeserializer
    producer:
      bootstrap-servers: localhost:8098
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

curl --location --request POST 'localhost:8080/test' \
--header 'Content-Type: application/json' \
--data-raw '{
    "message": "SomeMessage"
}'

The class ‘com.codersee.kafkaexample.dto.ExampleDto’ is not in the trusted packages: [java.util, java.lang]. If you believe this class is safe to deserialize, please provide its name. If the serialization is only done by a trusted source, you can also enable trust all (*).

Similarly, to fix this issue we have to add new property- properties.spring.json.trusted.packages– and either point to the package, where our DTOs live or set a wildcard.

In my case, the final application.yaml file looks, as follows:

spring:
  kafka:
    consumer:
      bootstrap-servers: localhost:8098
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring:
          json:
            trusted:
              packages: "com.codersee.kafkaexample.dto"
          deserializer:
            value:
              delegate:
                class: org.springframework.kafka.support.serializer.JsonDeserializer
    producer:
      bootstrap-servers: localhost:8098
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

4.9. Final Testing

Finally, we can rerun our beloved Spring Boot Kafka with Kotlin application and try the request once again:

curl --location --request POST 'localhost:8080/test' \
--header 'Content-Type: application/json' \
--data-raw '{
    "message": "SomeMessage"
}'

As a result, we should see the following:

Message received: [SomeMessage]
Message received: [ExampleDto(someMessage=SomeMessage)]
Message received: [UserDto(id=26, name=SomeMessage)]

As we can see, all 3 Apache Kafka producers and consumers are working, as expected (and if no, then let me know in the comment section).

5. Apache Kafka With Spring Boot and Kotlin Summary

And that would be all for this article about how to work with Apache Kafka With Spring Boot and Kotlin. If you’d like to see the whole project source code, then visit this GitHub repository.

I really hope that you enjoyed this step-by-step guide and will be delighted if you would like to share your feedback with me and the others in the comments section below.

Take care and have a great day! 😀

The post Apache Kafka With Spring Boot and Kotlin appeared first on Codersee blog- Kotlin on the backend.

In this last of the 3 articles about setting up Apache Kafka, I would like to show you how to set up Apache Kafka Without Zookeeper using Docker Compose.

After reading it, you will know precisely:

• what does the Docker Compose configuration look like to set up 3 Kafka Brokers,
• what environment variables we have to set,
• and what workaround we have to do in order to make it run.

If after finishing it, you will still be eager to broaden your knowledge, then you can find the previous two articles right here:

• How To Deploy Multiple Kafka Brokers With Docker Compose?
• How To Set Up Apache Kafka With Docker?

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. Apache Kafka Without Zookeeper

Before we dive into the practice part, let’s take a second to understand the motivation behind the Zookeeper removal (I’ve already written about Zookeeper responsibilities in the first part, so I’ll skip it here).

To put it simply, the motivation behind it can be summarized with these points:

• simplifying the deployment and configuration process (Zookeeper is a separate system, with its own syntax, configs, and deployment patterns),
• scalability improvement,
• enabling support for more partitions.

The whole change has been introduced for the first time as a KIP-500 (Kafka Improvement Proposal) and has been offered as an early access version with the 2.8.0 release.

If you would like to understand deeply the reasons and arguments behind the Zookeeper removal, I highly encourage you to visit the KIP-500 ticket page linked above.

[elementor-template id=”9007393″]

3. Kafka Without Zookeeper Docker Compose File

With that being said, let’s take a look at the docker-compose.yml file:

version: '3'
services:
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka1:9092,CONTROLLER://kafka1:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9092,CONTROLLER://kafka2:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka3:9092,CONTROLLER://kafka3:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"

Don’t worry if you feel overwhelmed at this point, I will walk you step by step through the configuration values in the following paragraphs.

Nevertheless, don’t run the docker compose up command yet, we will need one more file to run it 🙂

4. Add Necessary Workarounds

As the next step, we need to run a few commands when containers start.

In my example, I will call this file run_workaround.sh. And although the name is not important, it has to match values specified when mounting volumes and running the command from within the containers:

#!/bin/sh

sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure

sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure

echo "kafka-storage format --ignore-formatted -t NqnEdODVKkiLTfJvqd1uqQ== -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure

As we can see, the above script contains exactly 3 commands.

The first one is responsible for turning off the check for the KAFKA_ZOOKEEPER_CONNECT parameter. Without this line, the error will be thrown: Command [/usr/local/bin/dub ensure KAFKA_ZOOKEEPER_CONNECT] FAILED !

The second one ignores the cub zk-ready. Without it, everything will end up with another exception: zk-ready: error: too few arguments.

And finally, the last line is a KRaft required step, which formats the storage directory with a new cluster identifier. Please keep in mind that the specified UUID value (NqnEdODVKkiLTfJvqd1uqQ== in my case) has to be 16 bytes of a base64-encoded UUID.

5. Docker Compose Instructions

With all of the above being said, we can get back to the docker-compose.yml file.

5.1. Volumes and Commands

Although I will skip the things like the services, images, or container names (I’ve covered them in the previous articles), let’s take a look at these 3 lines:

volumes:
  - ./run_workaround.sh:/tmp/run_workaround.sh
command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. With the above setting, we inform Docker that we want the run_workaround.sh file to be mounted inside the tmp directory inside the container with the same name. Of course, we could change the name of this file inside, but then, the change should also be reflected in the command.

When it comes to the command, it simply overrides the default command. With the above config, we instruct Docker to run our container with our custom script.

6. Apache Kafka Brokers Environment Variables

Finally, let’s go through all the environments we had to set in order to make everything work.

6.1. KAFKA_NODE_ID Environment Variable

As the name suggest, the KAFKA_NODE_ID variable is used to specify a unique identifier of our node in the cluster.

6.2. Kafka Listeners and Protocol Mapping

Nextly, let’s see the settings responsible for appropriate hosts and ports mapping:

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP – with this one, we simply define the key-value pairs of listeners’ names with security protocols used with them
• KAFKA_LISTENERS – with this variable, we define all exposed listeners
• KAFKA_ADVERTISED_LISTENERS – this one, on the other hand, contains all listeners used by clients

It’s worth mentioning here that when working with Docker Compose, the container name becomes a hostname- like kafka1, kafka2, and kafka3 in our examples.

6.3. KAFKA_ADVERTISED_LISTENERS Setting

As the next step, let’s see the KAFKA_ADVERTISED_LISTENER. This one, simply allows us to specify a list of coma-separated listeners’ names used by the controller. It’s necessary when running in a KRaft mode.

Additionally, none of the values specified here can be put inside the advertised listeners covered in the previous paragraph. Otherwise, the following error will be thrown:

The advertised.listeners config must not contain KRaft controller listeners from controller.listener.names when process.roles contains the broker role because Kafka clients that send requests via advertised listeners do not send requests to KRaft controllers — they only send requests to KRaft brokers.

6.4. KAFKA_CONTROLLER_QUORUM_VOTERS

As the next one comes the KAFKA_CONTROLLER_QUORUM_VOTERS environment variable. And this one allows us to specify the set of voters in our node. When it comes to the format, values have to be put in the following order: {id}@{host}:{port} .

So, in our case, we do specify 9093 of our nodes, because this is the port exposed as a controller.

6.5. KAFKA_PROCESS_ROLES Variable

And the last one- the KAFKA_PROCESS_ROLES variable lets us specify the role of the particular broker. The value can be set to either broker, controller, or both of them (just like above).

7. Testing

And finally, we can proceed to the part where we will verify if everything is working, as expected. In my example, I will show you how to use Console Producer/Consumer, but feel free to connect with anything you like.

Firstly, let’s start our services with the docker compose up:

The broker has been unfenced. Transitioning from RECOVERY to RUNNING.

After some time, we should see the above for each of our containers.

Additionally, we can validate with a docker ps command:

CONTAINER ID IMAGE                       COMMAND                CREATED            STATUS            PORTS    NAMES
dc3966ba58a4 confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka3
10242d9d1e6a confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka2
c404c866cde6 confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka1

As we can clearly see, every container is up and running. Moreover, the desired 9092 port is exposed for each of them.

When it comes to the testing strategy, I would like to connect to each broker separately and perform a different action:

• firstly, connect to the kafka1 and create a new topic called exampleTopic
• secondly, to kafka2 and produce some messages
• and finally, read all of the messages from within the kafka3

To do so, let’s start with the following:

docker exec -it kafka1 /bin/bash
kafka-topics --bootstrap-server kafka1:9092 --create --topic exampleTopic

# Output:
Created topic exampleTopic.

# To exit the container, please specify exit

As we can see, the exampleTopic was created successfully.

So as the next thing, let’s produce some messages from kafka2 broker:

docker exec -it kafka2 /bin/bash
kafka-console-producer --bootstrap-server kafka2:9092 --topic exampleTopic

>codersee
>rules 

# To exit the producer, please hit Ctrl + D. Then to exit the container please type 'exit'.

No errors have been thrown, which let us assume that everything is good so far.

Nevertheless, to fully make sure, let’s consume these messages:

docker exec -it kafka3 /bin/bash
kafka-console-consumer --bootstrap-server kafka3:9092 --topic exampleTopic --from-beginning

# Output:
codersee
rules

# To exit the consumer, please hit Ctrl + C. As the output, we should see:
^CProcessed a total of 2 messages

# And again, to exit the container please type 'exit'.

As we can see, everything worked as expected and our brokers are correctly communicating within the cluster.

8. Kafka Without Zookeeper Using Docker Compose Summary

And that’s all for today’s article on how to set up Kafka Without Zookeeper using Docker Compose.

I really hope that you enjoyed this and my previous articles in an Apache Kafka series and I will be forever thankful if you would like to share your feedback with me in the comments section below (positive and negative, as well).

Take care and have a great week! 🙂

The post How To Set Up Kafka Without Zookeeper using Docker Compose? appeared first on Codersee blog- Kotlin on the backend.

Hello, in this step-by-step guide, I will show you how to set up multiple Kafka brokers with Docker Compose 🙂

In this article, together we will:

• see the minimum Docker Compose config required to spin up 3 Kafka brokers,
• take a closer look at the configuration,
• get information about active brokers with the zookeeper-shell command,
• verify whether everything is working as expected with Console Producer and Consumer tools.

Unlike the previous article about Kafka, I would like to start this one by showing the configuration you’ll need to set up 3 Kafka brokers. After that, we will go through it as thoroughly as possible to understand it even better and in the end, we will learn how to verify and test our config.

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. Multiple Kafka Brokers Docker Compose

With that being said, let’s see the docker-compose.yaml:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
    ports:
      - "8097:8097"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8097,INTERNAL://kafka1:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
    ports:
      - "8098:8098"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8098,INTERNAL://kafka2:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3
    ports:
      - "8099:8099"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8099,INTERNAL://kafka3:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

[elementor-template id=”9007393″]

2.1. Docker Images Versions and Containers Names

Firstly, let’s take a look at docker images versions and containers names:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
...
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
...
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
...
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3

As we can see, in our example we will use Confluent Community Docker Image for Apache Kafka and Confluent Docker Image for Zookeeper– both in version 7.2.1.

Additionally, we explicitly set names for our containers, which are discoverable at a hostname identical to the container name. Simply put, container exposed port 9092 of kafka1 will be visible for other containers at kafka1:9092.

2.2. Containers Ports

Nextly, let’s have a look at the exposed ports:

...
  kafka1:
    ports:
      - "8097:8097"
...
  kafka2:
    ports:
      - "8098:8098"
...
  kafka3:
    ports:
      - "8099:8099"

As can be seen, we did it for each service except the zookeeper.

You might already know this, but ports is responsible for exposing container ports to the host machine. Simply put, when running docker-compose on our computer with a 123:456 setting, we will be able to access a container’s 456 port through localhost:123.

2.3. Docker Compose Depends On

The last thing, I would like to cover before we head to the environment settings of our Kafka brokers is the depends_on .

As we can see, we put it for each Kafka service specifying the zookeeper service as an argument. With this one, we simply make sure that when running a docker compose up command, the zookeeper service will be started before the brokers.

3. Multiple Kafka Brokers Environment Variables

If you’ve seen my previous article on how to set up a single-node cluster with docker-compose, then you will notice that it looks a bit different. The main reason behind this is that with multiple Kafka brokers in our node, they have to know how to communicate with each other.

3.1. Zookeeper Client Port

As the first step, let’s take a look at the ZOOKEEPER_CLIENT_PORT.

To put it simply, this one instructs Zookeeper where to listen for connections by clients- Kafka brokers in our example.

3.2. Kafka Broker ID

Following, check the KAFKA_BROKER_ID variable.

By default, when we don’t set this one, the Zookeeper will automatically assign a unique broker identifier. Nevertheless, if we would like to specify it explicitly, then we can take care of it with this environment variable. Of course, please keep in mind that this value has to be unique across our Kafka brokers.

3.3. Kafka Zookeeper Connect

As the next thing, we set the KAFKA_ZOOKEEPER_CONNECT in each service to point to the Zookeeper.

As the value, we have to set our Zookeeper container name with a port set with the ZOOKEEPER_CLIENT_PORT directive.

3.4. Kafka Listener Settings

Finally, let’s see the rest of the environment variables responsible for communication:

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
• KAFKA_ADVERTISED_LISTENERS
• KAFKA_INTER_BROKER_LISTENER_NAME

Unlike the previous ones, I believe it has more sense to take a look at all of them at once to get a better understanding.

With a KAFKA_ADVERTISED_LISTENERS, we set a comma-separated list of listeners with their host/IP and port. This value must be set in a multi-node environment because otherwise, clients will attempt to connect to the internal host address. Please keep in mind that the listeners’ values (EXTERNAL and INTERNAL in our case), must be unique here.

When it comes to the naming here, we can specify our own values, just as I did with EXTERNAL/INTERNAL. Nevertheless, in such a case, we have to specify the correct mapping with KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:

• EXTERNAL will be mapped to PLAINTEXT protocol,
• INTERNAL will be mapped the same way, as well

If you are wondering why have we done it, then the reason is mentioned above- listeners’ values have to be unique, so we can’t specify PLAINTEXT://localhost:8099,PLAINTEXT://kafka3:9092 .

Finally, with KAFKA_INTER_BROKER_LISTENER_NAME we explicitly specify which listener to use for inter-broker communication.

4. Multiple Kafka With Docker Compose Testing

With all of that being said, we can finally make use of our docker-compose.yaml file.

To do so, let’s build and run our services with the following command:

docker compose up

Depending on our machine, it may take a while until everything is up and running.

In the end, we should see the following:

...Recorded new controller, from now on will use broker...

4.1. Verify All Containers Running

As the next step, let’s open up a new terminal window and list up and running containers with docker ps:

# Output:
CONTAINER ID IMAGE                           COMMAND                CREATED        STATUS        PORTS                            NAMES
cf892aeb5793 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8098->8098/tcp, 9092/tcp kafka2
a10bef7b6a54 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8099->8099/tcp, 9092/tcp kafka3
46c9ca4862b6 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8097->8097/tcp, 9092/tcp kafka1
eef5ebec8519 confluentinc/cp-zookeeper:7.2.1 "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 2181/tcp, 2888/tcp, 3888/tcp     zookeeper

As we can clearly see, all Kafka containers have their ports exposed to our host and Zookeeper is reachable through 2181, as well.

4.2. Verify Active Kafka Brokers Using zookeeper-shell

Following, let’s run a zookeeper-shell on the zookeeper container:

docker exec -it zookeeper /bin/zookeeper-shell localhost:2181

# Output:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

What’s important to mention here is that we run a zookeeper-shell command from within the container. That’s the reason why the zookeeper_host value is set to localhost and accessible, even though we haven’t exposed it.

Thanks to the -it flags, we can use the shell from our terminal.

4.3. Get IDs of Currently Active Brokers

So, to get the identifiers of currently active Kafka Brokers, we can use the following:

ls /brokers/ids

# Output:
[1, 2, 3]

The above output clearly indicates that everything is working, as expected.

Note: to close the zookeeper-shell, please hit Ctrl + C (and please do it before heading to the Producer/Consumer parts)

5. Publish Messages With Console Producer

After we know all of the above, we can finally send messages using the Console Producer.

As the first step, let’s run bash from a randomly picked Kafka broker container:

docker exec -it kafka2 /bin/bash

Nextly, let’s create a new topic called randomTopic:

kafka-topics --bootstrap-server localhost:9092 --create --topic randomTopic

# Output:
Created topic randomTopic.

Please note, that given the kafka-topics script is run inside of the container, we point to the 9092 port of localhost.

Following, we can produce some messages to our new topic:

kafka-console-producer --bootstrap-server localhost:9092 --topic randomTopic 

> lorem
> ipsum

# When we finish, we have to hit ctrl + d

Of course, we could do that from any other broker container (and in order to exit bash, we need to type exit in the command prompt).

6. Read Messages With Console Consumer

As the last step, let’s connect to another Apache Kafka broker and read previously sent messages:

docker exec -it kafka3 /bin/bash

kafka-console-consumer --bootstrap-server localhost:9092 --topic randomTopic --from-beginning

# Output:
lorem
ipsum

As we can clearly see, the output contains all messages, we’ve sent using Console Producer.

7. Summary

And that would be all for this article on how to set up multiple Apache Kafka brokers using Docker Compose and produce/consume messages. As a follow-up, I would recommend checking out this Confluent article about connecting to Kafka with different configurations.

Finally, let me know your thoughts in the comment section bellow, dear friend! 😀

The post How To Deploy Multiple Kafka Brokers With Docker Compose? appeared first on Codersee blog- Kotlin on the backend.

If you would like to learn how to set up Apache Kafka using Docker Compose then you’ve come to the right place 🙂

In this article, together we will:

• see the bare minimum Docker Compose configuration we must provide to spin up an environment with exactly one Kafka broker and Zookeeper instance,
• take a closer look at the required configurations,
• verify our setup using Console Producer and Consumer tools.

If you will enjoy this tutorial, then I highly encourage you to check out the continuations:

• How To Deploy Multiple Kafka Brokers With Docker Compose?
• How To Set Up Kafka Without Zookeeper using Docker Compose

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. What is Apache Kafka?

Before we start, I feel obliged to provide at least a brief explanation of what Apache Kafka and Zookeeper are. Please skip the following two paragraphs if you want to get straight to the practice part of this tutorial.

Well, Apache Kafka by definition is an open-source distributed event streaming platform. Simply put, it’s a platform widely used to work with real-time streaming data pipelines and to integrate applications to work with such streams. It’s mainly responsible for:

• publishing/subscribing to the stream of records,
• their real-time processing,
• and their ordered storage

But let’s just be honest- it’s almost impossible to describe Kafka in a couple of sentences and I highly encourage you to visit the Apache Intro page after this tutorial right here.

3. What is Apache ZooKeeper?

Or the question should rather be why do I even need another service to work with Apache Kafka?

Just like a zookeeper in a zoo takes care of the animals, Apache Zookeeper “takes care” of all the Kafka brokers in a cluster. It’s mainly responsible for:

• tracking the cluster state- like which brokers are a part of the cluster, sending notifications to Kafka in case of changes
• topics configuration
• controller election
• access control lists and quotas

Nevertheless, please keep in mind that the Kafka team is constantly working on replacing the ZooKeeper with Apache Kafka Raft (KRaft). To be more specific- the 2.8.0 version introduced early access to this functionality, but at the moment of publishing this article, it’s still not production-ready.

4. Bare Minimum Kafka Docker Compose

With all of that being said, let’s have a look at the bare minimum version of the docker-compose.yml file. This configuration is necessary to spin up exactly 1 Kafka and Zookeeper instance:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka
    ports:
      - "8098:8098"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:8098
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Again, before we head to the next commands, let’s take a couple of minutes to understand, what exactly is going on here.

4.1. Docker Images Version And Containers

Firstly, let’s take a look at this piece of code:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
...   
  kafka:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka

As we can see, with this config we will spin up two services named kafka and zookeeper. The container_name will set specified names for our containers, instead of letting Docker generate them. Finally, we will use the 7.2.1 versions of Confluent Community Docker Image for Apache Kafka and Confluent Docker Image for Zookeeper.

4.2. Additional Kafka Container Configs

As the next step, let’s have a quick look at these four lines:

ports:
  - "8098:8098"
depends_on:
  - zookeeper

As the name suggests, the ports command is responsible for exposing ports, so with this setting, port 8098 of our container will be accessible through the 8098 port of the host machine. Additionally, the depends_on will make sure to start the zookeeper container before the kafka.

5. Required Settings

As the next step, let’s take a closer look at the required settings for both Kafka and Zookeeper in our Docker Compose settings.

5.1. Required Zookeeper Settings

As we can see in our example, the only environment we specified is ZOOKEEPER_CLIENT_PORT. This one instructs Zookeeper where it should listen for connections by clients- Kafka in our case.

Additionally, when running in clustered mode, we have to set the ZOOKEEPER_SERVER_ID (but it’s not the case here).

5.2. Required Confluent Kafka Settings

When it comes to the Confluent Kafka Docker image, here is the list of the required environment variables:

• KAFKA_ZOOKEEPER_CONNECT – instructing Kafka where it can find the Zookeeper
• KAFKA_ADVERTISED_LISTENERS – specifying the advertised hostname, which clients can reach out to. Moreover, this value is sent to the Zookeeper

Please keep in mind, that in order to run only one instance of Kafka (aka single-node cluster), we have to additionally specify KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR set to 1. Its default value is 3 and if we don’t do that, we will get into this error: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1

6. Verify Kafka Docker Compose Config

With all of that being said, we can finally check out if everything is working, as expected.

6.1. Start Containers

As the first step, we have to start our containers:

docker compose up -d

The -d flag instructs the docker to run containers in a detached mode.

Of course, we can verify everything by running the docker ps command:

# example output:
CONTAINER ID IMAGE                           COMMAND                CREATED        STATUS       PORTS                            NAMES
38b41bc43676 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 42 seconds ago Up 4 seconds 0.0.0.0:8098->8098/tcp, 9092/tcp kafka
45484184f330 confluentinc/cp-zookeeper:7.2.1 "/etc/confluent/dock…" 42 seconds ago Up 5 seconds 2181/tcp, 2888/tcp, 3888/tcp     zookeeper

6.2. Create Kafka Topic

Nextly, let’s create a new Kafka topic:

docker exec kafka kafka-topics --bootstrap-server localhost:8098 --create --topic example

As a word of explanation, the kafka-topics is a shell script that allows us to execute a TopicCommand tool. It’s a command-line tool that can manage and list topics in a Kafka cluster. Additionally, we have to specify the listener hostname (specified with KAFKA_ADVERTISED_LISTENERS) with –bootstrap-server.

When the topic is created we should see the following message: “Created topic example.”

6.3. Run Kafka Producer

Following, let’s run a console producer with the kafka-console-producer:

docker exec --interactive --tty kafka kafka-console-producer --bootstrap-server localhost:8098 --topic example

Please keep in mind that this command will start the producer and it will wait for our input (and you should notice the > sign). Please specify a couple of messages and hit Ctrl + D after you finish:

>lorem
>ipsum
# hit Ctrl + D

6.4. Run Kafka Consumer

As the last step, let’s run a console consumer with the kafka-console-consumer command:

docker exec --interactive --tty kafka kafka-console-consumer --bootstrap-server localhost:8098 --topic example --from-beginning

This time, we should see that our messages are printed to the output successfully (and to finish please hit Ctrl+ C):

lorem
ipsum

7. Kafka With Docker Compose Summary

And that would be all for this step-by-step guide on How To Set Up Apache Kafka With Docker Compose.

Let me know about your thoughts in the comment section, or reach out through the contact form- I highly appreciate your thoughts, comments and feedback.

Take care brother/sister!

The post How To Set Up Apache Kafka With Docker? appeared first on Codersee blog- Kotlin on the backend.