To be even more specific- we will work with an application that exposes a REST API and connects to MongoDB. If you would like to learn how to do that step-by-step, then you can check out my other article. (But don’t worry, we will see its code snippets in this tutorial, too).

For testing, we’re going to use JUnit 5, MockK, REST Assured, as well as the @MicronautTest and Micronaut Test Resources.

Video Tutorial

If you prefer video content, then check out my video:

If you find this content useful, please leave a subscription  😉

Micronaut Project Overview

Let’s start everything by checking the project we will test today. Of course, we will focus on the most important parts, and for the full source code, I invite you to this GitHub repository.

Application Properties

In our project, we introduce 3 application properties files:

// application.properties
micronaut.application.name=micronaut-testing

// application-mongo.properties
mongodb.uri=mongodb://localhost:27017/example

// application-test.properties
mongodb.package-names=com.codersee.model

The application name is the default property inserted when generating a new project.

Nevertheless, the two remaining are created on purpose. With their names- *mongo, *test– we instruct Micronaut that it should consider them only when mongo and test profiles are active.

But why this way? Well, to avoid two issues.

MongoDB Connection Issues with Micronaut Test Resources

So, let’s start everything by explaining why we don’t put the Mongo URI inside the main properties.

Well, when we use the Micronaut Test Resources, MongoDB support will start a MongoDB container and provide the value of the mongodb.uri property.

Nevertheless, this won’t happen when we explicitly set the URI in the application properties. And that’s why we want to make this setting available only when we set the environment to mongo.

CodecCacheKey Issue

Additionally, if we don’t specify explicitly the package name, where our model classes live, we’re going to end up with the following issue when running our Micronaut tests:

[io-executor-thread-1] ERROR i.m.http.server.RouteExecutor - Unexpected error occurred: Can't find a codec for CodecCacheKey{clazz=class com.codersee.model.AppUser, types=null}.
org.bson.codecs.configuration.CodecConfigurationException: Can't find a codec for CodecCacheKey{clazz=class com.codersee.model.AppUser, types=null}.
  at org.bson.internal.ProvidersCodecRegistry.lambda$get$0(ProvidersCodecRegistry.java:87)
  at java.base/java.util.Optional.orElseGet(Optional.java:364)
  at org.bson.internal.ProvidersCodecRegistry.get(ProvidersCodecRegistry.java:80)
  at org.bson.internal.ProvidersCodecRegistry.get(ProvidersCodecRegistry.java:50)
  at com.mongodb.internal.operation.Operations.createFindOperation(Operations.java:188)
...

To me, looks like a bug. But thankfully, we can easily avoid that by putting that in the application-test.properties (which is used in tests by default).

Models

Following, we introduce a bunch of models:

// Address.kt: 

import io.micronaut.serde.annotation.Serdeable

@Serdeable.Serializable
@Serdeable.Deserializable
data class Address(
  val street: String,
  val city: String,
  val code: Int
)

// AppUser.kt:

import io.micronaut.data.annotation.GeneratedValue
import io.micronaut.data.annotation.Id
import io.micronaut.data.annotation.MappedEntity

@MappedEntity
data class AppUser(
  @field:Id
  @field:GeneratedValue
  val id: String? = null,
  val firstName: String,
  val lastName: String,
  val email: String,
  val address: Address
)

// AppUserRequest.kt: 

import io.micronaut.serde.annotation.Serdeable

@Serdeable.Deserializable
data class AppUserRequest(
  val firstName: String,
  val lastName: String,
  val email: String,
  val street: String,
  val city: String,
  val code: Int
)

// SearchRequest.kt:

import io.micronaut.serde.annotation.Serdeable

@Serdeable.Deserializable
data class SearchRequest(val name: String)

Those classes are either used to persist/retrieve data from MongoDB or when deserializing JSON payloads into the objects.

Repository

Following, we have a repository layer responsible for communicating with our storage:

// AppUserRepository.kt: 

import com.codersee.model.AppUser
import io.micronaut.data.mongodb.annotation.MongoFindQuery
import io.micronaut.data.mongodb.annotation.MongoRepository
import io.micronaut.data.repository.CrudRepository

@MongoRepository
interface AppUserRepository : CrudRepository<AppUser, String> {

  fun findByFirstNameEquals(firstName: String): List<AppUser>

  @MongoFindQuery("{ firstName: { \$regex: :name}}")
  fun findByFirstNameLike(name: String): List<AppUser>
}

Service

Nextly, the service layer, where we inject the repository and encapsulate its methods:

// AppUserService.kt:

import com.codersee.model.Address
import com.codersee.model.AppUser
import com.codersee.model.AppUserRequest
import com.codersee.repository.AppUserRepository
import io.micronaut.http.HttpStatus
import io.micronaut.http.exceptions.HttpStatusException
import jakarta.inject.Singleton

@Singleton
class AppUserService(
  private val appUserRepository: AppUserRepository
) {

  fun create(userRequest: AppUserRequest): AppUser =
    appUserRepository.save(
      userRequest.toAppUserEntity()
    )

  fun findAll(): List<AppUser> =
    appUserRepository
      .findAll()
      .toList()

  fun findById(id: String): AppUser =
    appUserRepository.findById(id)
      .orElseThrow { HttpStatusException(HttpStatus.NOT_FOUND, "User with id: $id was not found.") }

  fun update(id: String, updateRequest: AppUserRequest): AppUser {
    val foundUser = findById(id)

    val updatedEntity =
      updateRequest
        .toAppUserEntity()
        .copy(id = foundUser.id)

    return appUserRepository.update(updatedEntity)
  }

  fun deleteById(id: String) {
    val foundUser = findById(id)

    appUserRepository.delete(foundUser)
  }

  fun findByNameLike(name: String): List<AppUser> =
    appUserRepository
      .findByFirstNameLike(name)

  private fun AppUserRequest.toAppUserEntity(): AppUser {
    val address = Address(
      street = this.street,
      city = this.city,
      code = this.code
    )

    return AppUser(
      id = null,
      firstName = this.firstName,
      lastName = this.lastName,
      email = this.email,
      address = address
    )
  }
}

Controller

And lastly, the place where we expose our REST endpoints:

// AppUserController.kt: 

import com.codersee.model.AppUserRequest
import com.codersee.service.AppUserService
import com.codersee.model.SearchRequest
import com.codersee.model.AppUser
import io.micronaut.http.HttpStatus
import io.micronaut.http.annotation.*
import io.micronaut.scheduling.TaskExecutors
import io.micronaut.scheduling.annotation.ExecuteOn

@Controller("/users")
@ExecuteOn(TaskExecutors.IO)
class AppUserController(
  private val appUserService: AppUserService
) {

  @Get
  fun findAllUsers(): List<AppUser> =
    appUserService.findAll()

  @Get("/{id}")
  fun findById(@PathVariable id: String): AppUser =
    appUserService.findById(id)

  @Post
  @Status(HttpStatus.CREATED)
  fun createUser(@Body request: AppUserRequest): AppUser =
    appUserService.create(request)

  @Post("/search")
  fun searchUsers(@Body searchRequest: SearchRequest): List<AppUser> =
    appUserService.findByNameLike(
      name = searchRequest.name
    )

  @Put("/{id}")
  fun updateById(
    @PathVariable id: String,
    @Body request: AppUserRequest
  ): AppUser =
    appUserService.update(id, request)

  @Delete("/{id}")
  @Status(HttpStatus.NO_CONTENT)
  fun deleteById(@PathVariable id: String) =
    appUserService.deleteById(id)
}

Simple Kotlin Testing in Micronaut

Excellent. At this point, we know what exactly we are going to test.

And as the first approach we’re going to see will be a plain, old unit test.

Why?

Because at the end of the day, that’s what we will be dealing with most of the time.

So with that said, let’s take a look at the example test:

  import com.codersee.repository.AppUserRepository
import io.mockk.every
import io.mockk.mockk
import io.mockk.verify
import org.junit.jupiter.api.Assertions.assertTrue
import org.junit.jupiter.api.Test

class AppUserServiceTest {

  private val appUserRepository = mockk<AppUserRepository>()

  private val appUserService = AppUserService(appUserRepository)

  @Test
  fun `should return empty user list`() {
    every {
      appUserRepository.findAll()
    } returns emptyList()

    val result = appUserService.findAll()

    assertTrue(result.isEmpty())

    verify(exactly = 1) { appUserRepository.findAll() }
  }
}

As we can see above, we simply mock (with MockK) the AppUserRepository and inject it into the AppUserService instance.

Then, we specify that all invocations of the findAll method must return the empty list (with every).

Lastly, we assert that the findAll method from our service returns an empty list (assertTrue) and that the findAll method from the repository was invoked only once (verify).

Nevertheless, we’re not going to spend too much here right now as this tutorial focuses on Micronaut testing in Kotlin. If you are interested in learning more about these tools, then let me know in the comments section 🙂

Integration Testing with @MicronautTest

With all of that done, let’s see how the Micronaut framework helps us with testing and a few interesting cases that will help us in real life.

What is a @MicronautTest?

Let’s start everything by explaining what exactly is the @MicronautTest.

In simple words, it’s an annotation that we can put on the test class to mark it a Micronaut test (no sh… Sherlock 🙂 :

import io.micronaut.test.extensions.junit5.annotation.MicronautTest

@MicronautTest
class SomeTest {
  // tests
}

In practice, it means that when we run a test with that annotation, it will run a real application. It will use internal Micronaut features with no mocking. So at this point, we can clearly see that this will be the right solution for integration testing.

Moreover, this annotation can be used not only with JUnit 5, but also with Spock, Kotest, and Kotest 5.

@MicronautTest Configuration Options

The @MicronautTest allows us to configure a few properties, like the environments we want to run our test with, the packages to scan, or whether the automatic transaction wrapping should be enabled, or not:

import io.micronaut.test.extensions.junit5.annotation.MicronautTest

@MicronautTest(
  environments = ["env-1, env-2"],
  packages = ["com.codersee.somepackage"],
  transactional = false
)
class SomeTest {
  // tests
}

For the full list of options, I highly encourage you to check out the docs (we can do that for instance in IntelliJ IDEA by clicking on the annotation with the left mouse button when we keep the left ctrl pressed).

What Are Micronaut Test Resources?

At the beginning of this tutorial, I mentioned the Micronaut Test Resources and I would like to make sure that we are on the same page with them.

So basically, Micronaute Test Resources integrates seamlessly with Testcontainers to provide throwaway containers for testing. But, we need to remember that Testcontainers require Docker-API compatible container runtime (in simple words- either Docker installed locally, or the Testcontainers Cloud).

In our case, we would like to do integration tests that require MongoDB. We could make use of the real database (which sometimes may be the case for you), provide some H2 database (not the best approach), or integrate Testcontainers. Thankfully, we don’t need to do that and the only thing we need is the appropriate import in our project:

plugins {
    id("io.micronaut.test-resources") version "4.2.1"
}

As a reminder, I want to emphasize that test resources will be used only when the datasources.default.url is missing. (and that’s why we removed the URI for MongoDB from tests).

Integration Tests With REST Assured

With all of that said, let’s combine everything together and test our Kotlin Micronaut application with JUnit 5 and REST Assured.

To do so, we must remember to add the necessary import:

testImplementation("io.micronaut.test:micronaut-test-rest-assured")

Verify Status Codes and Headers

Let’s start with a simple request to the GET /users endpoint:

@MicronautTest
class AppUserControllerTestWithoutMocking {

  @Test
  fun `should return 200 OK on GET users`(spec: RequestSpecification) {
    spec
      .`when`()
      .get("/users")
      .then()
      .statusCode(200)
      .header("Content-Type", "application/json")
  }

}

As I mentioned previously- by default, with @MicronautTest the real application is started. Moreover, nothing here is mocked, so the Mongo test container delivered by the Micronaut Test Resources is used.

Thanks to the dedicated module we added, we can inject the RequestSpecification into our tests and easily validate our endpoint.

In this case, we perform a GET request and verify that the response status code is 200 OK and the response Content-Type header value is application/json.

Verify 404 Not Found

Similarly, we can verify that no entry is present in the database:

@Test
fun `should return 404 Not Found on GET user by ID`(spec: RequestSpecification) {
  spec
    .`when`()
    .get("/users/123123123123123123121231")
    .then()
    .statusCode(404)
}

This test will be also good proof that entries inserted to MongoDB in other tests do not affect other tests.

Extract and JSON Array in REST Assured

Nextly, let’s take a look at the more complicated example:

@Test
fun `should create a user`(spec: RequestSpecification) {
  val request = AppUserRequest(
    firstName = "Piotr",
    lastName = "Wolak",
    email = "contact@codersee.com",
    street = "Street",
    city = "City",
    code = 123,
  )

  spec
    .`when`()
    .contentType(ContentType.JSON)
    .body(request)
    .post("/users")
    .then()
    .statusCode(201)

  val list = spec
    .`when`()
    .get("/users")
    .then()
    .statusCode(200)
    .body("size()", `is`(1))
    .extract()
    .`as`(object : TypeRef<List<AppUser>>() {})

  assertEquals(1, list.size)

  val createdUser = list.first()

  assertEquals(request.firstName, createdUser.firstName)
  assertEquals(request.street, createdUser.address.street)
}

This time, we do a bunch of more interesting things.

Firstly, we make the POST request with a request body and verify that 201 Creates is returned.

After that, we call the GET /users endpoint to get a list of users. When we get the response, we verify that the JSON array size is equal to 1 (with the .body("size()", is(1)) call). Lastly, we make use of the extract().`as` to convert the payload into the List of AppUser instances. Thanks to that we can easily perform assertions with JUnit 5 functions.

Add Kotlin Utils For REST Assured

As you probably know, when and as are keywords in Kotlin and that’s why we had to use backticks (“).

And to make our lives easier, let’s add the util package in test and create the TestUtil.kt:

import io.restassured.common.mapper.TypeRef
import io.restassured.response.ValidatableResponse
import io.restassured.specification.RequestSpecification


fun RequestSpecification.whenever(): RequestSpecification {
  return this.`when`()
}

fun <T> ValidatableResponse.extractAs(clazz: Class<T>) =
  this.extract()
    .`as`(clazz)

fun <T> ValidatableResponse.extractAs(typeRef: TypeRef<T>) =
  this.extract()
    .`as`(typeRef)

As a result, from now on we can simply use the whenever() instead of `when`() and extractAs() instead of extract().`as`().

Reference The Server / Context

As the next step, let’s take a look at how to reference the server or current application context:

@MicronautTest
class InjectingExample{

  @Inject
  private lateinit var context: ApplicationContext

  @Inject
  private lateinit var server: EmbeddedServer

  // tests
}

Sometimes it can be useful, and as we can see, we can simply inject them using the @Inject annotation.

Conditionally Run Tests

After that, let’s see how we can skip tests execution.

Why would we need that?

Well, integration tests sometimes can take a lot of time, and the bigger our project becomes, the bigger the chance they become annoying. And because of that, generally, it’s a good approach to somehow separate them from the faster unit tests.

When testing in Micronaut, we can easily achieve that using the @Requires annotation:

import io.micronaut.context.annotation.Requires
import io.micronaut.test.extensions.junit5.annotation.MicronautTest
import org.junit.jupiter.api.Assertions.assertTrue
import org.junit.jupiter.api.Test

@MicronautTest
@Requires(env = ["integration-test"])
class SomeIntegrationTest {

  @Test
  fun `should perform integration test`() {
    assertTrue(false)
  }

}

The @MicronautTest annotation turns tests into beans. And that’s why we can use the @Requires, just like we would do with a “standard” bean.

As a result, tests inside the SomeIntegrationTest will run only, when the integration-test the environment is active. (And we can set that for example by setting the environment variable- MICRONAUT_ENVIRONMENTS=integration-test).

Testing Micronaut With MockK Mocks

As the last thing in our tutorial about testing Micronaut with Kotlin, let’s take a look at how we can mock things using the MockK.

Could we use Mockito? Yes. However, I find MockK better for Kotlin (DSLs <3 ).

In order to mock a bean in Micronaut, the only thing we need to do is annotate the method/inner class with the @MockBean annotation:

@MicronautTest
class AppUserControllerTestWithMocking {

  private val repo: AppUserRepository = mockk<AppUserRepository>()

  @MockBean(AppUserRepository::class)
  fun appUserRepository(): AppUserRepository = repo
}

As we can see, we introduce the AppUserRepository mock as the repo property, so that later, we will be able to refer to it easily in our test cases. Additionally, we add the function annotated with @MockBean, thus informing Micronaut that the AppUserRepository is the type we want to replace with our mock.

Moreover, thanks to this approach this mock will be limited only to tests defined in this class. So we can “rest assured” (hue hue) it won’t interfere with other classes.

As a result, our tests will look, as follows:

@MicronautTest
class AppUserControllerTestWithMocking {

  private val repo: AppUserRepository = mockk<AppUserRepository>()

  @Test
  fun `should return 200 OK on GET users`(spec: RequestSpecification) {
    every {
      repo.findAll()
    } returns emptyList()

    spec
      .whenever()
      .get("/users")
      .then()
      .statusCode(200)
      .header("Content-Type", "application/json")
  }

  @Test
  fun `should return user by ID`(spec: RequestSpecification) {
    val foundUser = AppUser(
      id = "123",
      firstName = "Piotr",
      lastName = "Wolak",
      email = "contact@codersee.com",
      address = Address(
        street = "street",
        city = "city",
        code = 123
      )
    )

    every {
      repo.findById("123")
    } returns Optional.of(foundUser)

    spec
      .whenever()
      .get("/users/123")
      .then()
      .statusCode(200)
      .body("id", equalTo("123"))
      .body("firstName", equalTo("Piotr"))
      .body("address.street", equalTo("street"))
      .body("address.code", equalTo(123))
  }

  @Test
  fun `should return user by ID and verify with extract`(spec: RequestSpecification) {
    val foundUser = AppUser(
      id = "123",
      firstName = "Piotr",
      lastName = "Wolak",
      email = "contact@codersee.com",
      address = Address(
        street = "street",
        city = "city",
        code = 123
      )
    )

    every {
      repo.findById("123")
    } returns Optional.of(foundUser)

    val extracted = spec
      .whenever()
      .get("/users/123")
      .then()
      .statusCode(200)
      .extractAs(AppUser::class.java)

    assertEquals(foundUser, extracted)
  }

  @Test
  fun `should create a user`(spec: RequestSpecification) {
    val request = AppUserRequest(
      firstName = "Piotr",
      lastName = "Wolak",
      email = "contact@codersee.com",
      street = "Street",
      city = "City",
      code = 123,
    )

    val createdUser = AppUser(
      id = "123",
      firstName = "Piotr",
      lastName = "Wolak",
      email = "contact@codersee.com",
      address = Address(
        street = "street",
        city = "city",
        code = 123
      )
    )

    every {
      repo.save(any())
    } returns createdUser

    val extracted = spec
      .whenever()
      .contentType(ContentType.JSON)
      .body(request)
      .post("/users")
      .then()
      .statusCode(201)
      .extractAs(AppUser::class.java)

    assertEquals(createdUser, extracted)
  }

  @MockBean(AppUserRepository::class)
  fun appUserRepository(): AppUserRepository = repo
}

Summary

And that’s all for this tutorial on how to perform testing in Micronaut with Kotlin.

Together, we’ve discovered a bunch of interesting things that I hope will be useful in your projects.

If you would like to see the codebase for this tutorial, then check out this GitHub repository. Or, if you are interested in learning more about Micronaut, then check out my other posts.

Let me know your thoughts in the comments section below! 🙂

The post Testing Micronaut Application in Kotlin appeared first on Codersee blog- Kotlin on the backend.

Hello and welcome to my next step-by-step guide, in which we will learn how to deploy a Ktor Server application using Docker.

Firstly, we will prepare a simple app with a REST endpoint and a database connection. This way, we will see one of the most common issues and how we can deal with it.

Finally, I will show you three approaches you can use in your project.

Create a Ktor Server Skeleton

So, as the first step, let’s navigate to the https://start.ktor.io/ page and generate a Ktor Server project.

Note: If you are using IntelliJ IDEA Ultimate Edition, then you can generate a new project with it, as well.

Please specify whatever values you want, except the “Configuration in”- let’s stick to the HOCON file to be on the same page:

Nextly, let’s click the “Add plugins” button and add the “Routing” plugin:

After that, let’s hit the “Generate project” button, save it to our machine, and import it to the IDE (for example, IntelliJ IDEA).

Configure Database Connection

Although with all of that done we are ready to learn how to deploy a Ktor server with Docker, let me show you one more thing. Most of the time, when creating server-side applications, we will have to connect to some database. And then, we can easily end up with connection issues, which I would like to show you in this tutorial.

Of course, if that’s not the case in your project, then please feel free to skip this part. Otherwise, I do recommend going through this part and adjusting it to your needs.

Configure Imports

Firstly, let’s modify the build.gradle.kts file:

val mysql_connector_version: String by project
val exposed_version: String by project

dependencies {
  // other dependencies
  implementation("org.jetbrains.exposed:exposed-core:$exposed_version")
  implementation("org.jetbrains.exposed:exposed-dao:$exposed_version")
  implementation("org.jetbrains.exposed:exposed-jdbc:$exposed_version")

  implementation("com.mysql:mysql-connector-j:$mysql_connector_version")
}

Of course, with this approach we need to add two more lines to gradle.properties, as well:

mysql_connector_version=8.0.31
exposed_version = 0.41.1

As we can see, we just added the Exposed– an ORM framework for Kotlin- and MySQL Connector/J– a JDBC Type 4 driver for MySQL.

Database Config

As the next step, let’s add a couple of lines to the application.conf:

db {
  url = ${DB_URL}
  user = ${DB_USER}
  password = ${DB_PASSWORD}
  driver = "com.mysql.cj.jdbc.Driver"
}

As can be seen, we externalized the URL, user, and password settings to the environment variables.

This way, we can easily provide different values (for example, for different environments) without modifying the code. Nevertheless, we don’t do that for the driver, as this should be constant across all envs.

Following, let’s head to the plugins directory and create the Database.kt file:

fun Application.configureDatabase() {
  val url = environment.config.property("db.url").getString()
  val user = environment.config.property("db.user").getString()
  val password = environment.config.property("db.password").getString()
  val driverClassName = environment.config.property("db.driver").getString()

  Database.connect(
      url = url,
      driver = driverClassName,
      user = user,
      password = password
  )

  transaction {
      AppUser.all()
          .toList()
  }
}

object AppUsers : IntIdTable(name = "app_user") {
  val email = varchar("email", length = 255).uniqueIndex()
}

class AppUser(id: EntityID) : IntEntity(id) {
  companion object : IntEntityClass(AppUsers)

  var email by AppUsers.email
}

To put it simply, the above code is responsible for making a connection to the database based on values from application.conf file and fetching all users from the “app_user” table.

Verification

With all of that being done, we can run the application and verify the output.

If we don’t pass any environment variables, we will see the following:

Exception in thread “main” com.typesafe.config.ConfigException$UnresolvedSubstitution: application.conf Could not resolve substitution to a value: ${DB_PASSWORD}

Process finished with exit code 1

In IntelliJ, we can set variables by editing the running configuration:

Please note that values should not be quoted (example: “value”), otherwise, we can end up with something like this:

Exception in thread “main” java.lang.IllegalStateException: Can’t resolve dialect for connection: …
Process finished with exit code 1

So, if everything is set correctly, we should see the following:

# Logs:
INFO Application - Autoreload is disabled because the development mode is off.
DEBUG Exposed - SELECT app_user.id, app_user.email FROM app_user
INFO Application - Application started in 0.801 seconds.
INFO Application - Responding at http://127.0.0.1:8080

# And for request:
GET http://localhost:8080/

# Response:
Hello World!

And although we didn’t do anything, the Routing.kt file with this endpoint was added automatically when generating the project.

Deploy Ktor Server With Docker – Manual Approach

So finally, we can see the first- manual- approach with which we can deploy the Ktor app with Docker.

Create Dockerfile

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

To do so, let’s a file named Dockerfile to the root of the project:

FROM openjdk:17-jdk-alpine3.14
RUN mkdir /app
COPY ./build/libs/com.codersee.ktor-docker-all.jar /app/app.jar
ENTRYPOINT ["java","-jar","/app/app.jar"]

As we can see, we will use the OpenJDK alpine image and run a built jar.

Build The Image

As the next step, let’s build a jar file using a gradle wrapper:

./gradlew build

Note: if the build is failing, then please navigate to the test directory and comment out the test.

If everything is fine, we should see the “BUILD SUCCESSFUL” message and we can create a Docker image:

docker build -t my-example-app .

# Verification: 
docker images

# Result:
REPOSITORY     TAG    IMAGE ID     CREATED       SIZE
my-example-app latest f50b5432ea63 9 seconds ago 346MB

As we can see, the image was created successfully and we can proceed.

Docker Run

With all of that done, we can deploy our Ktor Server using the docker run command:

docker run -p 8080:8080 --name example-container -e DB_URL=jdbc:mysql://host.docker.internal:3306/boards -e DB_USER=root -e DB_PASSWORD=p@ssword my-example-app

# Output: 
2023-01-13 16:30:43.854 [main] INFO Application - Autoreload is disabled because the development mode is off.
2023-01-13 16:30:44.537 [main] DEBUG Exposed - SELECT app_user.id, app_user.email FROM app_user
2023-01-13 16:30:44.543 [main] INFO Application - Application started in 0.719 seconds.
2023-01-13 16:30:44.746 [DefaultDispatcher-worker-1] INFO Application - Responding at http://0.0.0.0:8080

As can be seen, the server is up and running, which means that it successfully connected to our MySQL instance.

Note: Please note that when working with Docker on your local machine, you can’t provide localhost as a host value for the connection anymore.

Nevertheless, the host.docker.internal is supported for:

• Docker-for-mac or Docker-for-Windows 18.03+
• Docker-for-Linux 20.10.0+ (if you started your Docker container with --add-host host.docker.internal:host-gateway)

Of course, there are more ways to go in such a case, but this is the simplest one in my opinion.

Deploy Ktor Server With Docker – Hybrid Approach

With that covered, let’s figure out another approach to deploy the Ktor app using Docker.

This time, we will build a docker image to a .tar file using the Ktor plugin. If you generated the project with me, then we don’t need to do anything. Otherwise, please remember to add it to the build.gradle.kts file:

plugins {
  // Other plugins
  id("io.ktor.plugin") version "2.2.2"
}

Create New Image

And as the next step, let’s run the buildImage command:

./gradlew buildImage

As a result, we should see the jib-image.tar file inside the build directory.

If that’s the case, then let’s load an image from a tar archive with docker load:

docker load -i build/jib-image.tar

# Verification: docker images # Result: 
REPOSITORY        TAG    IMAGE ID     CREATED      SIZE
ktor-docker-image latest 31b35e55ec24 53 years ago 286MB

And although it was not created 53 years ago 🙂 , we can clearly see that the size is reduced compared to the previous approach.

Deploy Server

And finally, let’s verify again.

But this time, we need to specify the ktor-docker-image as the image name:

docker run -p 8080:8080 --name example-container -e DB_URL=jdbc:mysql://host.docker.internal:3306/boards -e DB_USER=root -e DB_PASSWORD=p@ssword ktor-docker-image

# Output: 
2023-01-13 16:30:43.854 [main] INFO Application - Autoreload is disabled because the development mode is off.
2023-01-13 16:30:44.537 [main] DEBUG Exposed - SELECT app_user.id, app_user.email FROM app_user
2023-01-13 16:30:44.543 [main] INFO Application - Application started in 0.719 seconds.
2023-01-13 16:30:44.746 [DefaultDispatcher-worker-1] INFO Application - Responding at http://0.0.0.0:8080

Customization

Of course, we can customize the image name and tag, as well.

To do so, let’s add the following inside the build.gradle.kts:

ktor {
  docker {
    localImageName.set("my-custom-image-name")
    imageTag.set("alpha")
  }
}

So this time, when we repeat the previous steps, we should see the following image my-custom-image-name:alpha.

Deploy Ktor Server With Docker – Built-in Approach

Lastly, I just wanted to mention that the Ktor plugin comes not only with the buildImage task when it comes to Docker.

With this plugin, we can make use of 3 more tasks:

• publishImageToLocalRegistry – which is a combination of the buildImage and gradle load from the previous step paragraph.
• publishImage– which takes care of pushing to an external registry.
• runDocker – which additionally launches the Ktor server to listen on the 8080 port of localhost (unless we specify otherwise).

And although I show this approach as the last one in this tutorial, in my opinion, we should delegate the deployment process to these tasks as often, as it is possible.

Summary

And that’s all for this tutorial on how to deploy a Ktor Server with Docker. Together, we’ve learned 3 approaches, which can help us do that with ease and as always, you can find the source code on GitHub.

If you enjoyed the content or just would like to share your feedback or thoughts, then let me know in the comments section 🙂

Lastly, if you would like to learn a bit more about Ktor, then check out my other articles about Ktor with MongoDB, or Ktor with PostgreSQL.

The post Easily Deploy Your Ktor Server with Docker appeared first on Codersee blog- Kotlin on the backend.

In this last of the 3 articles about setting up Apache Kafka, I would like to show you how to set up Apache Kafka Without Zookeeper using Docker Compose.

After reading it, you will know precisely:

• what does the Docker Compose configuration look like to set up 3 Kafka Brokers,
• what environment variables we have to set,
• and what workaround we have to do in order to make it run.

If after finishing it, you will still be eager to broaden your knowledge, then you can find the previous two articles right here:

• How To Deploy Multiple Kafka Brokers With Docker Compose?
• How To Set Up Apache Kafka With Docker?

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. Apache Kafka Without Zookeeper

Before we dive into the practice part, let’s take a second to understand the motivation behind the Zookeeper removal (I’ve already written about Zookeeper responsibilities in the first part, so I’ll skip it here).

To put it simply, the motivation behind it can be summarized with these points:

• simplifying the deployment and configuration process (Zookeeper is a separate system, with its own syntax, configs, and deployment patterns),
• scalability improvement,
• enabling support for more partitions.

The whole change has been introduced for the first time as a KIP-500 (Kafka Improvement Proposal) and has been offered as an early access version with the 2.8.0 release.

If you would like to understand deeply the reasons and arguments behind the Zookeeper removal, I highly encourage you to visit the KIP-500 ticket page linked above.

[elementor-template id=”9007393″]

3. Kafka Without Zookeeper Docker Compose File

With that being said, let’s take a look at the docker-compose.yml file:

version: '3'
services:
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka1:9092,CONTROLLER://kafka1:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9092,CONTROLLER://kafka2:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka3:9092,CONTROLLER://kafka3:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"

Don’t worry if you feel overwhelmed at this point, I will walk you step by step through the configuration values in the following paragraphs.

Nevertheless, don’t run the docker compose up command yet, we will need one more file to run it 🙂

4. Add Necessary Workarounds

As the next step, we need to run a few commands when containers start.

In my example, I will call this file run_workaround.sh. And although the name is not important, it has to match values specified when mounting volumes and running the command from within the containers:

#!/bin/sh

sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure

sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure

echo "kafka-storage format --ignore-formatted -t NqnEdODVKkiLTfJvqd1uqQ== -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure

As we can see, the above script contains exactly 3 commands.

The first one is responsible for turning off the check for the KAFKA_ZOOKEEPER_CONNECT parameter. Without this line, the error will be thrown: Command [/usr/local/bin/dub ensure KAFKA_ZOOKEEPER_CONNECT] FAILED !

The second one ignores the cub zk-ready. Without it, everything will end up with another exception: zk-ready: error: too few arguments.

And finally, the last line is a KRaft required step, which formats the storage directory with a new cluster identifier. Please keep in mind that the specified UUID value (NqnEdODVKkiLTfJvqd1uqQ== in my case) has to be 16 bytes of a base64-encoded UUID.

5. Docker Compose Instructions

With all of the above being said, we can get back to the docker-compose.yml file.

5.1. Volumes and Commands

Although I will skip the things like the services, images, or container names (I’ve covered them in the previous articles), let’s take a look at these 3 lines:

volumes:
  - ./run_workaround.sh:/tmp/run_workaround.sh
command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. With the above setting, we inform Docker that we want the run_workaround.sh file to be mounted inside the tmp directory inside the container with the same name. Of course, we could change the name of this file inside, but then, the change should also be reflected in the command.

When it comes to the command, it simply overrides the default command. With the above config, we instruct Docker to run our container with our custom script.

6. Apache Kafka Brokers Environment Variables

Finally, let’s go through all the environments we had to set in order to make everything work.

6.1. KAFKA_NODE_ID Environment Variable

As the name suggest, the KAFKA_NODE_ID variable is used to specify a unique identifier of our node in the cluster.

6.2. Kafka Listeners and Protocol Mapping

Nextly, let’s see the settings responsible for appropriate hosts and ports mapping:

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP – with this one, we simply define the key-value pairs of listeners’ names with security protocols used with them
• KAFKA_LISTENERS – with this variable, we define all exposed listeners
• KAFKA_ADVERTISED_LISTENERS – this one, on the other hand, contains all listeners used by clients

It’s worth mentioning here that when working with Docker Compose, the container name becomes a hostname- like kafka1, kafka2, and kafka3 in our examples.

6.3. KAFKA_ADVERTISED_LISTENERS Setting

As the next step, let’s see the KAFKA_ADVERTISED_LISTENER. This one, simply allows us to specify a list of coma-separated listeners’ names used by the controller. It’s necessary when running in a KRaft mode.

Additionally, none of the values specified here can be put inside the advertised listeners covered in the previous paragraph. Otherwise, the following error will be thrown:

The advertised.listeners config must not contain KRaft controller listeners from controller.listener.names when process.roles contains the broker role because Kafka clients that send requests via advertised listeners do not send requests to KRaft controllers — they only send requests to KRaft brokers.

6.4. KAFKA_CONTROLLER_QUORUM_VOTERS

As the next one comes the KAFKA_CONTROLLER_QUORUM_VOTERS environment variable. And this one allows us to specify the set of voters in our node. When it comes to the format, values have to be put in the following order: {id}@{host}:{port} .

So, in our case, we do specify 9093 of our nodes, because this is the port exposed as a controller.

6.5. KAFKA_PROCESS_ROLES Variable

And the last one- the KAFKA_PROCESS_ROLES variable lets us specify the role of the particular broker. The value can be set to either broker, controller, or both of them (just like above).

7. Testing

And finally, we can proceed to the part where we will verify if everything is working, as expected. In my example, I will show you how to use Console Producer/Consumer, but feel free to connect with anything you like.

Firstly, let’s start our services with the docker compose up:

The broker has been unfenced. Transitioning from RECOVERY to RUNNING.

After some time, we should see the above for each of our containers.

Additionally, we can validate with a docker ps command:

CONTAINER ID IMAGE                       COMMAND                CREATED            STATUS            PORTS    NAMES
dc3966ba58a4 confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka3
10242d9d1e6a confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka2
c404c866cde6 confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka1

As we can clearly see, every container is up and running. Moreover, the desired 9092 port is exposed for each of them.

When it comes to the testing strategy, I would like to connect to each broker separately and perform a different action:

• firstly, connect to the kafka1 and create a new topic called exampleTopic
• secondly, to kafka2 and produce some messages
• and finally, read all of the messages from within the kafka3

To do so, let’s start with the following:

docker exec -it kafka1 /bin/bash
kafka-topics --bootstrap-server kafka1:9092 --create --topic exampleTopic

# Output:
Created topic exampleTopic.

# To exit the container, please specify exit

As we can see, the exampleTopic was created successfully.

So as the next thing, let’s produce some messages from kafka2 broker:

docker exec -it kafka2 /bin/bash
kafka-console-producer --bootstrap-server kafka2:9092 --topic exampleTopic

>codersee
>rules 

# To exit the producer, please hit Ctrl + D. Then to exit the container please type 'exit'.

No errors have been thrown, which let us assume that everything is good so far.

Nevertheless, to fully make sure, let’s consume these messages:

docker exec -it kafka3 /bin/bash
kafka-console-consumer --bootstrap-server kafka3:9092 --topic exampleTopic --from-beginning

# Output:
codersee
rules

# To exit the consumer, please hit Ctrl + C. As the output, we should see:
^CProcessed a total of 2 messages

# And again, to exit the container please type 'exit'.

As we can see, everything worked as expected and our brokers are correctly communicating within the cluster.

8. Kafka Without Zookeeper Using Docker Compose Summary

And that’s all for today’s article on how to set up Kafka Without Zookeeper using Docker Compose.

I really hope that you enjoyed this and my previous articles in an Apache Kafka series and I will be forever thankful if you would like to share your feedback with me in the comments section below (positive and negative, as well).

Take care and have a great week! 🙂

The post How To Set Up Kafka Without Zookeeper using Docker Compose? appeared first on Codersee blog- Kotlin on the backend.

Hello, in this step-by-step guide, I will show you how to set up multiple Kafka brokers with Docker Compose 🙂

In this article, together we will:

• see the minimum Docker Compose config required to spin up 3 Kafka brokers,
• take a closer look at the configuration,
• get information about active brokers with the zookeeper-shell command,
• verify whether everything is working as expected with Console Producer and Consumer tools.

Unlike the previous article about Kafka, I would like to start this one by showing the configuration you’ll need to set up 3 Kafka brokers. After that, we will go through it as thoroughly as possible to understand it even better and in the end, we will learn how to verify and test our config.

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. Multiple Kafka Brokers Docker Compose

With that being said, let’s see the docker-compose.yaml:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
    ports:
      - "8097:8097"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8097,INTERNAL://kafka1:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
    ports:
      - "8098:8098"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8098,INTERNAL://kafka2:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3
    ports:
      - "8099:8099"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8099,INTERNAL://kafka3:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

[elementor-template id=”9007393″]

2.1. Docker Images Versions and Containers Names

Firstly, let’s take a look at docker images versions and containers names:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
...
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
...
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
...
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3

As we can see, in our example we will use Confluent Community Docker Image for Apache Kafka and Confluent Docker Image for Zookeeper– both in version 7.2.1.

Additionally, we explicitly set names for our containers, which are discoverable at a hostname identical to the container name. Simply put, container exposed port 9092 of kafka1 will be visible for other containers at kafka1:9092.

2.2. Containers Ports

Nextly, let’s have a look at the exposed ports:

...
  kafka1:
    ports:
      - "8097:8097"
...
  kafka2:
    ports:
      - "8098:8098"
...
  kafka3:
    ports:
      - "8099:8099"

As can be seen, we did it for each service except the zookeeper.

You might already know this, but ports is responsible for exposing container ports to the host machine. Simply put, when running docker-compose on our computer with a 123:456 setting, we will be able to access a container’s 456 port through localhost:123.

2.3. Docker Compose Depends On

The last thing, I would like to cover before we head to the environment settings of our Kafka brokers is the depends_on .

As we can see, we put it for each Kafka service specifying the zookeeper service as an argument. With this one, we simply make sure that when running a docker compose up command, the zookeeper service will be started before the brokers.

3. Multiple Kafka Brokers Environment Variables

If you’ve seen my previous article on how to set up a single-node cluster with docker-compose, then you will notice that it looks a bit different. The main reason behind this is that with multiple Kafka brokers in our node, they have to know how to communicate with each other.

3.1. Zookeeper Client Port

As the first step, let’s take a look at the ZOOKEEPER_CLIENT_PORT.

To put it simply, this one instructs Zookeeper where to listen for connections by clients- Kafka brokers in our example.

3.2. Kafka Broker ID

Following, check the KAFKA_BROKER_ID variable.

By default, when we don’t set this one, the Zookeeper will automatically assign a unique broker identifier. Nevertheless, if we would like to specify it explicitly, then we can take care of it with this environment variable. Of course, please keep in mind that this value has to be unique across our Kafka brokers.

3.3. Kafka Zookeeper Connect

As the next thing, we set the KAFKA_ZOOKEEPER_CONNECT in each service to point to the Zookeeper.

As the value, we have to set our Zookeeper container name with a port set with the ZOOKEEPER_CLIENT_PORT directive.

3.4. Kafka Listener Settings

Finally, let’s see the rest of the environment variables responsible for communication:

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
• KAFKA_ADVERTISED_LISTENERS
• KAFKA_INTER_BROKER_LISTENER_NAME

Unlike the previous ones, I believe it has more sense to take a look at all of them at once to get a better understanding.

With a KAFKA_ADVERTISED_LISTENERS, we set a comma-separated list of listeners with their host/IP and port. This value must be set in a multi-node environment because otherwise, clients will attempt to connect to the internal host address. Please keep in mind that the listeners’ values (EXTERNAL and INTERNAL in our case), must be unique here.

When it comes to the naming here, we can specify our own values, just as I did with EXTERNAL/INTERNAL. Nevertheless, in such a case, we have to specify the correct mapping with KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:

• EXTERNAL will be mapped to PLAINTEXT protocol,
• INTERNAL will be mapped the same way, as well

If you are wondering why have we done it, then the reason is mentioned above- listeners’ values have to be unique, so we can’t specify PLAINTEXT://localhost:8099,PLAINTEXT://kafka3:9092 .

Finally, with KAFKA_INTER_BROKER_LISTENER_NAME we explicitly specify which listener to use for inter-broker communication.

4. Multiple Kafka With Docker Compose Testing

With all of that being said, we can finally make use of our docker-compose.yaml file.

To do so, let’s build and run our services with the following command:

docker compose up

Depending on our machine, it may take a while until everything is up and running.

In the end, we should see the following:

...Recorded new controller, from now on will use broker...

4.1. Verify All Containers Running

As the next step, let’s open up a new terminal window and list up and running containers with docker ps:

# Output:
CONTAINER ID IMAGE                           COMMAND                CREATED        STATUS        PORTS                            NAMES
cf892aeb5793 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8098->8098/tcp, 9092/tcp kafka2
a10bef7b6a54 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8099->8099/tcp, 9092/tcp kafka3
46c9ca4862b6 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8097->8097/tcp, 9092/tcp kafka1
eef5ebec8519 confluentinc/cp-zookeeper:7.2.1 "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 2181/tcp, 2888/tcp, 3888/tcp     zookeeper

As we can clearly see, all Kafka containers have their ports exposed to our host and Zookeeper is reachable through 2181, as well.

4.2. Verify Active Kafka Brokers Using zookeeper-shell

Following, let’s run a zookeeper-shell on the zookeeper container:

docker exec -it zookeeper /bin/zookeeper-shell localhost:2181

# Output:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

What’s important to mention here is that we run a zookeeper-shell command from within the container. That’s the reason why the zookeeper_host value is set to localhost and accessible, even though we haven’t exposed it.

Thanks to the -it flags, we can use the shell from our terminal.

4.3. Get IDs of Currently Active Brokers

So, to get the identifiers of currently active Kafka Brokers, we can use the following:

ls /brokers/ids

# Output:
[1, 2, 3]

The above output clearly indicates that everything is working, as expected.

Note: to close the zookeeper-shell, please hit Ctrl + C (and please do it before heading to the Producer/Consumer parts)

5. Publish Messages With Console Producer

After we know all of the above, we can finally send messages using the Console Producer.

As the first step, let’s run bash from a randomly picked Kafka broker container:

docker exec -it kafka2 /bin/bash

Nextly, let’s create a new topic called randomTopic:

kafka-topics --bootstrap-server localhost:9092 --create --topic randomTopic

# Output:
Created topic randomTopic.

Please note, that given the kafka-topics script is run inside of the container, we point to the 9092 port of localhost.

Following, we can produce some messages to our new topic:

kafka-console-producer --bootstrap-server localhost:9092 --topic randomTopic 

> lorem
> ipsum

# When we finish, we have to hit ctrl + d

Of course, we could do that from any other broker container (and in order to exit bash, we need to type exit in the command prompt).

6. Read Messages With Console Consumer

As the last step, let’s connect to another Apache Kafka broker and read previously sent messages:

docker exec -it kafka3 /bin/bash

kafka-console-consumer --bootstrap-server localhost:9092 --topic randomTopic --from-beginning

# Output:
lorem
ipsum

As we can clearly see, the output contains all messages, we’ve sent using Console Producer.

7. Summary

And that would be all for this article on how to set up multiple Apache Kafka brokers using Docker Compose and produce/consume messages. As a follow-up, I would recommend checking out this Confluent article about connecting to Kafka with different configurations.

Finally, let me know your thoughts in the comment section bellow, dear friend! 😀

The post How To Deploy Multiple Kafka Brokers With Docker Compose? appeared first on Codersee blog- Kotlin on the backend.

If you would like to learn how to set up Apache Kafka using Docker Compose then you’ve come to the right place 🙂

In this article, together we will:

• see the bare minimum Docker Compose configuration we must provide to spin up an environment with exactly one Kafka broker and Zookeeper instance,
• take a closer look at the required configurations,
• verify our setup using Console Producer and Consumer tools.

If you will enjoy this tutorial, then I highly encourage you to check out the continuations:

• How To Deploy Multiple Kafka Brokers With Docker Compose?
• How To Set Up Kafka Without Zookeeper using Docker Compose

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. What is Apache Kafka?

Before we start, I feel obliged to provide at least a brief explanation of what Apache Kafka and Zookeeper are. Please skip the following two paragraphs if you want to get straight to the practice part of this tutorial.

Well, Apache Kafka by definition is an open-source distributed event streaming platform. Simply put, it’s a platform widely used to work with real-time streaming data pipelines and to integrate applications to work with such streams. It’s mainly responsible for:

• publishing/subscribing to the stream of records,
• their real-time processing,
• and their ordered storage

But let’s just be honest- it’s almost impossible to describe Kafka in a couple of sentences and I highly encourage you to visit the Apache Intro page after this tutorial right here.

3. What is Apache ZooKeeper?

Or the question should rather be why do I even need another service to work with Apache Kafka?

Just like a zookeeper in a zoo takes care of the animals, Apache Zookeeper “takes care” of all the Kafka brokers in a cluster. It’s mainly responsible for:

• tracking the cluster state- like which brokers are a part of the cluster, sending notifications to Kafka in case of changes
• topics configuration
• controller election
• access control lists and quotas

Nevertheless, please keep in mind that the Kafka team is constantly working on replacing the ZooKeeper with Apache Kafka Raft (KRaft). To be more specific- the 2.8.0 version introduced early access to this functionality, but at the moment of publishing this article, it’s still not production-ready.

4. Bare Minimum Kafka Docker Compose

With all of that being said, let’s have a look at the bare minimum version of the docker-compose.yml file. This configuration is necessary to spin up exactly 1 Kafka and Zookeeper instance:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka
    ports:
      - "8098:8098"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:8098
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Again, before we head to the next commands, let’s take a couple of minutes to understand, what exactly is going on here.

4.1. Docker Images Version And Containers

Firstly, let’s take a look at this piece of code:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
...   
  kafka:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka

As we can see, with this config we will spin up two services named kafka and zookeeper. The container_name will set specified names for our containers, instead of letting Docker generate them. Finally, we will use the 7.2.1 versions of Confluent Community Docker Image for Apache Kafka and Confluent Docker Image for Zookeeper.

4.2. Additional Kafka Container Configs

As the next step, let’s have a quick look at these four lines:

ports:
  - "8098:8098"
depends_on:
  - zookeeper

As the name suggests, the ports command is responsible for exposing ports, so with this setting, port 8098 of our container will be accessible through the 8098 port of the host machine. Additionally, the depends_on will make sure to start the zookeeper container before the kafka.

5. Required Settings

As the next step, let’s take a closer look at the required settings for both Kafka and Zookeeper in our Docker Compose settings.

5.1. Required Zookeeper Settings

As we can see in our example, the only environment we specified is ZOOKEEPER_CLIENT_PORT. This one instructs Zookeeper where it should listen for connections by clients- Kafka in our case.

Additionally, when running in clustered mode, we have to set the ZOOKEEPER_SERVER_ID (but it’s not the case here).

5.2. Required Confluent Kafka Settings

When it comes to the Confluent Kafka Docker image, here is the list of the required environment variables:

• KAFKA_ZOOKEEPER_CONNECT – instructing Kafka where it can find the Zookeeper
• KAFKA_ADVERTISED_LISTENERS – specifying the advertised hostname, which clients can reach out to. Moreover, this value is sent to the Zookeeper

Please keep in mind, that in order to run only one instance of Kafka (aka single-node cluster), we have to additionally specify KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR set to 1. Its default value is 3 and if we don’t do that, we will get into this error: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1

6. Verify Kafka Docker Compose Config

With all of that being said, we can finally check out if everything is working, as expected.

6.1. Start Containers

As the first step, we have to start our containers:

docker compose up -d

The -d flag instructs the docker to run containers in a detached mode.

Of course, we can verify everything by running the docker ps command:

# example output:
CONTAINER ID IMAGE                           COMMAND                CREATED        STATUS       PORTS                            NAMES
38b41bc43676 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 42 seconds ago Up 4 seconds 0.0.0.0:8098->8098/tcp, 9092/tcp kafka
45484184f330 confluentinc/cp-zookeeper:7.2.1 "/etc/confluent/dock…" 42 seconds ago Up 5 seconds 2181/tcp, 2888/tcp, 3888/tcp     zookeeper

6.2. Create Kafka Topic

Nextly, let’s create a new Kafka topic:

docker exec kafka kafka-topics --bootstrap-server localhost:8098 --create --topic example

As a word of explanation, the kafka-topics is a shell script that allows us to execute a TopicCommand tool. It’s a command-line tool that can manage and list topics in a Kafka cluster. Additionally, we have to specify the listener hostname (specified with KAFKA_ADVERTISED_LISTENERS) with –bootstrap-server.

When the topic is created we should see the following message: “Created topic example.”

6.3. Run Kafka Producer

Following, let’s run a console producer with the kafka-console-producer:

docker exec --interactive --tty kafka kafka-console-producer --bootstrap-server localhost:8098 --topic example

Please keep in mind that this command will start the producer and it will wait for our input (and you should notice the > sign). Please specify a couple of messages and hit Ctrl + D after you finish:

>lorem
>ipsum
# hit Ctrl + D

6.4. Run Kafka Consumer

As the last step, let’s run a console consumer with the kafka-console-consumer command:

docker exec --interactive --tty kafka kafka-console-consumer --bootstrap-server localhost:8098 --topic example --from-beginning

This time, we should see that our messages are printed to the output successfully (and to finish please hit Ctrl+ C):

lorem
ipsum

7. Kafka With Docker Compose Summary

And that would be all for this step-by-step guide on How To Set Up Apache Kafka With Docker Compose.

Let me know about your thoughts in the comment section, or reach out through the contact form- I highly appreciate your thoughts, comments and feedback.

Take care brother/sister!

The post How To Set Up Apache Kafka With Docker? appeared first on Codersee blog- Kotlin on the backend.