• what problems do they solve?
• what is an ETag and how is it calculated?
• how do they relate to If-None-Match and If-Match headers?
• what do 412 Precondition Failed and 304 Not Modified got do to with all of that?

Of course, this is more theoretical content, and you can expect some pragmatic examples with Spring Boot and Ktor soon😉

Video Content

As always, if you prefer a video content, then please check out my latest YouTube video:

If you find this content useful, please leave a subscription 🙂

What Problems ETags Solve?

Let’s start everything by figuring out the most important thing- what actual problems Etags help us solve? At the end of the day, what is the point of any additional work if it does not bring any value?😉

Well, long story short, the Entity Tags help us to deal with two problems:

• waste of bandwidth and resources to consume the response of unchanged data,
• updating data that changed since we last fetched them.

Problem #1- Waste of Bandwidth & Resources

So what do we exactly mean by problem #1- the waste of bandwidth? Let’s imagine a pretty standard client <-> server communication. The client asks the server for some data, like product details (also known as entity, or resource in terms of REST). The server fetches necessary data from the database, sometimes performs additional actions, and eventually returns them to the client (let’s say a mobile application) with 200 OK status code.

Then, our client consumes that response- for example, converts the returned JSON and displays that to the user.

And up until now, everything is perfectly fine. But, let’s imagine a situation, in which our product remains untouched- no price, no title, nothing changes- but our client keeps querying the server. In such a case, the bandwidth used to transfer the exact same data over and over, and the client resources used to process that could be utilized better, right?

And as you might have guessed- that’s where ETags come with help🙂

What is an ETag and How It is Calculated?

Long story short- ETag (aka Entity Tag)- uniquely identifies a version of the entity.

In other words, it is a bunch of characters (like 0633010ca130d326f83ef9fee25491e7e) that remain constant for a particular version of an entity. If the entity (product in our example) is updated, a new tag is generated and remains constant until the next update.

From the implementational perspective, this can be anything from a version field in the database, a hashed timestamp of an update, or a hash of the response. Oftentimes, this hash is obtained using the MD5 algorithm.

Solve Problem #1 With ETags

Alright, so how we can use this identifier/tag to fix the waste in our client <-> server scenario? Well, pretty easily.

From the server perspective, whenever clients ask us for the data, we fetch them, calculate the ETag, and return 200 OK response with an additional ETag header. For example: ETag: "0e927523c29b6de3a9e3653df0e11762d" .

The client, on the other hand, is responsible for persisting ETags locally and later, sending them as an additional If-None-Match header, example: If-None-Match: "0e927523c29b6de3a9e3653df0e11762d" .

And whenever our server detects the If-None-Match , it fetches the data from the database, calculates the hash, and compares them. If they match, it means the resource has not changed since the last request. And in such a case, the server will send an empty response body with 304 Not Modified HTTP status code. If they don’t match, then the request body is sent along with 200 OK status code (and another ETag value).

As we can see, this allows us to fix the issue of wasted bandwidth, and potentially, the client resources. However, this does not prevent the client from making requests to the server, and the server from computing the data. For that purpose, we should lean towards client caching and proper cache control.

Problem #2- Update Changed Entity

At this point, we know both the issue and solution for repeated queries. So in this paragraph, let’s take a closer look into issue #2.

This time, let’s imagine we have 2 users that simultaneously fetched the same version of the entity- product in our case. Now, user 1 decides to update the product title. He hits update, his application sends a PUT request and the update succeeds. As we are using HTTP, user 2 (client) is not aware of the update. The application still displays old data. So, if he decides now to update some other field, like description, the PUT request may override the title with old data!

How ETags Can Help With Collisions?

In order to avoid this situation, we must make a small change in our API.

So, just like previously, when both clients make their 1st request, they get 200 OK response with an additional ETag header. For example: ETag: "0e927523c29b6de3a9e3653df0e11762d" .

Now, whenever any client application wants to update the product, it sends a PUT request with an additional If-Match header, like: If-Match: "0e927523c29b6de3a9e3653df0e11762d" . And this time, the server calculates the ETag of the desired entity before making any changes. If it does not match, it means that the entity was changed in the meantime, and the server notifies our client with 412 Precondition Failed status code.

Thanks to that, the client application has the chance to inform users about the updated entity/resource and handle that appropriately (for example, by displaying the popup).

This time, we can clearly see that the solution is not about performance and resources- it is about consistency.

Summary

And basically that is all for this article about ETags.

I hope it helped you to understand them better and provide you with ideas on how to apply them in your project.

Let me know your thoughts in the comments section!🙂

The post What are HTTP ETags? appeared first on Codersee blog- Kotlin on the backend.

What does it look like? What if I already joined a course purchased through the Teachable platform? What is coming next?

Let’s figure out answers to those questions now 😉

What if I already joined a course purchased through the Teachable platform?

Well, the answer is the most popular answer in the IT world- “it depends” 😉

If you purchased “The Complete Kotlin Course” or “Kotlin Handbook”, then the only thing you need to do today is to navigate to the login page and click the “Lost your password?” link:

After that, you will be redirected to the “My Account” page and prompted for your email:

Please specify the e-mail you used when purchasing the course, and you will receive a reset link to your inbox. After you reset your password and successfully sign in, you will be automatically enrolled in the course on the platform.

If you encounter any issues during this process, or if you simply have any questions, then please do not hesitate and let me know to contact@codersee.com

How Does It Look Like?

One of the most important points of the migration was to improve the User Experience. And I really hope I managed to do so 😉

Just like above, if we already have our account, we can navigate to the login page using the button in the header:

When we click on it, we will see the login form and when signed in successfully, we will reach the “My Learning” page:

On this page, we will see all courses we are enrolled in with the progress bar and a link to the last step in the course.

Let’s click the “Go to course” button then:

As we can see we are redirected to the focus mode of our course. This is a great way to avoid any disruptions and focus on the learning process.

However, if we would like to get back, then the only thing we need to do is to click the logo in the left upper corner.

Whenever we are not in the focus mode, we can navigate to the “My Learning” page using the option in the header, or with the dropdown.

Additionally, we can see the link to Account Details:

In here, we can find all the details related to our account, like orders, downloads (products), billing addresses, as well as the password reset form and logout.

And basically that is all for this introduction, stay tuned, because great news are coming in the beginning of 2025 🙂

The post Codersee Course Platform is Ready appeared first on Codersee blog- Kotlin on the backend.

At the end of this tutorial, you will know precisely:

• what are Git client-side Hooks,
• what types of client-side hooks Git offers,
• how to implement an automated code check before we commit our changes,
• and how to combine that together with Kotlin and Gradle.

And although this guide may seem a bit specific, you can trust me that after reading it you will be able to adjust this knowledge to many other cases in your real-life scenarios.

Video Tutorial

If you prefer video content, then check out my video:

If you find this content useful, please leave a subscription  😉

Git Hooks And Client-Side Types

Before we see how to do a Ktlint check as a pre-commit hook, let’s learn a bit about Git Hooks.

Git Hooks, in simple words, are a way to trigger custom scripts, whenever a particular action happens in our repository. An action can be a rebase, a checkout, a merge, etc.

And in order to run our script, the only thing we must do is to put it inside the hooks directory, which we can find in the .git directory.

Moreover, when we navigate there, we will see that it already contains a bunch of useful examples:

If we would like to try any of the predefined samples, then we must simply remove the .sample suffix from the filename.

And what client-side Git Hooks can we work with? Well:

• pre-commit – run before creating a commit,
• prepare-commit-msg – run before the commit message editor is opened,
• commit-msg – triggered after the commit message is created but before the commit is finalized,
• post-commit – run after a commit is made,
• applypatch-msg – invoked during the git apply operation to edit patch messages,
• pre-applypatch / post-applypatch – run before/after applying changes from a patch,
• pre-rebase – run before starting a rebase operation,
• post-rewrite – triggered after commands that rewrite commit history,
• post-checkout – invoked after a successful git checkout operation,
• post-merge – run after a successful git merge operation,
• pre-push – run before a push to a remote repository.

The pre-receive, update, and post-receive are server-side hooks, which we won’t cover here.

Import Ktlint

Excellent!

At this point, we know what we’re dealing with today and we can start the practice part of the Ktlint pre-commit hook implementation.

As the first thing, let’s navigate to the build.gradle.kts and import the library:

plugins {
    kotlin("jvm") version "1.9.10"
    application

    id("org.jlleitschuh.gradle.ktlint") version "11.6.1"
}

At the moment of writing, the most recent version is 11.6.1 and you can always figure out what’s the current one here.

Following, let’s sync the project.

When the synchronization process finishes, we should see plenty of new tasks added, among which, these we will use the most:

• ktlintFormat– to format according to the code style all SourceSets Kotlin files and project Kotlin script files,
• ktlintCheck– to check all SourceSets and project Kotlin script files

Add Script

With that done, let’s create the scripts directory and put the pre-commit file there:

#!/bin/bash

echo "Running git pre-commit hook"

./gradlew ktlintCheck

ktlintCheckStatus=$?

# return 1 if check fails
if [[ $ktlintCheckStatus -ne 0 ]]; then
     exit 1
else
     exit 0
fi

But why don’t we put that directly to the .git/hooks directory?

Well, the problem is that by default, the .git directory is not tracked. It contains all the information about the repository, including the repository’s configuration, commit history, branches, and other metadata. And if the .git directory were tracked, it would create a kind of recursive loop, leading to potential issues and conflicts.

So, when working with Gradle and Kotlin, we will simply put our script in the scripts destination, and later configure Gradle to copy it to the desired folder.

And what is our script?

Well, it is a simple script, which will run the ktlint check command using the gradle wrapper. If it is successful, the command returns 0 and we return 0, too. In other case, we return 1 and the commit will simply fail.

Update build.gradle.kts

Following, let’s navigate to the build.gradle.kts file and add a new task- the copyPreCommitHook:

tasks.register<Copy>("copyPreCommitHook") {
    description = "Copy pre-commit git hook from the scripts to the .git/hooks folder."
    group = "git hooks"
    outputs.upToDateWhen { false }
    from("$rootDir/scripts/pre-commit")
    into("$rootDir/.git/hooks/")
}

As we can see, this task is responsible for copying the pre-commit file from the scripts/pre-commit to the .git/hooks/.

Moreover, we make a small workaround- outputs.upToDateWhen { false } – so that our task will never be cached.

When we sync our Gradle project, we should see that our task is available from now on inside the git hooks group:

At this point, we can run the task and verify that the script was moved successfully.

But is that all?

Well, we could stop right here, but if we automate things, it would be good to avoid manual run of the copyPreCommitHook task, right? And we can easily achieve that this way:

tasks.build {
    dependsOn("copyPreCommitHook")
}

With this setting, we simply instruct Gradle to run copyPreCommitHook before the build task.

Verification

With all of that done, we can finally verify if our ktlint check will be run as a pre-commit hook.

To do so, let’s make some changes that will fail ktlintCheck and try to commit. In IntelliJ, we should see the following:

And when we check the details, we will see a meaningful message:

Running git pre-commit hook
Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could not be reused, use --status for details
> Task :loadKtlintReporters UP-TO-DATE
> Task :runKtlintCheckOverKotlinScripts UP-TO-DATE
> Task :runKtlintCheckOverTestSourceSet NO-SOURCE
> Task :ktlintTestSourceSetCheck SKIPPED
> Task :ktlintKotlinScriptCheck
> Task :runKtlintCheckOverMainSourceSet
> Task :ktlintMainSourceSetCheck FAILED
C:\Users\Piotr\src\main\kotlin\Main.kt:2:1 Unexpected indentation (10) (should be 4)
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':ktlintMainSourceSetCheck'.
> A failure occurred while executing org.jlleitschuh.gradle.ktlint.worker.ConsoleReportWorkAction
   > KtLint found code style violations. Please see the following reports:
     - C:\Users\Piotr\build\reports\ktlint\ktlintMainSourceSetCheck\ktlintMainSourceSetCheck.txt
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 7s
5 actionable tasks: 3 executed, 2 up-to-date

And this snippet proves that our pre-commit hook was triggered and works as expected.

Moreover, when we correct our code, we will see that we can commit without any message!

Summary

And that’s all for this article about implementing ktlint check as a pre-commit Git Hook.

I hope you enjoyed this tutorial and that this will be a great start for adding more automation to your project. If you would like to get a ready-to-go project, then please navigate to my GitHub repo here.

Lastly, I would like to invite you to my free newsletter, so that you will never miss any important updates from both my blog and Kotlin world!

The post Ktlint with pre-commit Hook: Git Hooks in Kotlin Made Easy appeared first on Codersee blog- Kotlin on the backend.

But before we start, just a quick reminder that this is my second post in the series about GitHub Actions workflow. And if you would like to see the introduction, then check out my previous article and learn how to create your first workflow.

Video Tutorial

If you prefer video content, then check out my video:

If you find this content useful, please leave a subscription  😉

Trigger Workflow On workflow_run

Let’s assume that we would like to add a new workflow to our GitHub Actions, workflow B, which will trigger whenever workflow A completes successfully. We go to the events documentation and figure out that the workflow_run will be the right choice.

So, for the following workflow A:

name: Workflow A

on:
  push:
    branches: [ "main" ]

jobs:
  some_job:  
    runs-on: ubuntu-latest
    steps:
      - name: Some Step 
        run: echo "Hello!"

We pretty quickly came up with something, like this:

name: Workflow B

on:
  workflow_run:
    workflows: ["Workflow A"]
    types:
      - completed

jobs:  
  run_on_workflow_a_success:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    steps:
      - name: Run on workflow A success
        run: echo "Workflow A completes successfully"

Whenever the workflow A completes successfully, we simply print out a message to the output.

Problem With Passing Data

But life is not usually that simple and we pretty quickly realize that we need to find a way to pass the data between our GitHub workflows.

If we learned from my previous article or spent some time with the official documentation, we finally land on the contexts page and try to find the necessary information.

If are lucky enough and the data we need are present, we can simply access them with one line of code.

Just like we did previously:

github.event.workflow_run.conclusion

As we can see, the above line accesses information about conclusion from the github context.

But what if we would like to access some custom data produced by the workflow A within the workflow B?

Well- that’s the point where the story begins 🙂

Pass Data Between GitHub Workflows With Artifacts

Unfortunately, at the moment when I am writing this article, there’s no such thing as a custom context or something similar where we could put the data and access them inside another (dependent) workflow.

Fortunately, we can bypass this limitation by producing artifacts.

Let’s update our workflow A first:

name: Workflow A

on:
  push:
    branches: [ "main" ]

jobs:
  some_job:  
    runs-on: ubuntu-latest
    steps:
      - name: Some Step 
        run: echo "Hello!"

  job_producing_data:  
    runs-on: ubuntu-latest
    steps:
      - name: Some Step 
        run: |
            printf '{ 
              "prop_1": "Some value",  
              "prop_2": true
            }' >> context.json
      - uses: actions/upload-artifact@v3
        with:
          name: context.json
          path: ./

With this solution, we produce a JSON file called context.json, which then we publish as an artifact.

In simple words, GitHub artifacts are files (or collections of files), that we can produce during the workflow run and access later- from other workflows, through the API, or simply in the actions tab:

Following, let’s update the workflow B to consume this file and access its data:

name: Workflow B

on:
  workflow_run:
    workflows: ["Workflow A"]
    types:
      - completed

jobs:  
  run_on_workflow_a_success:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    steps:
      - name: Run on workflow A success
        run: echo "Workflow A completes successfully"

  download_context_artifact:
    runs-on: ubuntu-latest
    steps:
      - name: 'Download artifact'
        uses: actions/github-script@v6
        with:
          script: |
            let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
               owner: context.repo.owner,
               repo: context.repo.repo,
               run_id: context.payload.workflow_run.id,
            });
            
            let matchArtifact = allArtifacts.data.artifacts.filter((artifact) => {
              return artifact.name == "context.json"
            })[0];
            
            let download = await github.rest.actions.downloadArtifact({
               owner: context.repo.owner,
               repo: context.repo.repo,
               artifact_id: matchArtifact.id,
               archive_format: 'zip',
            });
            
            let fs = require('fs');
            fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/context.zip`, Buffer.from(download.data));
            
      - name: 'Unzip artifact'
        run: unzip context.zip

      - name: 'Return Parsed JSON'
        uses: actions/github-script@v6
        id: return-parsed-json
        with:
          script: |
            let fs = require('fs');
            let data = fs.readFileSync('./context.json');
            return JSON.parse(data);

    outputs:
      property_one: ${{fromJSON(steps.return-parsed-json.outputs.result).prop_1}}
      property_two: ${{fromJSON(steps.return-parsed-json.outputs.result).prop_2}}

  log_context_values:
    needs:
      - download_context_artifact
    runs-on: ubuntu-latest
    steps:
      - name: 'Log Context Values'
        run: |
          echo "${{ needs.download_context_artifact.outputs.property_one }}"
          echo "${{ needs.download_context_artifact.outputs.property_two }}"

As we can see, we added two new jobs: download_context_artifact and log_context_values.

The first one is responsible for downloading the artifact (unfortunately GitHub API supports only zip files at the moment). We list all articles from the triggering flow (workflow A), match the one named context.json and download it.

Then, we unzip the file and parse its JSON content.

As the last thing in the first job, we produce two outputs. The output is nothing else than a map accessible in all downstream jobs dependent on this job. And to put it simply, in all jobs which point to this one using needs.

The interesting thing here is that in order to access the parsed json- return JSON.parse(data)– we need to use the <step id>.outputs.result.

The second job, log_context_values, simply prints out our values.

Summary

And that’s all for this tutorial, in which we learned how to pass data between workflows in GitHub actions.

I hope you enjoyed this one and if you would like to share your feedback with me or ask about anything, then let me know in the comments section.

The post How To Pass Data Between Workflows in GitHub Actions? appeared first on Codersee blog- Kotlin on the backend.

At the end of this tutorial, you will know precisely how to:

• navigate in GitHub actions and create new workflows,
• trigger workflows manually,
• create a job that runs only when another job succeeds,
• use GitHub context and access its data.

Video Tutorial

If you prefer video content, then check out my video:

If you find this content useful, please leave a subscription  😉

What Is GitHub Actions?

GitHub Actions is nothing else than a CI/CD (continuous integration/continuous delivery) platform.

To put it simply, it is a platform, that allows us to automate builds, testing, deployments, and many other things related to the process in your company, or the project.

In practice, we define so-called workflows, which are triggered whenever some event happens in our repository. An event can be a merge to the main, a push to the remote branch, a deployment update, and plenty more.

Each workflow consists of jobs running inside their own machine runner and each job can have multiple steps responsible for performing an actual action, like code check, running a test, or creating a deployment.

Lastly, each workflow is nothing else than a YAML file placed within the .github/workflows directory.

Create First Workflow

With all of that said, let’s navigate to our GitHub repository and select the actions tab. Alternatively, you can do that with the following link: https://github.com/codersee-blog/{YOUR_REPO_NAME}/actions/new.

Nextly, let’s click the configure button:

As a result, we should see the editor with the following code:

# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the "main" branch
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3

      # Runs a single command using the runners shell
      - name: Run a one-line script
        run: echo Hello, world!

      # Runs a set of commands using the runners shell
      - name: Run a multi-line script
        run: |
          echo Add other actions to build,
          echo test, and deploy your project.

So, let’s clean it up a bit to avoid distraction:

name: CI

on:
  push:
    branches: [ "main" ]
  
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Run a one-line script
        run: echo Hello, world!

      - name: Run a multi-line script
        run: |
          echo Add other actions to build,
          echo test, and deploy your project.

To sum up, we’ve defined a new GitHub Actions workflow, named CI, which will be triggered whenever we push a new commit to the main branch.

Our workflow consists of one job, build, which will be run on the Ubuntu runner. This job will invoke has two steps, which will be run one after another:

• firstly, it will print out a one-line script,
• lastly, it will invoke the multiline script.

Quite easy, isn’t it?

So with all of that done, let’s change the name of the file to ci.yml and let’s commit our changes to the main branch:

How To View Workflow Run?

With all of that done, let’s see how we can see the result of our triggered workflow.

To do so, let’s navigate to the following URL: https://github.com/codersee-blog/{your-repo-name}/actions/workflows/ci.yml.

On this page, we should see that our workflow run has completed successfully:

As the next step, let’s click on the build job and let’s see the output:

Excellent!

Both our scripts run successfully and we can see the expected output.

How To Trigger Flow Manually?

At this point, you might be wondering if (and if yes, then how) we can manually trigger the flow.

Well, unfortunately, to do so, we must edit our YAML file a little:

name: CI

on:
  push:
    branches: [ "main" ]
  workflow_dispatch:  
  
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Run a one-line script
        run: echo Hello, world!

      - name: Run a multi-line script
        run: |
          echo Add other actions to build,
          echo test, and deploy your project.

As we can see, the only thing we need to add is the workflow_dispatch.

This event, allows us to trigger our workflow using the GitHub API, GitHub CLI, or GitHub browser interface:

How To Disable The Workflow?

But can we disable the workflow without deleting the file?

Yes, we can.

We can do that simply by using the Disable workflow button from the workflow runs page:

Contexts And Debugging

Wonderful. At this point, we already know the basics of workflows and navigation in GitHub Actions.

As the next step, let’s learn a bit more about contexts.

Contexts allow us to access information about the workflow runs, variables, runner environments, jobs, and steps. If you are looking for commit sha, PR author, branch name, or maybe GitHub access token, then that’s the place you should check out.

For the full list of contexts with descriptions please take a look at the official documentation.

But from my end, I wanted to show you a useful tip if you would like to debug anything:

name: CI

on:
  push:
    branches: [ "main" ]
  workflow_dispatch:  
  
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Run a one-line script
        run: echo Hello, world!

      - name: Run a multi-line script
        run: |
          echo Add other actions to build,
          echo test, and deploy your project.
  
      - name: Who initialized the flow? 
        run: echo "${{ github.actor }} initialized the flow."

      - name: Debug GitHub context
        run: |
            echo " ${{ toJSON(github) }}"

As we can see, the third step – Who initialized the flow?- prints out the “actor” (or simply the account) that initialized the flow.

The fourth step- Debug GitHub context- prints out github context to the output.

The toJson function returns a pretty-print JSON representation of our value.

As a result, we should see the following:

Whatsoever, when debugging we don’t need to worry about exposing our secrets.

GitHub hides them by default.

Run Job After Another Job Finishes Successfully

Excellent! At this point, we learned quite a lot about the GitHub Actions workflows.

We know how to declare them, how to debug contexts, and how to navigate inside the GitHub world.

As the last thing for our first meeting, let’s learn how we can make one job dependent on the other.

To do so, let’s edit our YAML file once again:

name: CI

on:
  push:
    branches: [ "main" ]
  workflow_dispatch:  
  
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Run a one-line script
        run: echo Hello, world!

      - name: Run a multi-line script
        run: |
          echo Add other actions to build,
          echo test, and deploy your project.
  
  second_job:        
    needs: build
    runs-on: ubuntu-latest
    
    steps:
      - name: I run only when build succeeds
        run: echo Hi!

As we can see, the only thing we need here is needs: build.

To be more specific, the needs instruction allows us to specify jobs that must be completed successfully before our job starts.

Of course, we can define multiple jobs using the array syntax: needs: [job1, job2].

Summary

And that’s all for this quick tutorial on how to create GitHub Actions workflows.

If you are interested in learning more about this topic, then I highly encourage you to check out their documentation. I’m pretty sure you will find there anything you need.

But if that’s not the case, then let me know in the comments section or by using the contact form 🙂

The post How To Create a GitHub Actions Workflow appeared first on Codersee blog- Kotlin on the backend.

If you follow my blog a bit longer, then you probably realized that you can find here a bunch of various how-to articles and tutorials, mostly related to Kotlin and frameworks, or libraries that we can use with it. And although I am happy to write about any topic, at the end of the day, it is you, dear reader, who consumes this content.

Ebook For A Survey

So, without any further ado, I’d like to ask you to take part in the really short survey (2-3 minutes), which will help me to provide you with better-suited content so that you will become a better programmer, and my work will actually make sense.

It’s a classic win-win situation, and furthermore, in return for your precious time, you’ll receive an ebook containing 50 Spring Framework interview questions.

Please find the survey below (you’ll receive the link to the ebook immediately after you submit the answers):

Once again, thank you for your time, and see you in the next posts friend.

The post How Can Codersee Help You? appeared first on Codersee blog- Kotlin on the backend.

In this last of the 3 articles about setting up Apache Kafka, I would like to show you how to set up Apache Kafka Without Zookeeper using Docker Compose.

After reading it, you will know precisely:

• what does the Docker Compose configuration look like to set up 3 Kafka Brokers,
• what environment variables we have to set,
• and what workaround we have to do in order to make it run.

If after finishing it, you will still be eager to broaden your knowledge, then you can find the previous two articles right here:

• How To Deploy Multiple Kafka Brokers With Docker Compose?
• How To Set Up Apache Kafka With Docker?

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. Apache Kafka Without Zookeeper

Before we dive into the practice part, let’s take a second to understand the motivation behind the Zookeeper removal (I’ve already written about Zookeeper responsibilities in the first part, so I’ll skip it here).

To put it simply, the motivation behind it can be summarized with these points:

• simplifying the deployment and configuration process (Zookeeper is a separate system, with its own syntax, configs, and deployment patterns),
• scalability improvement,
• enabling support for more partitions.

The whole change has been introduced for the first time as a KIP-500 (Kafka Improvement Proposal) and has been offered as an early access version with the 2.8.0 release.

If you would like to understand deeply the reasons and arguments behind the Zookeeper removal, I highly encourage you to visit the KIP-500 ticket page linked above.

[elementor-template id=”9007393″]

3. Kafka Without Zookeeper Docker Compose File

With that being said, let’s take a look at the docker-compose.yml file:

version: '3'
services:
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka1:9092,CONTROLLER://kafka1:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9092,CONTROLLER://kafka2:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://kafka3:9092,CONTROLLER://kafka3:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:9093,2@kafka2:9093,3@kafka3:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'
    volumes:
      - ./run_workaround.sh:/tmp/run_workaround.sh
    command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"

Don’t worry if you feel overwhelmed at this point, I will walk you step by step through the configuration values in the following paragraphs.

Nevertheless, don’t run the docker compose up command yet, we will need one more file to run it 🙂

4. Add Necessary Workarounds

As the next step, we need to run a few commands when containers start.

In my example, I will call this file run_workaround.sh. And although the name is not important, it has to match values specified when mounting volumes and running the command from within the containers:

#!/bin/sh

sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure

sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure

echo "kafka-storage format --ignore-formatted -t NqnEdODVKkiLTfJvqd1uqQ== -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure

As we can see, the above script contains exactly 3 commands.

The first one is responsible for turning off the check for the KAFKA_ZOOKEEPER_CONNECT parameter. Without this line, the error will be thrown: Command [/usr/local/bin/dub ensure KAFKA_ZOOKEEPER_CONNECT] FAILED !

The second one ignores the cub zk-ready. Without it, everything will end up with another exception: zk-ready: error: too few arguments.

And finally, the last line is a KRaft required step, which formats the storage directory with a new cluster identifier. Please keep in mind that the specified UUID value (NqnEdODVKkiLTfJvqd1uqQ== in my case) has to be 16 bytes of a base64-encoded UUID.

5. Docker Compose Instructions

With all of the above being said, we can get back to the docker-compose.yml file.

5.1. Volumes and Commands

Although I will skip the things like the services, images, or container names (I’ve covered them in the previous articles), let’s take a look at these 3 lines:

volumes:
  - ./run_workaround.sh:/tmp/run_workaround.sh
command: "bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'"

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. With the above setting, we inform Docker that we want the run_workaround.sh file to be mounted inside the tmp directory inside the container with the same name. Of course, we could change the name of this file inside, but then, the change should also be reflected in the command.

When it comes to the command, it simply overrides the default command. With the above config, we instruct Docker to run our container with our custom script.

6. Apache Kafka Brokers Environment Variables

Finally, let’s go through all the environments we had to set in order to make everything work.

6.1. KAFKA_NODE_ID Environment Variable

As the name suggest, the KAFKA_NODE_ID variable is used to specify a unique identifier of our node in the cluster.

6.2. Kafka Listeners and Protocol Mapping

Nextly, let’s see the settings responsible for appropriate hosts and ports mapping:

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP – with this one, we simply define the key-value pairs of listeners’ names with security protocols used with them
• KAFKA_LISTENERS – with this variable, we define all exposed listeners
• KAFKA_ADVERTISED_LISTENERS – this one, on the other hand, contains all listeners used by clients

It’s worth mentioning here that when working with Docker Compose, the container name becomes a hostname- like kafka1, kafka2, and kafka3 in our examples.

6.3. KAFKA_ADVERTISED_LISTENERS Setting

As the next step, let’s see the KAFKA_ADVERTISED_LISTENER. This one, simply allows us to specify a list of coma-separated listeners’ names used by the controller. It’s necessary when running in a KRaft mode.

Additionally, none of the values specified here can be put inside the advertised listeners covered in the previous paragraph. Otherwise, the following error will be thrown:

The advertised.listeners config must not contain KRaft controller listeners from controller.listener.names when process.roles contains the broker role because Kafka clients that send requests via advertised listeners do not send requests to KRaft controllers — they only send requests to KRaft brokers.

6.4. KAFKA_CONTROLLER_QUORUM_VOTERS

As the next one comes the KAFKA_CONTROLLER_QUORUM_VOTERS environment variable. And this one allows us to specify the set of voters in our node. When it comes to the format, values have to be put in the following order: {id}@{host}:{port} .

So, in our case, we do specify 9093 of our nodes, because this is the port exposed as a controller.

6.5. KAFKA_PROCESS_ROLES Variable

And the last one- the KAFKA_PROCESS_ROLES variable lets us specify the role of the particular broker. The value can be set to either broker, controller, or both of them (just like above).

7. Testing

And finally, we can proceed to the part where we will verify if everything is working, as expected. In my example, I will show you how to use Console Producer/Consumer, but feel free to connect with anything you like.

Firstly, let’s start our services with the docker compose up:

The broker has been unfenced. Transitioning from RECOVERY to RUNNING.

After some time, we should see the above for each of our containers.

Additionally, we can validate with a docker ps command:

CONTAINER ID IMAGE                       COMMAND                CREATED            STATUS            PORTS    NAMES
dc3966ba58a4 confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka3
10242d9d1e6a confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka2
c404c866cde6 confluentinc/cp-kafka:7.2.1 "bash -c '/tmp/run_w…" About a minute ago Up About a minute 9092/tcp kafka1

As we can clearly see, every container is up and running. Moreover, the desired 9092 port is exposed for each of them.

When it comes to the testing strategy, I would like to connect to each broker separately and perform a different action:

• firstly, connect to the kafka1 and create a new topic called exampleTopic
• secondly, to kafka2 and produce some messages
• and finally, read all of the messages from within the kafka3

To do so, let’s start with the following:

docker exec -it kafka1 /bin/bash
kafka-topics --bootstrap-server kafka1:9092 --create --topic exampleTopic

# Output:
Created topic exampleTopic.

# To exit the container, please specify exit

As we can see, the exampleTopic was created successfully.

So as the next thing, let’s produce some messages from kafka2 broker:

docker exec -it kafka2 /bin/bash
kafka-console-producer --bootstrap-server kafka2:9092 --topic exampleTopic

>codersee
>rules 

# To exit the producer, please hit Ctrl + D. Then to exit the container please type 'exit'.

No errors have been thrown, which let us assume that everything is good so far.

Nevertheless, to fully make sure, let’s consume these messages:

docker exec -it kafka3 /bin/bash
kafka-console-consumer --bootstrap-server kafka3:9092 --topic exampleTopic --from-beginning

# Output:
codersee
rules

# To exit the consumer, please hit Ctrl + C. As the output, we should see:
^CProcessed a total of 2 messages

# And again, to exit the container please type 'exit'.

As we can see, everything worked as expected and our brokers are correctly communicating within the cluster.

8. Kafka Without Zookeeper Using Docker Compose Summary

And that’s all for today’s article on how to set up Kafka Without Zookeeper using Docker Compose.

I really hope that you enjoyed this and my previous articles in an Apache Kafka series and I will be forever thankful if you would like to share your feedback with me in the comments section below (positive and negative, as well).

Take care and have a great week! 🙂

The post How To Set Up Kafka Without Zookeeper using Docker Compose? appeared first on Codersee blog- Kotlin on the backend.

Hello, in this step-by-step guide, I will show you how to set up multiple Kafka brokers with Docker Compose 🙂

In this article, together we will:

• see the minimum Docker Compose config required to spin up 3 Kafka brokers,
• take a closer look at the configuration,
• get information about active brokers with the zookeeper-shell command,
• verify whether everything is working as expected with Console Producer and Consumer tools.

Unlike the previous article about Kafka, I would like to start this one by showing the configuration you’ll need to set up 3 Kafka brokers. After that, we will go through it as thoroughly as possible to understand it even better and in the end, we will learn how to verify and test our config.

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. Multiple Kafka Brokers Docker Compose

With that being said, let’s see the docker-compose.yaml:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
    ports:
      - "8097:8097"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8097,INTERNAL://kafka1:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
    ports:
      - "8098:8098"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8098,INTERNAL://kafka2:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3
    ports:
      - "8099:8099"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8099,INTERNAL://kafka3:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

[elementor-template id=”9007393″]

2.1. Docker Images Versions and Containers Names

Firstly, let’s take a look at docker images versions and containers names:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
...
  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka1
...
  kafka2:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka2
...
  kafka3:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka3

As we can see, in our example we will use Confluent Community Docker Image for Apache Kafka and Confluent Docker Image for Zookeeper– both in version 7.2.1.

Additionally, we explicitly set names for our containers, which are discoverable at a hostname identical to the container name. Simply put, container exposed port 9092 of kafka1 will be visible for other containers at kafka1:9092.

2.2. Containers Ports

Nextly, let’s have a look at the exposed ports:

...
  kafka1:
    ports:
      - "8097:8097"
...
  kafka2:
    ports:
      - "8098:8098"
...
  kafka3:
    ports:
      - "8099:8099"

As can be seen, we did it for each service except the zookeeper.

You might already know this, but ports is responsible for exposing container ports to the host machine. Simply put, when running docker-compose on our computer with a 123:456 setting, we will be able to access a container’s 456 port through localhost:123.

2.3. Docker Compose Depends On

The last thing, I would like to cover before we head to the environment settings of our Kafka brokers is the depends_on .

As we can see, we put it for each Kafka service specifying the zookeeper service as an argument. With this one, we simply make sure that when running a docker compose up command, the zookeeper service will be started before the brokers.

3. Multiple Kafka Brokers Environment Variables

If you’ve seen my previous article on how to set up a single-node cluster with docker-compose, then you will notice that it looks a bit different. The main reason behind this is that with multiple Kafka brokers in our node, they have to know how to communicate with each other.

3.1. Zookeeper Client Port

As the first step, let’s take a look at the ZOOKEEPER_CLIENT_PORT.

To put it simply, this one instructs Zookeeper where to listen for connections by clients- Kafka brokers in our example.

3.2. Kafka Broker ID

Following, check the KAFKA_BROKER_ID variable.

By default, when we don’t set this one, the Zookeeper will automatically assign a unique broker identifier. Nevertheless, if we would like to specify it explicitly, then we can take care of it with this environment variable. Of course, please keep in mind that this value has to be unique across our Kafka brokers.

3.3. Kafka Zookeeper Connect

As the next thing, we set the KAFKA_ZOOKEEPER_CONNECT in each service to point to the Zookeeper.

As the value, we have to set our Zookeeper container name with a port set with the ZOOKEEPER_CLIENT_PORT directive.

3.4. Kafka Listener Settings

Finally, let’s see the rest of the environment variables responsible for communication:

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
• KAFKA_ADVERTISED_LISTENERS
• KAFKA_INTER_BROKER_LISTENER_NAME

Unlike the previous ones, I believe it has more sense to take a look at all of them at once to get a better understanding.

With a KAFKA_ADVERTISED_LISTENERS, we set a comma-separated list of listeners with their host/IP and port. This value must be set in a multi-node environment because otherwise, clients will attempt to connect to the internal host address. Please keep in mind that the listeners’ values (EXTERNAL and INTERNAL in our case), must be unique here.

When it comes to the naming here, we can specify our own values, just as I did with EXTERNAL/INTERNAL. Nevertheless, in such a case, we have to specify the correct mapping with KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:

• EXTERNAL will be mapped to PLAINTEXT protocol,
• INTERNAL will be mapped the same way, as well

If you are wondering why have we done it, then the reason is mentioned above- listeners’ values have to be unique, so we can’t specify PLAINTEXT://localhost:8099,PLAINTEXT://kafka3:9092 .

Finally, with KAFKA_INTER_BROKER_LISTENER_NAME we explicitly specify which listener to use for inter-broker communication.

4. Multiple Kafka With Docker Compose Testing

With all of that being said, we can finally make use of our docker-compose.yaml file.

To do so, let’s build and run our services with the following command:

docker compose up

Depending on our machine, it may take a while until everything is up and running.

In the end, we should see the following:

...Recorded new controller, from now on will use broker...

4.1. Verify All Containers Running

As the next step, let’s open up a new terminal window and list up and running containers with docker ps:

# Output:
CONTAINER ID IMAGE                           COMMAND                CREATED        STATUS        PORTS                            NAMES
cf892aeb5793 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8098->8098/tcp, 9092/tcp kafka2
a10bef7b6a54 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8099->8099/tcp, 9092/tcp kafka3
46c9ca4862b6 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 0.0.0.0:8097->8097/tcp, 9092/tcp kafka1
eef5ebec8519 confluentinc/cp-zookeeper:7.2.1 "/etc/confluent/dock…" 54 minutes ago Up 54 minutes 2181/tcp, 2888/tcp, 3888/tcp     zookeeper

As we can clearly see, all Kafka containers have their ports exposed to our host and Zookeeper is reachable through 2181, as well.

4.2. Verify Active Kafka Brokers Using zookeeper-shell

Following, let’s run a zookeeper-shell on the zookeeper container:

docker exec -it zookeeper /bin/zookeeper-shell localhost:2181

# Output:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

What’s important to mention here is that we run a zookeeper-shell command from within the container. That’s the reason why the zookeeper_host value is set to localhost and accessible, even though we haven’t exposed it.

Thanks to the -it flags, we can use the shell from our terminal.

4.3. Get IDs of Currently Active Brokers

So, to get the identifiers of currently active Kafka Brokers, we can use the following:

ls /brokers/ids

# Output:
[1, 2, 3]

The above output clearly indicates that everything is working, as expected.

Note: to close the zookeeper-shell, please hit Ctrl + C (and please do it before heading to the Producer/Consumer parts)

5. Publish Messages With Console Producer

After we know all of the above, we can finally send messages using the Console Producer.

As the first step, let’s run bash from a randomly picked Kafka broker container:

docker exec -it kafka2 /bin/bash

Nextly, let’s create a new topic called randomTopic:

kafka-topics --bootstrap-server localhost:9092 --create --topic randomTopic

# Output:
Created topic randomTopic.

Please note, that given the kafka-topics script is run inside of the container, we point to the 9092 port of localhost.

Following, we can produce some messages to our new topic:

kafka-console-producer --bootstrap-server localhost:9092 --topic randomTopic 

> lorem
> ipsum

# When we finish, we have to hit ctrl + d

Of course, we could do that from any other broker container (and in order to exit bash, we need to type exit in the command prompt).

6. Read Messages With Console Consumer

As the last step, let’s connect to another Apache Kafka broker and read previously sent messages:

docker exec -it kafka3 /bin/bash

kafka-console-consumer --bootstrap-server localhost:9092 --topic randomTopic --from-beginning

# Output:
lorem
ipsum

As we can clearly see, the output contains all messages, we’ve sent using Console Producer.

7. Summary

And that would be all for this article on how to set up multiple Apache Kafka brokers using Docker Compose and produce/consume messages. As a follow-up, I would recommend checking out this Confluent article about connecting to Kafka with different configurations.

Finally, let me know your thoughts in the comment section bellow, dear friend! 😀

The post How To Deploy Multiple Kafka Brokers With Docker Compose? appeared first on Codersee blog- Kotlin on the backend.

If you would like to learn how to set up Apache Kafka using Docker Compose then you’ve come to the right place 🙂

In this article, together we will:

• see the bare minimum Docker Compose configuration we must provide to spin up an environment with exactly one Kafka broker and Zookeeper instance,
• take a closer look at the required configurations,
• verify our setup using Console Producer and Consumer tools.

If you will enjoy this tutorial, then I highly encourage you to check out the continuations:

• How To Deploy Multiple Kafka Brokers With Docker Compose?
• How To Set Up Kafka Without Zookeeper using Docker Compose

In my articles, I focus on the practice and getting things done.

However, if you would like to get a strong understading of DevOps concepts, which are more and more in demand nowadays, then I highly recommend to check out KodeKloud courses, like this Docker and Kubernetes learning paths.

2. What is Apache Kafka?

Before we start, I feel obliged to provide at least a brief explanation of what Apache Kafka and Zookeeper are. Please skip the following two paragraphs if you want to get straight to the practice part of this tutorial.

Well, Apache Kafka by definition is an open-source distributed event streaming platform. Simply put, it’s a platform widely used to work with real-time streaming data pipelines and to integrate applications to work with such streams. It’s mainly responsible for:

• publishing/subscribing to the stream of records,
• their real-time processing,
• and their ordered storage

But let’s just be honest- it’s almost impossible to describe Kafka in a couple of sentences and I highly encourage you to visit the Apache Intro page after this tutorial right here.

3. What is Apache ZooKeeper?

Or the question should rather be why do I even need another service to work with Apache Kafka?

Just like a zookeeper in a zoo takes care of the animals, Apache Zookeeper “takes care” of all the Kafka brokers in a cluster. It’s mainly responsible for:

• tracking the cluster state- like which brokers are a part of the cluster, sending notifications to Kafka in case of changes
• topics configuration
• controller election
• access control lists and quotas

Nevertheless, please keep in mind that the Kafka team is constantly working on replacing the ZooKeeper with Apache Kafka Raft (KRaft). To be more specific- the 2.8.0 version introduced early access to this functionality, but at the moment of publishing this article, it’s still not production-ready.

4. Bare Minimum Kafka Docker Compose

With all of that being said, let’s have a look at the bare minimum version of the docker-compose.yml file. This configuration is necessary to spin up exactly 1 Kafka and Zookeeper instance:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka
    ports:
      - "8098:8098"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:8098
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Again, before we head to the next commands, let’s take a couple of minutes to understand, what exactly is going on here.

4.1. Docker Images Version And Containers

Firstly, let’s take a look at this piece of code:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.1
    container_name: zookeeper
...   
  kafka:
    image: confluentinc/cp-kafka:7.2.1
    container_name: kafka

As we can see, with this config we will spin up two services named kafka and zookeeper. The container_name will set specified names for our containers, instead of letting Docker generate them. Finally, we will use the 7.2.1 versions of Confluent Community Docker Image for Apache Kafka and Confluent Docker Image for Zookeeper.

4.2. Additional Kafka Container Configs

As the next step, let’s have a quick look at these four lines:

ports:
  - "8098:8098"
depends_on:
  - zookeeper

As the name suggests, the ports command is responsible for exposing ports, so with this setting, port 8098 of our container will be accessible through the 8098 port of the host machine. Additionally, the depends_on will make sure to start the zookeeper container before the kafka.

5. Required Settings

As the next step, let’s take a closer look at the required settings for both Kafka and Zookeeper in our Docker Compose settings.

5.1. Required Zookeeper Settings

As we can see in our example, the only environment we specified is ZOOKEEPER_CLIENT_PORT. This one instructs Zookeeper where it should listen for connections by clients- Kafka in our case.

Additionally, when running in clustered mode, we have to set the ZOOKEEPER_SERVER_ID (but it’s not the case here).

5.2. Required Confluent Kafka Settings

When it comes to the Confluent Kafka Docker image, here is the list of the required environment variables:

• KAFKA_ZOOKEEPER_CONNECT – instructing Kafka where it can find the Zookeeper
• KAFKA_ADVERTISED_LISTENERS – specifying the advertised hostname, which clients can reach out to. Moreover, this value is sent to the Zookeeper

Please keep in mind, that in order to run only one instance of Kafka (aka single-node cluster), we have to additionally specify KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR set to 1. Its default value is 3 and if we don’t do that, we will get into this error: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1

6. Verify Kafka Docker Compose Config

With all of that being said, we can finally check out if everything is working, as expected.

6.1. Start Containers

As the first step, we have to start our containers:

docker compose up -d

The -d flag instructs the docker to run containers in a detached mode.

Of course, we can verify everything by running the docker ps command:

# example output:
CONTAINER ID IMAGE                           COMMAND                CREATED        STATUS       PORTS                            NAMES
38b41bc43676 confluentinc/cp-kafka:7.2.1     "/etc/confluent/dock…" 42 seconds ago Up 4 seconds 0.0.0.0:8098->8098/tcp, 9092/tcp kafka
45484184f330 confluentinc/cp-zookeeper:7.2.1 "/etc/confluent/dock…" 42 seconds ago Up 5 seconds 2181/tcp, 2888/tcp, 3888/tcp     zookeeper

6.2. Create Kafka Topic

Nextly, let’s create a new Kafka topic:

docker exec kafka kafka-topics --bootstrap-server localhost:8098 --create --topic example

As a word of explanation, the kafka-topics is a shell script that allows us to execute a TopicCommand tool. It’s a command-line tool that can manage and list topics in a Kafka cluster. Additionally, we have to specify the listener hostname (specified with KAFKA_ADVERTISED_LISTENERS) with –bootstrap-server.

When the topic is created we should see the following message: “Created topic example.”

6.3. Run Kafka Producer

Following, let’s run a console producer with the kafka-console-producer:

docker exec --interactive --tty kafka kafka-console-producer --bootstrap-server localhost:8098 --topic example

Please keep in mind that this command will start the producer and it will wait for our input (and you should notice the > sign). Please specify a couple of messages and hit Ctrl + D after you finish:

>lorem
>ipsum
# hit Ctrl + D

6.4. Run Kafka Consumer

As the last step, let’s run a console consumer with the kafka-console-consumer command:

docker exec --interactive --tty kafka kafka-console-consumer --bootstrap-server localhost:8098 --topic example --from-beginning

This time, we should see that our messages are printed to the output successfully (and to finish please hit Ctrl+ C):

lorem
ipsum

7. Kafka With Docker Compose Summary

And that would be all for this step-by-step guide on How To Set Up Apache Kafka With Docker Compose.

Let me know about your thoughts in the comment section, or reach out through the contact form- I highly appreciate your thoughts, comments and feedback.

Take care brother/sister!

The post How To Set Up Apache Kafka With Docker? appeared first on Codersee blog- Kotlin on the backend.