I can guarantee that you will benefit from this tutorial regardless of whether you saw previous articles about S3Client or S3Template, or not. But, I definitely encourage you to take a look at them, too:

• #1 Spring Boot with AWS S3, S3Client, and Kotlin
• #2 Spring Boot with Kotlin, AWS S3, and S3Template
• #3 (This article)

Video Tutorial

If you prefer video content, then check out my video that covers all three articles:

If you find this content useful, please leave a subscription 🙂

Prerequisites

Before heading to the guide, I just wanted to emphasize that today we will be working with Testcontainers. And this means that we must have a supported Docker environment.

So, if you do not have Docker configured on your local and want to follow this article, please check out their documentation.

Of course, you must have Java, IDE, and Spring Boot project too, but I believe this is quite obvious 😉

Testcontainers and LocalStack

Lastly, I would like to say a few words about the Testcontainers and LocalStack, which in my opinion are a great way to test Spring Boot S3 integration (and other AWS integrations, too).

Testcontainers

Testcontainers is a library for providing throwaway, lightweight instances of Docker containers. They are an excellent approach whenever need to test behavior dependent on external services, like AWS, or some external databases.

Long story short, instead of mocking, or manual set up of some test environment, we define test dependencies as code. Then, we can run our test code and disposable containers will be started and deleted after they finish.

Let’s take a look at the example from Spring Boot docs:

@Testcontainers
@SpringBootTest
class MyIntegrationTests {

  @Test
  fun myTest() {
    // ...
  }

  companion object {
    @Container
    @JvmStatic
    val neo4j = Neo4jContainer("neo4j:5");
  }
}

The above code runs a Neo4j docker container before the tests. Of course, this is just an example, so most of the time, we will need to add some more config.

Nevertheless, we can clearly see that this Testcontainers JUnit integration allows us to achieve our goal in an easy and neat manner.

Localstack

Localstack, on the other hand, is a cloud service emulator that runs in a single container. In other words, we can run AWS applications or Lambdas without connecting to the remote cloud provider.

And thanks to the Testcontainers module for LocalStack, we can test various AWS integrations with just a few lines of code.

Again, let’s take a look at the example, but this time from the LocalStack documentation:

DockerImageName localstackImage = DockerImageName.parse("localstack/localstack:3.5.0");

@Rule
public LocalStackContainer localstack = new LocalStackContainer(localstackImage)
        .withServices(S3);

You will find links to both documentation at the end of this article. But for now, let’s not distract ourselves and focus on what we came here for 😉

Configure Project

If you are following my S3 series, or you already have a Spring Boot project, then those are the necessary dependencies for us today:

testImplementation("org.springframework.boot:spring-boot-starter-webflux")
testImplementation("org.testcontainers:localstack")
testImplementation("org.springframework.boot:spring-boot-testcontainers")

As we can see, apart from LocalStack and TestContainers, we must provide the Spring Boot Starter WebFlux.

But why?

Well, this is necessary to work with WebTestClient- a client we will use to test our web servers (REST endpoints).

On the other hand, if you would like to set up a project from scratch, then please navigate to the Spring Initializr and select the following:

However, please keep in mind that LocalStack is not provided out of the box in Spring, so we must add it manually.

Moreover, as we have chosen the Spring Web, the WebFlux dependency is not present, too:

testImplementation("org.springframework.boot:spring-boot-starter-webflux")
testImplementation("org.testcontainers:localstack")

Testcontainers Singleton Approach

With all of that being done, let’s head to the practice part.

When working with Testcontainers, we can configure them in various ways:

• we can use the JUnit extension (Jupiter integration)- which allows us to use @Testcontainers and @Container annotations and makes JUnit responsible for the automatic startup and stop of containers in our tests.
• we can configure them manually in every test case,
• or, alternatively, we can use the singleton approach– in which we control containers’ lifecycle manually. But, thanks to that we can easily reuse them across multiple test classes.

Of course, these are not all approaches, and based on your needs you may want to configure Testcontainers differently. Nevertheless, in this tutorial, we will focus on the manual, reusable approach.

Introduce Base Class

Firstly, let’s introduce the LocalStackIntegrationTest:

@SpringBootTest(webEnvironment = RANDOM_PORT)
class LocalStackIntegrationTest { }

As we can see, we mark our class with @SpringBootTest – annotation necessary to run our integration tests and inject the instance of WebTestClient later in our subclasses.

Add Testcontainer

Following, let’s take a look at how to instantiate a LocalStack container:

companion object {
  val localStack: LocalStackContainer = LocalStackContainer(
    DockerImageName.parse("localstack/localstack:3.7.2")
  )
}

Right here, we create an instance of LocalStackContainer and we pass the name of a Docker image – localstack/localstack:3.7.2– to its constructor. Alternatively, if we are working only with AWS S3 Buckets service, then we can use a dedicated image- localstack:s3-latest. But personally, I am not a big fan of the latest tag, which can easily break our code.

Additionally, we put the LocalStackContainer instance in the companion object. Why? Because in the next steps, we will reference it in a function annotated with @DynamicPropertySource– and it must be static.

Note related to JUnit extension:

This is not the case here, as we want to take care of the container lifecycle manually, but, when using the Jupiter integration, containers declared as static fields will be shared between test methods. They will be started only once before any test method is executed and stopped after the last test method has executed. So, if in your case you pick the JUnit extension and don’t want that to happen, then you must not put the localstack in the companion object.

Control Testcontainer lifecycle

As we already know, with this approach we are responsible for the container lifecycle control. And although this may sound complicated, it basically means that without the extension we must start the container manually.

So the companion object after the update will look, as follows:

companion object {
  val localStack: LocalStackContainer = LocalStackContainer(
    DockerImageName.parse("localstack/localstack:3.7.2")
  ).apply {
    start()
  }
}

Basically, we use the Kotlin scope function (you can learn more about it in my Kotlin course) to invoke the start() function on the localStack instance. And as the name suggests, this function will start the container (and pull the image, if necessary).

And basically, that is all we need to do here. With the above code, the container will be started when the base class is loaded and shared across all inheriting test classes.

Of course, there is also the stop() function that we can invoke to kill and remove the container.

Nevertheless, we do not have to do it. Why? Let’s figure out.

Ryuk

Ryuk is a kind of “garbage collector” in Testcontainers.

Whenever we run integration tests, Testcontianers core starts one more container:

Long story short, this container is responsible for removing containers/networks/volumes created by our test cases. So, even if we do not clean the environment ourselves- for example with the stop() function- the Ryuk container will take care of that.

Test Properties With DynamicPropertySource

With that done, we need to update our environment configuration.

If we try to run our Spring Boot application at this point, our logic responsible for communication with Amazon S3 will try to reach the actual AWS instance. It will use the defaults, or make use of the things we configured in the application.yaml.

And this is not what we want, right? Instead, we would like to connect to the Testcontainer LocalStack instance.

In some examples, you might have seen the usage of application properties files. Nevertheless, if we want to be more flexible and make use of containers started on random ports, then the @DynamicPropertySource is our best friend here:

companion object {
  val localStack: LocalStackContainer = LocalStackContainer(
    DockerImageName.parse("localstack/localstack:3.7.2")
  ).apply {
    start()
  }

  @JvmStatic
  @DynamicPropertySource
  fun overrideProperties(registry: DynamicPropertyRegistry) {
    registry.add("spring.cloud.aws.region.static") { localStack.region }
    registry.add("spring.cloud.aws.credentials.access-key") { localStack.accessKey }
    registry.add("spring.cloud.aws.credentials.secret-key") { localStack.secretKey }
    registry.add("spring.cloud.aws.s3.endpoint") { localStack.getEndpointOverride(S3).toString() }
  }

}

Thanks to that annotation, we can dynamically provide values to our test environment based on the LocalStack instance.

Of course, we must remember that methods annotated with @DynamicPropertySource must be static! And that’s why we use it with the @JvmStatic annotation.

Utilize LocalStack AWS CLI

At this point, we have our base class ready, but before we head to the tests, I would like to show you the LocalStack AWS CLI and why and how to use it.

As the first step, let’s create the util package and add the LocalStackUtil.kt file:

import org.testcontainers.containers.localstack.LocalStackContainer

fun LocalStackContainer.createBucket(bucketName: String) {
  this.execInContainer("awslocal", "s3api", "create-bucket", "--bucket", bucketName)
}

fun LocalStackContainer.deleteBucket(bucketName: String) {
  this.execInContainer("awslocal", "s3api", "delete-bucket", "--bucket", bucketName)
}

fun LocalStackContainer.deleteObject(bucketName: String, objectName: String) {
  this.execInContainer("awslocal", "s3api", "delete-object", "--bucket", bucketName, "--key", objectName)
}

As we can see, we introduced 3 helper extension functions that we will later use to create and delete buckets and objects. This way, we can easily clean up between the tests (we use the shared approach, right?). Moreover, it will simplify the setup process for each test case.

The above code combines the execInContainer – which will run the passed command in our running LocalStack container, just like with the docker exec, and the awslocal– a LocalStack wrapper around the AWS CLI. So, if you’ve ever been working with the AWS command line interface, then you will see that this is 1:1.

Unfortunately, we must provide the command as a separate String value, because otherwise, we will end up with:

OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: “awslocal s3api create-bucket –bucket bucket-1”: executable file not found in $PATH: unknown

Write Integration Test Cases

With all of that LocalStack preparation done (I know, quite a bunch of things to learn, but once you learn this, it will be a simple copy-paste), we can finally write some integration tests for our Spring Boot S3 integration.

Firstly, let’s create the controller package and put the BucketControllerIntegrationTest class:

class BucketControllerIntegrationTest(
  @Autowired private val webTestClient: WebTestClient
) : LocalStackIntegrationTest() { }

As we can see, no annotations are required. We simply extend the LocalStackIntegrationTest class and inject the WebTestClient.

Test No Buckets Exist

Nextly, let’s introduce our first test case. If we do not do anything, we expect that no buckets exist in our S3 instance:

@Test
fun `Given no existing buckets When getting list of buckets Then return an empty array`() {
  val buckets = webTestClient
    .get().uri("/buckets")
    .exchange()
    .expectStatus().isOk()
    .expectBody(object : ParameterizedTypeReference<List<String>>() {})
    .returnResult()
    .responseBody

  assertNotNull(buckets)
  assertTrue(buckets.isEmpty())
}

As mentioned before, we use the WebTestClient to make a GET HTTP request to the /buckets endpoint. Then, we use a small hack with ParameterizedTypeReference– because the endpoint returns a list of Strings and we use Kotlin- and we obtain the response body.

Lastly, we have plain assertions. We verify that the response body is not null and that our S3 bucket list is empty.

Verify S3 Bucket Exists In LocalStack

Following, let’s see our helper functions in action:

@Test
fun `Given one existing bucket When getting list of buckets Then return an array with expected bucket name`() {
  val bucketName = "bucket-1"
  localStack.createBucket(bucketName)

  val expectedJson = """
    [ "Bucket #1: $bucketName" ]
  """

  webTestClient
    .get().uri("/buckets")
    .exchange()
    .expectStatus().isOk()
    .expectBody()
    .json(expectedJson)

  localStack.deleteBucket(bucketName)
}

This time, we utilize the createBucket and make sure that the /buckets endpoint returns the expected JSON. Please note that this is another way to assert the response body.

After all, we delete the existing bucket, so it won’t affect other test cases.

Assert Bucket Created Successfully

As the next step, let’s take a look at how we can check if our endpoint responsible for creating new S3 buckets works. And I see two paths we can go here.

The first one, using the execInContainer:

@Test
fun `Given no existing buckets When creating bucket Then create bucket successfully`() {
  val bucketName = "bucket-2"
  
  webTestClient
    .post().uri("/buckets")
    .bodyValue(BucketRequest(bucketName = bucketName))
    .exchange()
    .expectStatus().isOk()
  
  val execResult = localStack.execInContainer("awslocal", "s3api", "list-buckets").stdout
  
  assertTrue(execResult.contains(bucketName))  
  localStack.deleteBucket(bucketName)
}

The important thing to mention here is that the execInContainer returns the ExecResult. And thanks to that, we can read additional info, like stdout, stderr, or exitCode.

And thanks to the stdout, we can get this JSON to verify it contains particular bucket name (or even we could parse that to an object):

{
  "Buckets": [
    {
      "Name": "bucket-2",
      "CreationDate": "2024-09-19T05:28:42.000Z"
    }
  ],
  "Owner": {
    "DisplayName": "webfile",
    "ID": "75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a"
  }
}

Alternatively, we can use the /buckets endpoint once again, too:

@Test
fun `Given no existing buckets When creating bucket Then create bucket successfully`() {
  val bucketName = "bucket-2"

  webTestClient
    .post().uri("/buckets")
    .bodyValue(BucketRequest(bucketName = bucketName))
    .exchange()
    .expectStatus().isOk()

  val expectedJson = """
    [ "Bucket #1: $bucketName" ]
  """
  webTestClient
    .get().uri("/buckets")
    .exchange()
    .expectStatus().isOk()
    .expectBody()
    .json(expectedJson)

  localStack.deleteBucket(bucketName)
}

Test Remaining Cases

The remaining cases of our integration test use a more or less similar approach, so I will simply copy-paste them here so that you can analyze them.

At this point, I am pretty sure you understand the general idea behind what I understand by testing of Spring Boot S3 integration with LocalStack, so I don’t see the need for explaining them one- by one:

@Test
fun `Given no objects existing in the bucket When getting objects of a bucket Then return an empty array`() {
  val bucketName = "bucket-3"
  localStack.createBucket(bucketName)

  val objects = webTestClient
    .get().uri("/buckets/$bucketName/objects")
    .exchange()
    .expectStatus().isOk()
    .expectBody(object : ParameterizedTypeReference<List<String>>() {})
    .returnResult()
    .responseBody

  assertNotNull(objects)
  assertTrue(objects.isEmpty())

  localStack.deleteBucket(bucketName)
}

@Test
fun `Given no objects When creating example object Then return created object`() {
  val bucketName = "bucket-4"
  val objectName = "example.json"
  localStack.createBucket(bucketName)

  val expectedJson = """
    {
      "id": "123",
      "name": "Some name"
    }
  """

  webTestClient
    .post().uri("/buckets/$bucketName/objects")
    .exchange()
    .expectStatus().isOk()
    .expectBody()
    .json(expectedJson)

  localStack.deleteObject(bucketName, objectName)
  localStack.deleteBucket(bucketName)
}

@Test
fun `Given created object When getting list of objects Then return array with one object`() {
  val bucketName = "bucket-5"
  val objectName = "example.json"
  localStack.createBucket(bucketName)

  val expectedJson = """
    [ $objectName ]
  """

  webTestClient
    .post().uri("/buckets/$bucketName/objects")
    .exchange()
    .expectStatus().isOk()

  webTestClient
    .get().uri("/buckets/$bucketName/objects")
    .exchange()
    .expectStatus().isOk()
    .expectBody()
    .json(expectedJson)

  localStack.deleteObject(bucketName, objectName)
  localStack.deleteBucket(bucketName)
}

@Test
fun `Given existing object When getting object by key Then return object content`() {
  val bucketName = "bucket-6"
  val objectName = "example.json"
  localStack.createBucket(bucketName)

  val expected = """
    {
      "id": "123",
      "name": "Some name"
    }
  """

  webTestClient
    .post().uri("/buckets/$bucketName/objects")
    .exchange()

  webTestClient
    .get().uri("/buckets/$bucketName/objects/$objectName")
    .exchange()
    .expectStatus().isOk()
    .expectBody()
    .json(expected)

  localStack.deleteObject(bucketName, objectName)
  localStack.deleteBucket(bucketName)
}

@Test
fun `Given existing bucket with object When deleting bucket Then bucket is removed`() {
  val bucketName = "bucket-7"
  localStack.createBucket(bucketName)

  webTestClient
    .post().uri("/buckets/$bucketName/objects")
    .exchange()
    .expectStatus().isOk()

  webTestClient
    .delete().uri("/buckets/$bucketName")
    .exchange()
    .expectStatus().isOk()

  val buckets = webTestClient
    .get().uri("/buckets")
    .exchange()
    .expectStatus().isOk()
    .expectBody(object : ParameterizedTypeReference<List<String>>() {})
    .returnResult()
    .responseBody

  assertNotNull(buckets)
  assertTrue(buckets.isEmpty())
}

Summary

And that is all for this tutorial, in which we learned how to implement integration tests for Spring Boot AWS S3 integration with LocalStack and Testcontainers.

I hope you enjoyed it and for the source code, please visit this GitHub repository.

Have a great day and see you in the next articles! 🙂

The post Test Spring Boot AWS S3 with Localstack and Testcontainers appeared first on Codersee blog- Kotlin on the backend.

Of course, I highly encourage you to take a look at other articles in this series, too:

• #1 Spring Boot with AWS S3, S3Client, and Kotlin
• #2 (This article)
• #3 Test Spring Boot AWS S3 with Localstack and Testcontainers

Video Tutorial

If you prefer video content, then check out my video that covers all three articles:

If you find this content useful, please leave a subscription 🙂

What is S3Template?

Before we get our hands dirty with the code, let’s take a second to understand better with S3Template is and how it can make our lives easier when integrating a Spring Boot app with S3.

Let me quote the S3Template documentation here:

Higher level abstraction over S3Client providing methods for the most common use cases.

So, if you have already worked with, or you have seen my previous article with S3Client, then you saw that simple operations require some boilerplate. And that is exactly what S3Template solves.

And as a note from my end, I just wanted to note S3Template handles only some subset of S3Client operations, so in our projects those two will rather coexist, instead of being each others alternatives.

AWS S3Template Operations

With all of that said, let’s get to work.

Let’s add the controller package and BucketController class to it. We will use it to expose a bunch of endpoints triggering various operations on S3 buckets and files.

When it comes to the operations- we will use the same ones as in the previous article, and as the last one, I will show you how to serialize and deserialize objects with AWS S3 and S3Template.

List All Buckets

Although the S3Template does not expose any method that would help us with this task, I wanted to mention it as we discussed it in the previous tutorial.

Additionally, this is a great example of S3Client and S3Template co-existence:

@RestController
@RequestMapping("/buckets")
class BucketController(
  private val s3Template: S3Template,
  private val s3Client: S3Client,
) {

  @GetMapping
  fun listBuckets(): List<String> {
    val response = s3Client.listBuckets()

    return response.buckets()
      .mapIndexed { index, bucket ->
        "Bucket #${index + 1}: ${bucket.name()}"
      }
  }
}

As we can see, not too much S3Template could improve here, so I bet this is the reason why it was not introduced.

New S3 Bucket

Nextly, let’s take a look at how we can create a brand new bucket:

@PostMapping
fun createBucket(@RequestBody request: BucketRequest) {
  s3Template.createBucket(request.bucketName)
}

data class BucketRequest(val bucketName: String)

As we can see, no additional request classes- the only thing we need is the bucket name.

Of course, let’s rerun our application and verify if everything is working:

curl --location --request POST 'http://localhost:8080/buckets' \
--header 'Content-Type: application/json' \
--data-raw '{
    "bucketName": "codersee-awesome-bucket"
}'

As a result, we should get 200 OK without a response body.

Upload File to S3 Bucket

Nextly, let’s take a look at how to upload a new file to S3.

We have a few options, among which the store function is the easiest:

@PostMapping("/{bucketName}/objects")
fun createObject(@PathVariable bucketName: String, @RequestBody request: ObjectRequest) {
  s3Template.store(bucketName, request.objectName, request.content)
}

data class ObjectRequest(val objectName: String, val content: String)

As we can see, this function takes three arguments:

• the bucket name,
• filename,
• and the content to upload (to be specific Object object)

Alternatively, we could use the upload function, which allows us to send InputStream instance and metadata:

@Override
public S3Resource upload(
  String bucketName, 
  String key, 
  InputStream inputStream,
  @Nullable ObjectMetadata objectMetadata
) 

Lastly, let’s verify that everything is fine with the following curl:

curl --location --request POST 'http://localhost:8080/buckets/codersee-awesome-bucket/objects' \
--header 'Content-Type: application/json' \
--data-raw '{
    "objectName": "file-example.txt",
    "content": "My file content"
}'

If everything worked, then a new file should be present in our bucket 🙂

List Files From The Bucket

Nextly, let’s see how we can list the bucket content with S3Template:

@GetMapping("/{bucketName}/objects")
fun listObjects(@PathVariable bucketName: String): List<String> =
  s3Template.listObjects(bucketName, "")
    .map { s3Resource -> s3Resource.filename }

We can clearly see that this is not rocket science 😉

Nevertheless, it is worth mentioning that this time we get the S3Resource instance and instead of the key() we use it getFilename method.

And just like previously, let’s see the endpoint in action:

curl --location --request POST 'http://localhost:8080/buckets/codersee-awesome-bucket/objects' \
--header 'Content-Type: application/json' \
--data-raw '{
    "objectName": "file-example.txt",
    "content": "My file content"
}'

# Response status: 200 OK
# Response body: 
[
    "file-example.txt"
]

Download a File

So what about fetching files from the S3 bucket?

With S3Template, it’s a piece of cake:

@GetMapping("/{bucketName}/objects/{objectName}")
fun getObject(@PathVariable bucketName: String, @PathVariable objectName: String): String =
  s3Template.download(bucketName, objectName).getContentAsString(UTF_8)

We specify the bucket name and the object name, and as a result, we get the S3Resource that exposes a bunch of methods. Among others, the getContentAsString that is pretty descriptive 😉

Similarly, let’s hit the endpoint:

curl --location --request GET 'http://localhost:8080/buckets/codersee-awesome-bucket/objects/file-example.txt'

# Response status: 200 OK
# Response body: 
"My file content"

Delete the Bucket

Last before least, let’s take a look at how we can get rid of the bucket:

@DeleteMapping("/{bucketName}")
fun deleteBucket(@PathVariable bucketName: String) {
  s3Template.listObjects(bucketName, "")
    .forEach { s3Template.deleteObject(bucketName, it.filename) }

  s3Template.deleteBucket(bucketName)
}

And just like in the previous article- we must ensure the bucket does not contain any objects.

To do so, we list out objects by specifying the bucket name and objects prefix (as we don’t have any, we pass an empty String). Then, for each object, we use its key to delete it. And lastly, we simply invoke the deleteBucket by passing the name of a bucket to delete.

Of course, let’s verify this logic, too:

curl --location --request DELETE 'http://localhost:8080/buckets/codersee-awesome-bucket'

If we run this command and the S3 bucket exists, then we should see 200 OK and our bucket will disappear.

Serialize/Deserialize objects

As the last thing, let’s take a look at how easily we can persist objects using the combination of store and read functions:

@PostMapping("/{bucketName}/objects")
fun createExampleObject(@PathVariable bucketName: String): Example {
  val example = Example(id = UUID.randomUUID(), name = "Some name")

  s3Template.store(bucketName, "example.json", example)

  return s3Template.read(bucketName, "example.json", Example::class.java)
}

data class Example(val id: UUID, val name: String)

As we can see, we made a small update to our POST /{bucketName}/objects endpoint logic.

Basically, the first part is exactly the same, we use the store again to push the file to the bucket.

Nevertheless, instead of the download we saw previously, we use the read function that uses the S3ObjectConverter that will automatically deserialize the JSON into the Example class instance.

And for the last time, let’s hit our API:

curl --location --request POST 'http://localhost:8080/buckets/codersee-awesome-bucket/objects' \
--data-raw ''

# Response status: 200 OK 
# Response body: 
{
    "id": "3eacd8a3-48b2-4756-86db-e4c9f4e291da",
    "name": "Some name"
}

And as we can see, the output confirms that everything is working fine.

Summary

And that’s all for this article on how to make our lives easier in Spring Boot with S3Template.

I hope you enjoyed it, and again wanted to invite you to take a look at other content of this series:

• #1 Spring Boot with AWS S3, S3Client, and Kotlin
• #2 (This article)
• #3 Test Spring Boot AWS S3 with Localstack and Testcontainers

Lastly, just wanted to show that you can find the source code in this GitHub repository and that you can join my newsletter to stay up-to-date with Kotlin on the backend.

The post Spring Boot with Kotlin, AWS S3, and S3Template appeared first on Codersee blog- Kotlin on the backend.

What can you expect today?

Well, at the end of this tutorial, you will know precisely how to connect to the S3 service, resolve the most common issues related to that, as well as how to perform basic operations on buckets and objects using the S3Client approach.

In the future content, we will work a bit with an asynchronous client, learn when to use S3Template instead, and learn how to write tests properly, so do not hesitate to subscribe to my newsletter 😉

Video Tutorial

If you prefer video content, then check out my video that covers all three articles:

If you find this content useful, please leave a subscription 🙂

Project Setup

If you already have your Spring Boot project prepared, then feel free to skip this step.

But if that is not the case, then let’s navigate together to the Spring Initializr page and select the following settings:

As we can see, nothing related to the AWS S3 yet, but we imported the Spring Web dependency so that we could expose REST endpoints to test.

As always, please generate the project, extract it on your local, and import it to your favorite IDE.

AWS S3 Dependencies

It is worth mentioning that the S3Client is not a Spring Boot concept, but a class that comes from the AWS SDK.

However, to make our lives easier when working with Spring, we can make use of the Spring Cloud AWS, which simplifies using AWS-managed services in a Spring and Spring Boot applications.

To do so, let’s navigate to the build.gradle.kts and add the following:

implementation("io.awspring.cloud:spring-cloud-aws-starter-s3:3.1.1")

Why do I pick it over the AWS Java SDK?

Because it auto-configures various S3 integration-related components out of the box. Additionally, we can quickly configure a bunch of things using the application.yaml. And lastly, we are all Spring boyzz here, we don’t do things manually 🙂

Create Test S3 Bucket

Following, let’s navigate to the AWS Console to prepare a test bucket.

Again, feel free to skip it if you are looking for Spring Boot details.

Then, let’s find the S3 (aka “Simple Storage Service”) in the search bar:

Nextly, let’s hit the Create Bucket , provide a name for it, and leave the rest as is:

If everything succeeded, then we should see our bucket on the list:

Excellent, at this point we can get back to our Spring Boot project :).

Test S3Client Connection

With all of that done, let’s figure out whether we can connect our local app with AWS using the S3Client.

To do so, let’s create the BucketController class and introduce the GET endpoint:

@RestController
@RequestMapping("/buckets")
class BucketController(
  private val s3Client: S3Client
) {

  @GetMapping
  fun listBuckets(): List<String> {
    val response = s3Client.listBuckets()

    return response.buckets()
      .mapIndexed { index, bucket ->
        "Bucket #${index + 1}: ${bucket.name()}"
      }
  }
}

As we can see, the above logic will be responsible for exposing the GET /buckets endpoint and returning a list of bucket names as “Bucket #N: some-name”.

Moreover, the starter we are using automatically configures and registers an S3Client bean in the Spring Boot context. So, we simply inject that without any previous configuration.

Anyway, less talkie-talkie, and let’s test the endpoint:

curl --location --request GET 'http://localhost:8080/test' 

And depending on our local environment, we get the 200 OK with a bucket name:

[
    "Bucket #1: your-awesome-name"
]

Or the error related to AwsCredentialsProviderChain:

software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile ‘default’: ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container – neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]

Or even:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [software.amazon.awssdk.services.s3.S3ClientBuilder]: Factory method ‘s3ClientBuilder’ threw exception with message: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@15f35bc3: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@2bfb583b: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@7301eebe: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@76a805b7: Unable to contact EC2 metadata service.]

So, let’s learn why it worked (or not🙂).

Configuring AWS Credentials

Long story short, the Spring Cloud AWS starter configures the DefaultCredentialsProvider that looks for credentials in the following order:

1. Java System Properties – aws.accessKeyId and aws.secretAccessKey
2. Environment Variables – AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
3. Web Identity Token credentials from system properties or environment variables
4. Credential profiles file at the default location (~/.aws/ credentials) shared by all AWS SDKs and the AWS CLI
5. Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set and the security manager has permission to access the variable,
6. Instance profile credentials delivered through the Amazon EC2 metadata service

And if you got 200 OK, but you don’t remember specifying anything on your local machine, then I am pretty sure that the number 4  is the answer here 😉

The .aws folder inside the home directory is a default location for credentials, and you could have populated that unconsciously, for example when configuring AWS CLI with aws configure. And the DefaultCredentialsProvider was smart enough to use it without your knowledge.

The autoconfiguration is a wonderful thing, but as we can see, it can backfire sometimes 🙂

On the other hand, if none of these 6 satisfies you, then Spring Cloud AWS allows us to use the access keys, too.

To do so, the only thing we need to do is add the following to the application.yaml:

spring:
  cloud:
    aws:
      credentials:
        access-key: your-access-key
        secret-key: your-secret-key

Configure Region

OK, so at this point the problem related to the credentials should be gone. However, if we got the second error related to the region, then let’s see how it works internally, too.

Well, when we take a look at the DefaultAwsRegionProviderChain docs, we will see that it looks for the region in this order:

1. Check the aws.region system property for the region.
2. Check the AWS_REGION environment variable for the region.
3. Check the {user. home}/.aws/ credentials and {user. home}/.aws/config files for the region.
4. If running in EC2, check the EC2 metadata service for the region.

So, again, if we had the .aws credentials set, then this value came from there😉

And as you might already have guessed, yes, we can update that in the properties, too:

spring:
  cloud:
    aws:
      s3:
        region: us-east-1 #default is us-west-1

And at this point, when we rerun our application, then everything should be working, as expected.

AWS S3Client Operations

So, with all of that done, we can finally take a look at a few AWS S3Client capabilities. Of course, we will not cover all possible scenarios, so if you have a bit more specific use case, then I recommend checking the Spring Cloud AWS / AWS SDK documentation.

List All Buckets

Firstly, let’s get back to the listBuckets usage:

@GetMapping
fun listBuckets(): List<String> {
  val response = s3Client.listBuckets()

  return response.buckets()
    .mapIndexed { index, bucket ->
      "Bucket #${index + 1}: ${bucket.name()}"
    }
}

Long story short, this function returns ListBucketsResponse that contains a list of all buckets owned by us. It is worth mentioning that we must have the s3:ListAllMyBucket permission.

Nevertheless, I wanted to emphasize one, important thing.

AWS SDK methods throw exceptions quite heavily. For example, the above may result in SdkException, S3Exception, or SdkClientException to be thrown. In a production-ready code, we must keep that in mind, handle them according to our needs and (if necessary) translate to appropriate HTTP status codes.

Create New S3 Bucket

Following, let’s expose a POST /buckets endpoint that will be used to create new buckets:

@PostMapping
fun createBucket(@RequestBody request: BucketRequest) {
  val createBucketRequest = CreateBucketRequest.builder()
    .bucket(request.bucketName)
    .build()

  s3Client.createBucket(createBucketRequest)
}

data class BucketRequest(val bucketName: String)

This time our function looks quite different- we must prepare the CreateBucketRequest that we pass to the createBucket function.

And that is quite a common thing when dealing with AWS S3Client in Spring Boot. The SDK methods expect us to provide different objects of classes extending the S3Request, like CreateBucketRequest, DeleteObjectRequest, etc.

What the createBucket does is pretty obvious, but we must be cautious about the bucket name, because the function may throw BucketAlreadyExistsException, or BucketAlreadyOwnedByYouException.

Of course, to test that, the only thing we need to do is to hit the endpoint:

curl --location --request POST 'http://localhost:8080/buckets' \
--header 'Content-Type: application/json' \
--data-raw '{
    "bucketName": "your-awesome-name"
}'

And if everything is fine, a new bucket should be created.

Create Object In The Bucket

So at this point, we know how to create buckets in AWS. Nevertheless, we use them to organise uploaded files.

And it is the right time to learn how we can upload a file:

// we must add the typealias to avoid name clash for the @RequestBody annotation :) 
typealias PutObjectRequestBody = software.amazon.awssdk.core.sync.RequestBody

@RestController
@RequestMapping("/buckets")
class BucketController(
  private val s3Client: S3Client
) {

  @PostMapping("/{bucketName}/objects")
  fun createObject(@PathVariable bucketName: String, @RequestBody request: ObjectRequest) {
    val createObjectRequest = PutObjectRequest.builder()
      .bucket(bucketName)
      .key(request.objectName)
      .build()

    val fileContent = PutObjectRequestBody.fromString(request.content)

    s3Client.putObject(createObjectRequest, fileContent)
  }

  data class ObjectRequest(val objectName: String, val content: String)
}

As we can see, this time, we make use of the putObject and the PutObjectRequest (you see the pattern now 😉 ).

Moreover, when preparing the request we must specify both the bucket name and our object key.

curl --location --request POST 'http://localhost:8080/buckets/your-awesome-name/objects' \
--header 'Content-Type: application/json' \
--data-raw '{
    "objectName": "file-example.txt",
    "content": "My file content"
}'

As a result, a new text file named “file-example” with “My file content” in it should be created in the “your-awesome-name” bucket.

Of course, this is not the only method of S3Client that allows us to upload files, and sometimes the multipart upload might be a better choice for our use case.

List Objects From The Bucket

Nextly, let’s take a look what is the content of our bucket:

@GetMapping("/{bucketName}/objects")
fun listObjects(@PathVariable bucketName: String): List<String> {
  val listObjectsRequest = ListObjectsRequest.builder()
    .bucket(bucketName)
    .build()

  val response = s3Client.listObjects(listObjectsRequest)

  return response.contents()
    .map { s3Object -> s3Object.key() }
}

Similarly, we build the ListObjectsRequest instance, we perform the request using the listObjects and return an array with item names.

And this time, when we check with the following query:

curl --location --request GET 'http://localhost:8080/buckets/your-awesome-name/objects'

We should get the 200 OK with the array:

[
    "file-example.txt"
]

Fetch The Object From S3 Bucket

And although listing objects might be sometimes useful, I am pretty sure you would be more often interested in getting the actual object with a key:

@GetMapping("/{bucketName}/objects/{objectName}")
fun getObject(@PathVariable bucketName: String, @PathVariable objectName: String): String {
  val getObjectRequest = GetObjectRequest.builder()
    .bucket(bucketName)
    .key(objectName)
    .build()

  val response = s3Client.getObjectAsBytes(getObjectRequest)

  return response.asString(UTF_8)
}

Just like in the previous examples, we prepare the request with bucket name and object key, invoke the getObjectAsBytes and this time we print out the content to the output:

curl --location --request GET 'http://localhost:8080/buckets/your-awesome-name/objects/file-example.txt'

# Response: 
"My file content"

Of course, a friendly reminder that the AWS SDK throws the exceptions, and if the file does not exist, we will get the NoSuchKeyException.

Delete S3 Bucket

As the last step, let’s take a look at the logic necessary to delete a bucket:

@DeleteMapping("/{bucketName}")
fun deleteBucket(@PathVariable bucketName: String) {
  val listObjectsRequest = ListObjectsRequest.builder()
    .bucket(bucketName)
    .build()

  val listObjectsResponse = s3Client.listObjects(listObjectsRequest)

  val allObjectsIdentifiers = listObjectsResponse.contents()
    .map { s3Object ->
      ObjectIdentifier.builder()
        .key(s3Object.key())
        .build()
    }

  val del = Delete.builder()
    .objects(allObjectsIdentifiers)
    .build()

  val deleteObjectsRequest = DeleteObjectsRequest.builder()
    .bucket(bucketName)
    .delete(del)
    .build()

  s3Client.deleteObjects(deleteObjectsRequest)


  val deleteBucketRequest = DeleteBucketRequest.builder()
    .bucket(bucketName)
    .build()

  s3Client.deleteBucket(deleteBucketRequest)
}

As we can see, this time, we must perform our actions in a few steps.

Why?

The reason is simple:

Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: software.amazon.awssdk.services.s3.model.S3Exception: The bucket you tried to delete is not empty (Service: S3, Status Code: 409…

As we can see, the message above is pretty descriptive. Simply said- we cannot delete a bucket that is not empty.

So, our function utilizes the listObjects, so that we can get keys to delete, the deleteObjects to actually delete them, and deleteBucket to get rid of the bucket.

Summary

And that’s all for this first tutorial on how to integrate your Spring Boot application with AWS and make use of the S3Client to work with AWS S3. In the upcoming articles in the series, we will expand our knowledge to work with S3 even more efficiently.

Of course, I highly encourage you to take a look at the remaining articles in this series:

• #3 Test Spring Boot AWS S3 with Localstack and Testcontainers

For the source code, as always, please refer to this GitHub repository.

The post Spring Boot with AWS S3, S3Client and Kotlin appeared first on Codersee blog- Kotlin on the backend.