Configuring Minio server for use with Testcontainers

Question

My application uses Minio for S3-compatible object storage, and I'd like to use the Minio docker image in my integration tests via Testcontainers.

For some very basic tests, I run a GenericContainer using the minio/minio docker image and no configuration except MINIO_ACCESS_KEY and MINIO_SECRET_KEY. My tests then use Minio's Java Client SDK. These work fine and behave just like expected.

But for other integration tests, I need to set up separate users in Mino. As far as I can see, users can only be added to Minio using the Admin API, for which there is no Java client, only the minio/mc docker image (the mc CLI is not available in the minio/minio docker image used for the server).

On the command line, I can use the Admin API like this:

$ docker run --interactive --tty --detach --entrypoint=/bin/sh --name minio_admin minio/mc

The --interactive --tty is a bit of a hack to keep the container running so I can later run commands like this one:

$ docker exec --interactive --tty minio_admin mc admin user add ...

Using Testcontainers, I try to do the same like this:

public void testAdminApi() throws Exception {
    GenericContainer mc = new GenericContainer("minio/mc")
            .withCommand("/bin/sh")
            .withCreateContainerCmdModifier(new Consumer<CreateContainerCmd>() {
                @Override
                public void accept(CreateContainerCmd cmd) {
                    cmd
                            .withAttachStdin(true)
                            .withStdinOpen(true)
                            .withTty(true);
                }
            });

    mc.start();
    log.info("mc is running: {}", mc.isRunning());

    String command = "mc";
    Container.ExecResult result = mc.execInContainer(command);
    log.info("Executing command '{}' returned exit code '{}' and stdout '{}'", command, result.getExitCode(), result.getStdout());

    assertEquals(0, result.getExitCode());
}

The logs show the container being started, but executing a command against it returns exit code 126 and claims it's in a stopped state:

[minio/mc:latest] - Starting container with ID: 4f96fc7583fe62290925472c4c6b329fbeb7a55b38a3c0ad41ee797db1431841
[minio/mc:latest] - Container minio/mc:latest is starting: 4f96fc7583fe62290925472c4c6b329fbeb7a55b38a3c0ad41ee797db1431841
[minio/mc:latest] - Container minio/mc:latest started
minio.MinioAdminTests - mc is running: true
org.testcontainers.containers.ExecInContainerPattern - /kind_volhard: Running "exec" command: mc
minio.MinioAdminTests - Executing command 'mc' returned exit code '126'
  and stdout 'cannot exec in a stopped state: unknown'

java.lang.AssertionError: Expected: 0, Actual: 126

After fiddling around with this for hours, I'm running out of ideas. Can anyone help?

score 7 · Answer 1 · answered Apr 02 '19 at 19:51

Thanks to @glebsts and @bsideup I was able to get my integration tests to work. Here's a minimal example of how to add a user:

public class MinioIntegrationTest {

    private static final String ADMIN_ACCESS_KEY = "admin";
    private static final String ADMIN_SECRET_KEY = "12345678";
    private static final String USER_ACCESS_KEY = "bob";
    private static final String USER_SECRET_KEY = "87654321";

    private static GenericContainer minioServer;
    private static String minioServerUrl;

    @BeforeAll
    static void setUp() throws Exception {
        int port = 9000;
        minioServer = new GenericContainer("minio/minio")
                .withEnv("MINIO_ACCESS_KEY", ADMIN_ACCESS_KEY)
                .withEnv("MINIO_SECRET_KEY", ADMIN_SECRET_KEY)
                .withCommand("server /data")
                .withExposedPorts(port)
                .waitingFor(new HttpWaitStrategy()
                        .forPath("/minio/health/ready")
                        .forPort(port)
                        .withStartupTimeout(Duration.ofSeconds(10)));
        minioServer.start();

        Integer mappedPort = minioServer.getFirstMappedPort();
        Testcontainers.exposeHostPorts(mappedPort);
        minioServerUrl = String.format("http://%s:%s", minioServer.getContainerIpAddress(), mappedPort);

        // Minio Java SDK uses s3v4 protocol by default, need to specify explicitly for mc
        String cmdTpl = "mc config host add myminio http://host.testcontainers.internal:%s %s %s --api s3v4 && "
                + "mc admin user add myminio %s %s readwrite";
        String cmd = String.format(cmdTpl, mappedPort, ADMIN_ACCESS_KEY, ADMIN_SECRET_KEY, USER_ACCESS_KEY, USER_SECRET_KEY);

        GenericContainer mcContainer = new GenericContainer<>("minio/mc")
                .withStartupCheckStrategy(new OneShotStartupCheckStrategy())
                .withCreateContainerCmdModifier(containerCommand -> containerCommand
                        .withTty(true)
                        .withEntrypoint("/bin/sh", "-c", cmd));
        mcContainer.start();
    }

    @Test
    public void canCreateBucketWithAdminUser() throws Exception {
        MinioClient client = new MinioClient(minioServerUrl, ADMIN_ACCESS_KEY, ADMIN_SECRET_KEY);
        client.ignoreCertCheck();

        String bucketName = "foo";
        client.makeBucket(bucketName);
        assertTrue(client.bucketExists(bucketName));
    }

    @Test
    public void canCreateBucketWithNonAdminUser() throws Exception {
        MinioClient client = new MinioClient(minioServerUrl, USER_ACCESS_KEY, USER_SECRET_KEY);
        client.ignoreCertCheck();

        String bucketName = "bar";
        client.makeBucket(bucketName);
        assertTrue(client.bucketExists(bucketName));
    }

    @AfterAll
    static void shutDown() {
        if (minioServer.isRunning()) {
            minioServer.stop();
        }
    }
}

* @BeforeAll/!AfterAll makes no sense because testcontainers are nicely working with @ClassRule and you don't need manual cleanup * overwriting entrypoint - doubtful, all done because of gluing together giant superstring; * copying hardcoded constant from genericContainer is doubtful — glebsts, Apr 03 '19 at 07:52
* why would you actually need to expose port? [Networking documentation](https://github.com/testcontainers/testcontainers-java/blob/master/docs/features/networking.md) describes cases where you might need it, but here you have both _minio_ and _mc_ running in containers, putting them into same network would isolate them better, and you don't need to rely on text url when you can have internal IP as well; * your minio is not cleaned up between tests, env is dirty. That may lead to test inter-dependencies, or too much pain i.e. for adding test which require no user being added — glebsts, Apr 03 '19 at 07:53

bsideup · Answer 2 · 2019-03-29T15:38:46.143

4

You could run an one-off container (use OneShotStartupCheckStrategy) with mc and withCommand("your command"), connected to the same network as the minio server you're running (see Networking).

edited Mar 29 '19 at 15:38

answered Mar 29 '19 at 14:50

bsideup

2,845
17
21

glebsts · Answer 3 · 2019-04-01T14:58:05.827

As @bsideup suggested, you can use one-shot strategy, i.e. as in here. UPD: added working test. Here is important to know that

When the container is launched, it executes entrypoint + command (this is Docker in general and has nothing to do with Testcontainers). Source from TC github

public class TempTest {
    @Rule
    public Network network = Network.newNetwork();

    private String runMcCommand(String cmd) throws TimeoutException {
        GenericContainer container = new GenericContainer<>("minio/mc")
                .withCommand(cmd)
                .withNetwork(network)
                .withStartupCheckStrategy(new OneShotStartupCheckStrategy())
                .withCreateContainerCmdModifier(command -> command.withTty(true));
        container.start();
        WaitingConsumer waitingConsumer = new WaitingConsumer();
        ToStringConsumer toStringConsumer = new ToStringConsumer();
        Consumer<OutputFrame> composedConsumer = toStringConsumer.andThen(waitingConsumer);
        container.followOutput(composedConsumer);
        waitingConsumer.waitUntilEnd(4, TimeUnit.SECONDS);
        return toStringConsumer.toUtf8String();
    }

    private void showCommandOutput(String cmd) throws TimeoutException {
        String res = runMcCommand(cmd);
        System.out.printf("Cmd '%s' result:\n----\n%s\n----%n", cmd, res);
    }

    @Test
    public void testAdminApi() throws Exception {
        showCommandOutput("ls");
        showCommandOutput("version");
    }
}

Another option is to use content of dockerfile of minio/mc, which is small, modify executed command (one-off "mc" by default), and run own container once per test, which, compared to one-off container, will save some time if you need to execute multiple commands:

@Rule
public Network network = Network.newNetwork();

@Rule
public GenericContainer mc = new GenericContainer(new ImageFromDockerfile()
  .withDockerfileFromBuilder(builder ->
    builder
      .from("alpine:3.7")
      .run("apk add --no-cache ca-certificates && apk add --no-cache --virtual .build-deps curl && curl https://dl.minio.io/client/mc/release/linux-amd64/mc > /usr/bin/mc && chmod +x /usr/bin/mc && apk del .build-deps")
      .cmd("/bin/sh", "-c", "while sleep 3600; do :; done")
      .build())
    )
  .withNetwork(network);

public void myTest() {
  mc.execInContainer("mc blah");
  mc.execInContainer("mc foo");
}

Basically, it runs image with mc installed, and sleeps for 1h which is enough for your tests. While it runs, you can execute commands etc. After you finish, it is killed. Your minio container can be in same network.

Basically this "sleep" is same that you do by leaving container detached and running. — glebsts, Apr 01 '19 at 08:12
added example with one-off containers. Works on my machine(tm) — glebsts, Apr 01 '19 at 14:58

GnanaJeyam · Answer 4 · 2022-08-12T05:56:06.723

Minio with docker compose:

For those who are looking for s3 with minio object server integration test.
The current implementation is based on docker-compose.
The current implementation utilising AWS S3 client for CURD opertations

docker-compose file:

version: '3.7'
services:
  minio-service:
    image: quay.io/minio/minio
    command: minio server /data
    ports:
      - "9000:9000"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123

The actual IntegrationTest class:

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.S3Object;
import org.junit.jupiter.api.*;
import org.testcontainers.containers.DockerComposeContainer;

import java.io.File;

@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class MinioIntegrationTest {

    private static final DockerComposeContainer minioContainer = new DockerComposeContainer<>(new File("src/test/resources/docker-compose.yml"))
            .withExposedService("minio-service", 9000);
    private static final String MINIO_ENDPOINT = "http://localhost:9000";
    private static final String ACCESS_KEY = "minio";
    private static final String SECRET_KEY = "minio123";
    private AmazonS3 s3Client;

    @BeforeAll
    void setupMinio() {
        minioContainer.start();
        initializeS3Client();
    }

    @AfterAll
    void closeMinio() {
        minioContainer.close();
    }

    private void initializeS3Client() {
        String name = Regions.US_EAST_1.getName();
        AwsClientBuilder.EndpointConfiguration endpoint = new AwsClientBuilder.EndpointConfiguration(MINIO_ENDPOINT, name);
         s3Client = AmazonS3ClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY)))
                .withEndpointConfiguration(endpoint)
                .withPathStyleAccessEnabled(true)
                .build();
    }

    @Test
    void shouldReturnActualContentBasedOnBucketName() throws Exception{
        String bucketName = "test-bucket";
        String key = "s3-test";
        String content = "Minio Integration test";
        s3Client.createBucket(bucketName);
        s3Client.putObject(bucketName, key, content);
        S3Object object = s3Client.getObject(bucketName, key);
        byte[] actualContent = new byte[22];
        object.getObjectContent().read(actualContent);
        Assertions.assertEquals(content, new String(actualContent));
    }
}

i like this answer a lot because it uses the s3 client not the minio java sdk — Nicholas DiPiazza, Aug 11 '22 at 21:31

Configuring Minio server for use with Testcontainers

4 Answers4