Apache Pulsar disk usage in a container

Question

I am running Apache pulsar in a Docker container in standalone mode.

I am publishing log file lines into the pulsar topic. The total logs are about 60G in size.

The pulsar container keeps running out of disk space on the / disk and has to terminate.

It is a default configuration. The docker-compose file looks as follows

  pulsar:
    image: apachepulsar/pulsar
    container_name: pulsar
    command: bin/pulsar standalone
    ports:
        - 6650:6650
        - 8080:8080

Is there a way to limit pulsar disk usage so that it does not run out of all the space ?

set a volume to outside the container? – Aaron_ab Dec 02 '21 at 12:32 — Aaron_ab, Dec 02 '21 at 12:32

score 3 · Answer 1 · answered Jun 24 '20 at 21:57

Apache Pulsar allows to specify retention policies and expiry for a namespace.

To specify a retention policy, you can use the Pulsar admin CLI:

bin/pulsar-admin namespaces set-retention my-tenant/my-namespace --size 10G --time 1d

Another way is to execute a POST request to the Pulsar REST API:

localhost:8080/admin/v2/namespaces/my-tenant/my-namespace/retention

where the body has the following JSON structure:

{
    "retentionTimeInMinutes": 1440,
    "retentionSizeInMB": 10240
}

Update the commands above with your values for my-tenant, my-namespace, localhost

Apache Pulsar disk usage in a container

1 Answers1