Only allowing consumers from certain hosts/IPs access to kafka topics

Question

We have a Kafka-based system, and a large team of devs who are working on this system. We are still in development and testing phase, and have not gone to production yet. For majority of local dev testing, we do not need to consume Kafka messages, but we do need to interact with databases and other applications in the QA environment, so devs will use the QA profile properties when running their application (these are Java Spring Boot apps).

The problem is when a dev runs an application locally with a QA profile, the local app is consuming messages from the QA environment and disrupting testing. We have a larger concern too that when we go to production, a dev might run with a production profile to be able to look at some production data, and unintentionally consume production messages, causing significant problems.

Is there a way to configure Kafka to only allow certain hosts/IPs to consume it's messages?

score 2 · Answer 1 · answered May 30 '20 at 12:14

You need to configure Authorisation using ACL.

How to enable ACL:

In your server.properties file, you need to create an Authorizer by adding the following line:

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

Now you need to follow the docs in order to properly configure ACL based on your use cases.

Adding ACLs

Now once everything is in place, let's assume you have a topic called testTopic to which you want to grant read and write access only to user called Bob from a host with IP 197.5.6.1:

bin/kafka-acls \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --add \
  --allow-principal User:'Bob' --allow-host '197.5.6.1' \
  --operation Read --operation Write \
  --topic testTopic \

Thanks! We're going to try this out, will report back how it goes — theckler, Jun 01 '20 at 18:13

Only allowing consumers from certain hosts/IPs access to kafka topics

1 Answers1