Access to amazon msk cluster locally

Question

I have an amazon msk cluster (kafka) in a vpc and I can access it through a client (EC2 instance) that is in the same vpc. my question is .. can i access my cluster from a client that is on my local computer? How can I do this configuration? I tried changing the routing tables of the vpc and the sub networks and still can not.

Answer 1

This is not possible directly. As per the MSK FAQ

Yes, the only way data can be produced and consumed from an Amazon MSK cluster is over a private connection between your clients in your VPC and the Amazon MSK cluster. Amazon MSK does not support public endpoints.

Instead you will either need a VPN connection inbound to your VPC or a bastion host to proxy traffic from your local host to the MSK.

Answer 2

I had the same problem and I found out that the answer in this article works. Basically you look for the ENIs of your cluster, which should be five - 3 for each zookeeper node and 2 for each broker. They should come with a description similar to: DO NOT DELETE - Amazon MSK network interface for cluster arn:aws:kafka:....

You create an Elastic IP for each of them, which might mean asking the support to up your limit as I think the standard EIP limit per account per region is 5 or 6 and you probably use some elsewhere. Then from the ENI interface in the EC2 console you associate an EIP with each MSK ENI.

I had a problem with the URLs though and editing my hosts file didn't seem to work. The URLs of the brokers and zookeeper were still resolved to the private IPs, so I just ended up using the public IPs directly.

Edit: Okay, it turns out I was mistaken. You have to edit your hosts file at least for the bootstrap servers. I thought I removed them from my hosts file and tested and it worked but they were probably cached. So you need to map the public EIPs to the bootstrap servers and you need to map the correct IP to the correct URL, otherwise you'll get a metadata error. For some reason, though, zookeeper seems to work without mapping its public IPs to its URLs - just by using the public IPs. Whereas without mapping the public IPs to the bootstrap server URLs, it seems that the public IP is transformed into the URL which is then resolved to its private IP.

Answer 3

Amazon MSK now supports public access. You can use this feature to access your cluster anywhere from the open internet. There are certain per-requists like having encryption turned on and Client-Broker communication being enabled. Please see details here: https://docs.aws.amazon.com/msk/latest/developerguide/public-access.html

Answer 4

1. Create public end point . Make sure plain text access is disabled.Security & NACL are enabled for your source location.
2. Go to window local machine
3. set the profile in .aws/config & .aws/credential
    in config-
      [default]
      region = ap-northeast-1
      output = json

      [profile <Your profile name>]
      region = ap-northeast-1
      output = json
    
    in credential file - Create a user in AWS and give the required MSK permission.
            Generate secret id & secret key and set in below file
         [<Your profile name>]
         aws_access_key_id = <AWS USER ACCESS KEY ID>
         aws_secret_access_key = <AWS SECRET ACCESS KEY>

  4. Now go to CMD prompt of window and set the below variable
     set AWS_PROFILE=<Your profile name>

  5. Create the client.properties file in config folder of kafka and put below values
ssl.truststore.location=<PATH_TO_TRUST_STORE_FILE>
security.protocol=SASL_SSL 
sasl.mechanism=AWS_MSK_IAM 
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
  
  5. Now get public end point of broker on port 9198. Ready to execute public end point outside of VPC.
/kafka-console-producer.sh --bootstrap-server <public broker servers on enter code here`ort 9198> --topic <Your topic name> --producer.config config/client.properties