Stack Overflow
Stack Overflow

Questions tagged [kafka-rest]

Provides a RESTful interface to a Kafka cluster, so you can produce and consume messages, as well as perform administrative actions through any REST client.

Confluent REST Proxy

(From the official Readme.md linked below)

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients. Examples of use cases include reporting data to Kafka from any frontend app built in any language, ingesting messages into a stream processing framework that doesn't yet support Kafka, and scripting administrative actions.

It is a Confluent open-source project, included as part of their Confluent Platform.

Resources:

Introduction: https://github.com/confluentinc/kafka-rest/blob/master/README.md

Github Repository: https://github.com/confluentinc/kafka-rest

Complete documentation: https://docs.confluent.io/current/kafka-rest/docs/

116 questions
7
votes
2 answers

Ensuring that all messages have been read from Kafka topic using REST Proxy

I'm new to Kafka, and our team is investigating patterns for inter-service communication. The goal We have two services, P (Producer) and C (Consumer). P is the source of truth for a set of data that C needs. When C starts up it needs to load all of…
TrueWill
  • 25,132
  • 10
  • 101
  • 150
6
votes
1 answer

Kafka Confluent REST API: Kafka Included?

I have an exisiting Kafka Cluster. I want to install the Kafka REST Proxy: https://github.com/confluentinc/kafka-rest If I install confluent does that come with Kafka? I am afraid if I still it on my master Kafka node confluent will override all my…
wwjdm
  • 2,568
  • 7
  • 32
  • 61
5
votes
1 answer

how to properly register Protobuf schema with Schema Registry / Kafka-Rest

I am trying to post a Protobuf schema to the Schema Registry using kafka-rest interface: curl -X POST -H "Content-Type: application/vnd.kafka.protobuf.v2+json" \ -H "Accept: application/vnd.kafka.v2+json" \ --data '{"value_schema":…
Eugene Goldberg
  • 14,286
  • 20
  • 94
  • 167
5
votes
1 answer

Is there a way to produce Kafka messages with headers using Kafka Confluent REST API?

I'm trying to send messages to Kafka with some custom headers, but I could't find a way to do it. According to the documatation, The POST /topics/(string: topic_name) API does not have a way to attach custom headers to messages. Does anyone know a…
ArmWich
  • 103
  • 1
  • 7
4
votes
2 answers

Reading only one message from the topic using REST Proxy

I use Kafka version 2.2.0cp2 through Rest Proxy (in the Docker container). I need the consumer to always read only one message. I set the value max.poll.records=1 in the file /etc/kafka/consumer.properties as follows: consumer.max.poll.records=1 OR:…
Daniel Grave
  • 41
  • 2
4
votes
0 answers

Can I send message header while using kafka-rest proxy api?

I want to send message header for kafka message using kafka-rest proxy api. https://docs.confluent.io/current/kafka-rest/docs/api.html How can I do this?
Amit Gurav
  • 41
  • 2
4
votes
2 answers

Confluent's Kafka REST Proxy vs Kafka Client

I am curious about the advantages and disadvantages of Confluent's Kafka REST Proxy and the producer/consumer implemented with the kafka official client library. i know that Confluent's Kafka REST Proxy is used for administrative tasks and for…
김태우
  • 1,033
  • 1
  • 12
  • 29
3
votes
1 answer

What is the difference between confluent kafka-rest & Strimzi Kafka Bridge

I would like to use a HTTP proxy on top of Kafka. I see two projects with same purpose : https://github.com/confluentinc/kafka-rest https://github.com/strimzi/strimzi-kafka-bridge I use strimzi operator to spin up Kafka on Kubernetes. Both are…
Sairam Krish
  • 10,158
  • 3
  • 55
  • 67
3
votes
2 answers

Alternative of Confluent REST Proxy

We have some applications which want to communicate with Kafka using REST API calls to both consume and produce messages. If we do not want to use Confluent REST Proxy, what are the options ?
Anirban
  • 257
  • 4
  • 12
3
votes
2 answers

Can Kafka consumers consume from all partitions

I have a multi-partition topic that is consumed by multiple consumers(same group). My goal is to maximize the consuming processing, i.e. any consumer can consume messages from any partitions. I know that it looks impossible as only one consumer can…
zwush
  • 75
  • 2
  • 9
3
votes
2 answers

How do you set Kafka Rest Proxy Key Schema with JSON values?

I'm using Kafka 1.1 and Kafka Rest Proxy 4.1.2. I have been keying records from inside Kafka streams using String keys. I want to use Rest Proxy to insert records to be joined, but the keys are getting escaped quotation marks put around them. I'm…
OldDave2019
  • 33
  • 1
  • 3
3
votes
2 answers

Kafka - how to transform Rest service message to ingest into a kafka topic?

I'm new at kafka community and i'm facing a challenging question. I have two applications that talk each other via Rest webservice, with the body being a json message. How can I use kafka to be the middleware between these two application, with…
Higor
  • 33
  • 5
3
votes
0 answers

how to configure load balancer stickiness for Confluent Rest-Proxy

I have a setup of 3 Rest-Proxy(Kafka-rest) instances in Kubernetes. https://docs.confluent.io/current/kafka-rest/docs/index.html https://github.com/confluentinc/kafka-rest I have added a NetScaler Ingress on top of the deployment. I see in Citrix…
Ashu
  • 367
  • 5
  • 14
3
votes
1 answer

What are the benefits of the Kafka REST Proxy API?

I do not know the advantages of the Kafka REST Proxy API. It's a REST API, so I know it's handy for administration. Why do people use the Kafka REST Proxy API? Is it burdensome to add a Maven dependency on a producer or a consumer? Also, I know that…
김태우
  • 1,033
  • 1
  • 12
  • 29
2
votes
2 answers

not able to create Kafka topic with Kafka Rest Proxy - HTTP 415 Unsupported Media Type javax.ws.rs.NotSupportedException

I am following this documentation. I am able to get cluster's information like this: curl -sk -X GET "https://xx.xx.xx.xx:8443/v3/clusters/" the previous request works fine. However, when I try to create a topic I get HTTP 415 Unsupported Media…
roAl
  • 173
  • 1
  • 1
  • 16
1
2 3 4 5 6 7 8