1

I want to add data to Kinesis Stream by Hitting an HTTP Endpoint, specifying some headers and data.

I know its possible via API Gateway -> AWS Service Proxy -> Kinesis Stream. I dont want to use API Gateway.

Also I dont want to use the KPL Library.

The task is to provide an Endpoint to a Tibco Service and Tibco Service is going to ingest data into the stream.

Architecture Changes are welcome.!

Rudresh Ajgaonkar
  • 779
  • 1
  • 10
  • 27
  • So you don't want to use API Gateway because? If you don't want to use the AWS service designed for this sort of thing then you'll just have to build out an API yourself and run it somewhere else like an EC2 instance. What is your question exactly? – Mark B Nov 16 '17 at 16:41
  • Thanks for the reply mark. Is there any way that i can add records into a stream by hitting direct Kinesis endpoint [If there is any?]. let me know if i make sense when i say that. I want a enpoint like : http://someendpoint.kinesis.aws.com and pass headers to the endpoint and required data that i want to put in the stream. – Rudresh Ajgaonkar Nov 16 '17 at 16:46
  • All i think i want is to do the following: From a EC2 Instance do a *curl https:....* and pass some to the stream. Sorry for the silly query. – Rudresh Ajgaonkar Nov 16 '17 at 16:51
  • 2
    If your code is running on an EC2 instance, why not use the AWS CLI tool instead of `curl`? With something like curl you will have to run against the raw AWS API and sign all your requests with Sigv4, which is a huge pain. That's what's the CLI tool is going to do for you. It will also automatically use the EC2 instance profile. – Mark B Nov 16 '17 at 16:55
  • @MarkB https://stackoverflow.com/questions/39939225/call-rest-api-for-amazon-kinesis-with-setting-up-api-gateway I was looking for something in this direction. – Rudresh Ajgaonkar Nov 16 '17 at 17:02
  • 2
    That question is basically an exact duplicate of your question, and I would answer it the same way I've answered your question. You can go down that path and have to deal with writing the code to sign the request yourself, or you can use the AWS CLI Tool or the AWS SDK to handle that for you. In the end it's still just making an HTTPS request against the AWS API, but it handles the authentication for you. – Mark B Nov 16 '17 at 17:05

1 Answers1

2

The Kinesis API endpoints is what you're looking for.

However, handling the authentication and signing of the requests is going to be much work, which is already done for you in the AWS CLI and SDKs.

If possible, use those to submit your Kinesis data.

If this is not possible, you will need to create your own endpoint to receive the more "basic" request, and convert that into an AWS authenticated request. That's either done using the API gateway (which you state you do not want to do), or you host your own web server on an EC2 instance.

AWS has provided many tools to make things easy for you. Either use them, or roll your own.

Matt Houser
  • 33,983
  • 6
  • 70
  • 88
  • 1
    Well ... the OP could well argue that rather than "to make things easy for you" the design of AWS' micro-granular service offerings is to enable charging several times for each solution to a single user-interaction and to promote lock-in to their ecosystem by convincing customers to tie their applications to the specifics of AWS. Perhaps the OP wants to use the benefits of Kinesis stream processing without buying every product in the AWS store. – Dave Oct 22 '19 at 03:03