Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-firehose]

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift.

Firehose is part of the Amazon Kinesis streaming data family, along with Amazon Kinesis Streams. With Firehose, you do not need to write any applications or manage any resources.

You configure your data producers to send data to Firehose and it automatically delivers the data to the destination that you specified.

Links:

  1. Amazon Kinesis - Site

  2. Amazon Kinesis - Documentation

618 questions
143
votes
10 answers

Append data to an S3 object

Let's say that I have a machine that I want to be able to write to a certain log file stored on an S3 bucket. So, the machine needs to have writing abilities to that bucket, but, I don't want it to have the ability to overwrite or delete any files…
Theodore
  • 1,683
  • 2
  • 12
  • 13
33
votes
13 answers

Reading the data written to s3 by Amazon Kinesis Firehose stream

I am writing record to Kinesis Firehose stream that is eventually written to a S3 file by Amazon Kinesis Firehose. My record object looks like ItemPurchase { String personId, String itemId } The data is written to S3 looks…
learner_21
  • 593
  • 1
  • 5
  • 11
32
votes
3 answers

Write parquet from AWS Kinesis firehose to AWS S3

I would like to ingest data into S3 from Kinesis Firehose formatted as parquet. So far I have just find a solution that implies creating an EMR, but I am looking for something cheaper and faster like store the received JSON as parquet directly from…
bracana
  • 1,003
  • 2
  • 11
  • 20
28
votes
2 answers

Is there a way to manually set an ElasticSearch document id when inserting via AWS Kinesis Firehose?

I have an AWS Kinesis Firehose Stream set up to feed data into an AWS ElasticSearch cluster, and I can successfully insert documents by sending them to the Firehose Stream, which loads them into ElasticSearch. But I would like to be able to manually…
Alex Coleman
  • 601
  • 7
  • 18
21
votes
2 answers

Error creating API Gateway Integration Response: NotFoundException: Invalid Integration identifier specified

Objective Solution or workaround for the problem. Problem The Terraform API Gateway integration with Firehose below works if Firehose is created separately in advance. resource "aws_api_gateway_integration" "click_put" { rest_api_id =…
mon
  • 18,789
  • 22
  • 112
  • 205
21
votes
4 answers

Can I automatically append newlines to AWS Firehose records?

I am trying to configure a Kinesis Analytics application with the following settings: Input stream is a Kinesis Firehose which is taking stringified JSON values The SQL is a simple passthrough (it needs to be more complicated later but for testing,…
MrHen
  • 2,420
  • 2
  • 25
  • 39
20
votes
6 answers

Can I customize partitioning in Kinesis Firehose before delivering to S3?

I have a Firehose stream that is intended to ingest millions of events from different sources and of different event-types. The stream should deliver all data to one S3 bucket as a store of raw\unaltered data. I was thinking of partitioning this…
mowienay
  • 1,264
  • 4
  • 19
  • 32
20
votes
5 answers

AWS Kinesis Firehose not inserting data in Redshift

I try to have a Kinesis Firehose pushing data in a Redshift table. The firehose stream is working and putting data in S3. But nothing arrive in the destination table in Redshift. In the metrics DeliveryToRedshift Success is 0 (DeliveryToRedshift…
mathieu
  • 2,330
  • 2
  • 24
  • 44
18
votes
4 answers

Kinesis Firehose putting JSON objects in S3 without seperator comma

Before sending the data I am using JSON.stringify to the data and it looks like this {"data": [{"key1": value1, "key2": value2}, {"key1": value1, "key2": value2}]} But once it passes through AWS API Gateway and Kinesis Firehose puts it to S3 it…
sjishan
  • 3,392
  • 9
  • 29
  • 53
17
votes
1 answer

java.lang.ClassNotFoundException: com.amazonaws.ClientConfigurationFactory

I am using aws since last 6 months and I developed application that puts batch request to firehose. It was working fine till today but when I redeployed in my local system it is saying java.lang.ClassNotFoundException:…
Yubaraj
  • 3,800
  • 7
  • 39
  • 57
14
votes
4 answers

Concatenate s3 files when using AWS Firehose

I have an AWS Kinesis Firehose stream putting data in s3 with the following config: S3 buffer size (MB)* 2 S3 buffer interval (sec)* 60 Everything works fine. The only problem is that Firehose creates one s3 file for every chunk of data. (In…
Benjamin Crouzier
  • 40,265
  • 44
  • 171
  • 236
12
votes
2 answers

Multiple Destinations for Kinesis

Can we have multiple destinations from single Kinesis Firehose? I saw this picture From this, it looks like it is possible to add s3, redshift and elastic search from single firehose. I exactly want to do this. But when I do it from aws console,…
hatellla
  • 4,796
  • 8
  • 49
  • 101
11
votes
1 answer

cannot copy json - Dynamo db Streams to redshift

Following is the use case i am working on: I have configured enable Streams when creating DynamoDB with new and old Image.I have created a Kinesis Firehose delivery stream with Destination as Redshift(Intermediate s3). From Dynamodb my stream…
Prasanna Nandakumar
  • 4,295
  • 34
  • 63
11
votes
6 answers

Partition Kinesis firehose S3 records by event time

Firehose->S3 uses the current date as a prefix for creating keys in S3. So this partitions the data by the time the record is written. My firehose stream contains events which have a specific event time. Is there a way to create S3 keys containing…
bibac
  • 449
  • 8
  • 17
10
votes
3 answers

What is the difference between AWS Elastic MapReduce and AWS Kinesis Data Analytics?

I'm executing a Flink Job with this tools. I think both can do exactly the same with the proper configuration. Does Kinesis Data Analytics do something that EMR can not do or vice versa? Amazon Kinesis Data Analytics is the easiest way to analyze…
Turvo
  • 211
  • 2
  • 7
1
2 3
…
41 42