5

My use-case is as follows: I have JSON data coming in which needs to be stored in S3 in parquet format. So far so good, I can create a schema in Glue and attach a "DataFormatConversionConfiguration" to my firehose stream. BUT the data is coming from different "topics". Each topic has a particular "schema". From my understanding I will have to create multiple firehose streams, as one stream can only have one schema. But I have thousands of such topics with very high volume high throughput data incoming. It does not look feasible to create so many firehose resources (https://docs.aws.amazon.com/firehose/latest/dev/limits.html)

How should I go about building my pipeline.

Dexter
  • 1,710
  • 2
  • 17
  • 34

1 Answers1

4

IMO you can:

  • ask for upgrade of your Firehose limit and do everything with 1 Firehose/stream + add Lambda transformation to convert the data into a common schema - IMO not cost-efficient but you should see with your load.

  • create a Lambda for each Kinesis data stream, convert each event to the schema managed by a single Firehose and at the end can send the events directly to your Firehose stream with Firehose API https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecord.html (see "Q: How do I add data to my Amazon Kinesis Data Firehose delivery stream?" here https://aws.amazon.com/kinesis/data-firehose/faqs/) - but also, check the costs before because even though your Lambdas are invoked "on demand", you may have a lot of them invoked during long period of time.

  • use one of data processing frameworks (Apache Spark, Apache Flink, ...) and read your data from Kinesis in batches of 1 hour, starting every time when you terminated last time --> use available sinks to convert the data and write it in Parquet format. The frameworks use the notion of checkpointing and store the last processed offset in an external storage. Now, if you restart them every hour, they will start to read the data directly from the last seen entry. - It may be cost-efficient, especially if you consider using spot instances. On the other side, it requires more coding that 2 previous solutions and obviously may have higher latency.

Hope that helps. You could please give a feedback about chosen solution ?

Bartosz Konieczny
  • 1,985
  • 12
  • 27
  • 1
    Hey, we went ahead with a solution using Flink and Kinesis. We are running it on kinesis streams. Managed to use a single flink sink that dynamically detects new schemas and topics by adding a few custom extensions to the code :) – Dexter May 25 '20 at 11:11
  • @Dexter Did you use Kinesis Data Streams or Kinesis Data Analytics with Flink? I have exactly the same problem as your question. Also how was the errors signified, as I found with Firehose if there was some bad validation, it didn't give a proper error message. – mp252 Oct 22 '20 at 07:33