Is it any way to append new messages from Kafka to the same file in the HDFS using Gobblin? Now it creates a new file every time reading from Kafka. If run Gobblin job every minute for example,it will be the plenty of files.
Please help!
Asked
Active
Viewed 336 times
1 Answers
0
Not sure of Gobblin, but IMHO Apache Flume will be a better choice for you.
Sudev Ambadi
- 655
- 6
- 12
-
As I know, Flume cannot append to the same file as well. – Kateryna Khotkevych Aug 19 '16 at 17:30
-
1. Writing files into same folder, MR/Spark can read all files within a folder given path to folder. – Sudev Ambadi Aug 20 '16 at 07:56
-
Flume won't create a billion files if you configure properly it can roll events into same file. check this out. http://stackoverflow.com/questions/26169921/flume-hdfs-sink-generates-lots-of-tiny-files-on-hdfs – Sudev Ambadi Aug 20 '16 at 12:20