5

I am using Flume 1.3.1 ng, I am transfering a file from spoolDir to HDFS Sink and I need outpufile name same as input file. For example if input file name is sample.gz then output also need as sample.gz

Could you please provide the procedure how can achieve this functionality in Flume 1.3.1 ng.

kasi
  • 51
  • 2

2 Answers2

3

I am using following settings:

a1.sources.r1.basenameHeader = true
a1.sinks.k1.hdfs.filePrefix = %{basename}

These settings give me following transformation:

original_file.txt -> original_file.1457085474015.txt

And sadly seems you can't remove timestamp from filename :C

Samriang
  • 403
  • 6
  • 18
0

There is a parameter for spooldir about whether or not adding a header regarding the original file name. It is called fileHeader, and by default it is set to false, so set it ti true:

agentname.sources.mysource.type = spooldir
agentname.sources.mysource.spoolDir = /path/to/the/dir
agentname.soruces.mysource.fileHeader = true
frb
  • 3,738
  • 2
  • 21
  • 51
  • 1
    I added following two lines in the agent configuration agent1.soruces.spoolDirSource.fileHeader =true agent1.sinks.hdfsSink.hdfs.filePrefix = %{file} output file name I got as .1429872137679.gz. I didn't get same name as input file – kasi Apr 24 '15 at 11:47