How to set log filename in flume

Question

I am using Apache flume for log collection. This is my config file

httpagent.sources = http-source
httpagent.sinks = local-file-sink
httpagent.channels = ch3

#Define source properties 

httpagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
httpagent.sources.http-source.channels = ch3
httpagent.sources.http-source.port = 8082


# Local File Sink

httpagent.sinks.local-file-sink.type = file_roll
httpagent.sinks.local-file-sink.channel = ch3
httpagent.sinks.local-file-sink.sink.directory = /home/avinash/log_dir
httpagent.sinks.local-file-sink.sink.rollInterval = 21600

# Channels

httpagent.channels.ch3.type = memory
httpagent.channels.ch3.capacity = 1000

My application is working fine.My problem is that in the log_dir the files are using some random number (I guess its timestamp) timestamp as by default.

How to give a proper filename suffix for logfiles ?

Can you please provide sample logfile name generated and also contents of conf/log4j.properties file ? — Shubhangi, Jun 30 '15 at 17:39

score 3 · Accepted Answer · answered Jul 01 '15 at 05:51

Having a look on the documentation it seems there is no parameter for configuring the name of the files that are going to be created. I've gone to the sources looking for some hidden parameter, but there is no one :)

Going into the details of the implementation, it seems the name of the file is managed by the PathManager class:

private PathManager pathController;
...
@Override
public Status process() throws EventDeliveryException {
    ...
    if (outputStream == null) {
        File currentFile = pathController.getCurrentFile();
        logger.debug("Opening output stream for file {}", currentFile);
        try {
            outputStream = new BufferedOutputStream(new FileOutputStream(currentFile));
    ...
}

Which, as you already noticed, is based on the current timestamp (showing the constructor and the next file getter):

public PathManager() {
    seriesTimestamp = System.currentTimeMillis();
    fileIndex = new AtomicInteger();
}

public File nextFile() {
    currentFile = new File(baseDirectory, seriesTimestamp + "-" + fileIndex.incrementAndGet());
    return currentFile;
}

So, I think the only possibility you have is to extend the File Roll sink and override the process() method in order to use a custom path controller.

Thanks frb. I will look into it. – Avinash Sahu Jul 01 '15 at 05:54 — Avinash Sahu, Jul 01 '15 at 05:54

score -1 · Answer 2 · answered Jul 14 '15 at 19:53

For sources you have execute commands to tail and pre-pend or append details, based on shell scripting. Below is a sample:

# Describe/configure the source for tailing file
 httpagent.sources.source.type = exec
 httpagent.sources.source.shell = /bin/bash -c
 httpagent.sources.source.command = tail -F /path/logs/*_details.log
 httpagent.sources.source.restart = true
 httpagent.sources.source.restartThrottle = 1000
 httpagent.sources.source.logStdErr = true

How to set log filename in flume

2 Answers2