Spring integration tailing multiple files

Question

I am writing a spring integration application that is supposed to tail multiple files (could be as many as 100). I am using the OSDelegatingFileTailingMessageProducer as the message source, which is the beginning of a pipeline involving multiple filters and channels.

Tailing one file works fine with this pipeline with an XML configuration file for the channels and transformers, but tailing many of those files would mean multiplication of this XML configuration, which is not good programming practice in my eyes.

I guess I will have to build these pipelines within Java by programmatically constructing a Spring application context. Are there any other options?

EDIT:

Probably using the BeanFactoryPostProcessor is the way to go: https://stackoverflow.com/a/15773000/2069922 ?

Answer 1

I think it would be easiest to create the message producers programmatically, and wire them into the same outputChannel. There's not really any need to create a Spring Application context each time. Just get the channel from the context (e.g. @AutoWired) and set the outputChannel.

Polled adapters are a bit more complicated but, in this case, each tail adapter is a simple single bean.

Just be sure to invoke afterPropertiesSet() and start() after setting the properties.

However, if you want a unique downstream flow for each tailer, then you can use a technique similar to the dynamic ftp sample, with parameterized application contexts.

Answer 2

I ended up not registering the instances of OSDelegatingFileTailingMessageProducer with the Spring application context, because there was no need to, as Gary suggested. Instead I used a ApplicationListener and registered that with my Spring context. Then I created the tailers within the onApplicationEvent(...) method. Here is a minimal version:

public class MyApplicationListener implements ApplicationListener<ContextRefreshedEvent> {

  @Autowired 
  @Qualifier("outputChannel")
  private SubscribableChannel outputChannel;


  @Override
  public void onApplicationEvent(ContextRefreshedEvent event) {
    OSDelegatingFileTailingMessageProducer tailer = new OSDelegatingFileTailingMessageProducer();
    tailer.setOutputChannel(outputChannel);
    tailer.setFile(new File("/file/to/tail.txt"));
    tailer.setOptions("-f -n 0");
    tailer.afterPropertiesSet();
    tailer.start();
  } 
}

EDIT:

Also we ended up not using the OSDelegatingFileTailingMessageProducer, but the one from apache, since the tail command behaves differently on different flavors of Unix. On first sight we could not identify any performance differences.