0

I have an FTP Streaming Inbound Channel Adapter from Spring Integration which produces message with payloads of type InputStream, letting files be fetched without writing to the local file system.

@Bean
@InboundChannelAdapter(channel = Constants.CHANNEL_INBOUND_FTP_ADAPTER, poller = @Poller(fixedDelay = Constants.FIXED_POLLING_FROM_INBOUND_FTP_ADAPTER_DELAY))
public MessageSource<InputStream> ftpMessageSource() {
    FtpStreamingMessageSource ftpStreamingMessageSource = new FtpStreamingMessageSource(ftpRemoteFileTemplate());
    ftpStreamingMessageSource.setRemoteDirectory(ftpConnectionParameters.getRootDir());
    ftpStreamingMessageSource.setFilter(chainFileListFilter());
    ftpStreamingMessageSource.setMaxFetchSize(Constants.INBOUND_ADAPTER_MAX_FETCH_SIZE);
    return ftpStreamingMessageSource;
}

After I transform with

@Bean
@org.springframework.integration.annotation.Transformer(inputChannel = Constants.CHANNEL_INBOUND_FTP_ADAPTER, outputChannel = Constants.CHANNEL_STREAMED_DATA)
public Transformer transformer() {
    return new StreamTransformer(Charset.defaultCharset().name());
}

Then handle data to check it works and maybe for custom inteceptors for future:

@ServiceActivator(inputChannel = Constants.CHANNEL_STREAMED_DATA, outputChannel = "BATCH_ALARM_CHANNEL")
public Message<?> alarmHandler(Message<?> message) {
        System.out.println(Constants.CHANNEL_ALARM);
        System.out.println(message.getHeaders());
        return message;
}

After this according to official Spring Batch Integration documentation I have one more Transformer which let us transform to JobLaunchRequest

@Transformer
public JobLaunchRequest toRequest(Message<File> message) {
    JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
    jobParametersBuilder.addDate("dummy", new Date());
    return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}

Here we have Message from last BATCH_ALARM_CHANNEL which needs in Spring Batch Jobs, but JobParametersBuilder doesn't allow to put complex object only primitive types. So how I can pass message payload for JobLaunching and do the rest of the work such as read, parse and save to DB?

Jekshenov Chingiz
  • 15
  • 4
  • You don't as you shouldn't do it like that. Instead you probably should make (or find) a ftp based `ItemReader` and simply pass the filename/location to Spring Batch to retrieve it. Passng around an `InputStream` won't work especially as `JobParameters` as how should that be written to a database? – M. Deinum Oct 05 '22 at 09:14
  • I second what @M.Deinum said. Moving an input stream around is not a good idea, as things may get serialized in between. The file name/location is a better option as input to the batch job. – Mahmoud Ben Hassine Oct 05 '22 at 09:42
  • @M.Deinum then it makes no sense to use spring integration because I have already read file from Inbound Channel Adapter, I can create File using apache commons FileUtils.copyInputStreamToFile(inputStream, file) at Integration Layer without saving to file system it is not a problem. Do you recommend to read it twice from InboudChannelAdapter and then in Batch Job read it again by filename/location? I though that here Integration as a Transport and Batch as processing engine for messages from that transport. – Jekshenov Chingiz Oct 05 '22 at 09:54
  • @MahmoudBenHassine sure, but that would be redundant network connection to retrieve the file, which I already have. Looks illogical. Where are the benefits of using spring batch integration ? – Jekshenov Chingiz Oct 05 '22 at 10:01
  • 1
    You use the message to kickoff a batch process, but you shouldn't pass the binary data. That data will be stored in the database, so passing an `InputStream` around is thus a bad idea as there is no way to store that in a database. So you need a file or read it from FTP through Spring Batch. So actually Spring Integration isn't the transport so much but rather the triggering mechanism for the Batch part. – M. Deinum Oct 05 '22 at 11:18
  • @M.Deinum I'll take note of this, it seems that integration is indeed mounted to a remote source in the first place in order to become a trigger for launching batch processes, mount a couple of filters for files and get simple type meta information for the job. Thank you. – Jekshenov Chingiz Oct 05 '22 at 11:49

0 Answers0