4

I am reading spring cloud gcp storage documentation and there written that I can listen for a new files using GcsInboundFileSynchronizer or GcsStreamingMessageSource just configuring spring bean like this:

@Bean
@InboundChannelAdapter(channel = "streaming-channel", poller = @Poller(fixedDelay = "5000"))
public MessageSource<InputStream> streamingAdapter(Storage gcs) {
  GcsStreamingMessageSource adapter =
          new GcsStreamingMessageSource(new GcsRemoteFileTemplate(new GcsSessionFactory(gcs)));
  adapter.setRemoteDirectory("your-gcs-bucket");
  return adapter;
}

I have a coule of questions:

  1. What if my application is started on 2+ nodes. How the files will be distributed ? Round robin ? is there any way to configure batching ? Is it possible to accept repeated notifications (like in pub sub and any other MQ systems)?

  2. What does it mean "new files"? Lets say my bucket contains 2 files(1.txt and 2.txt). Then I start applciation for the first time. Will GcsStreamingMessageSource accept these files. Or let say application was crashed for some reasons. Then I put a new file to the bucket and start application againg.

  3. Is there any recovery abilities? let say the application was crashed for some reason while file processing. Will it redelivered ?

P.S.

For now we use bucket notifications which are sent to the pubsub. And application listens for a PubSub topic and downloads file based on notifications header. Is it more reliable way?

gstackoverflow
  • 36,709
  • 117
  • 359
  • 710
  • This seems to be far bread and deep to cover here. Maybe you should ask those questions on git-hub https://github.com/spring-cloud/spring-cloud-gcp/issues...? – vitooh Mar 06 '20 at 14:33
  • @vitooh I can only report a bug there but I don't think if it is a bug. – gstackoverflow Mar 13 '20 at 19:59
  • in general, if you need to keep state of what was processed and avoiding reprocessing, you'll need to store some data about this. in Spring Integration, it's the Metadata Store. If you do have more than 1 node, all nodes will receive the same information, and you'll need a shared metadata store (db, or redis). See https://docs.spring.io/spring-integration/reference/html/meta-data-store.html. Gcs message source works similar to the FTP message source: https://docs.spring.io/spring-integration/reference/html/ftp.html#more-on-file-filtering-and-incomplete-files – saturnism Aug 24 '20 at 19:05

0 Answers0