1

I'm new to Kafka Streams.

I use the suppress method of KTable in order to handle only the final result of a window like this:

myStream
    .windowedBy(TimeWindows.of(Duration.ofSeconds(10)).grace(Duration.ofMillis(500)))
    .aggregate(new Aggregation(),
        (k, v, a) -> a,  // Disabled the actual aggregation in order to eliminate possiblities of latency
        materialized.withLoggingDisabled())
    .suppress(untilWindowCloses(Suppressed.BufferConfig.unbounded()))
    .toStream().peek((k, v) -> log.info("delay " + (System.currentTimeMillis() - k.window().endTime().toEpochMilli())));

This way I get a log with the delay every 10 seconds with the difference between the window end and the actual time the peek was called. I would exect a very small number here, since this code practically does nothing...

Nevertheless, I get delay of 4-20 sec for each key/window.

I use a thread per task (5 threads for this topic).

Can someone please point out if I'm doing anything wrong?

Thanks!

Edit:

Using VirtualVM shows that ~99% of the time consumed over sun.nio.ch.SelectorImpl.select(). This means AFAIU, that the process is "idle" most of the time.

Edit:

It seems that changing "commit.interval.ms" (which was by default 30000) reduced the delay drastically.

Still delay has peaks of event 15 seconds, so the problem isn't solved yet...

user1028741
  • 2,745
  • 6
  • 34
  • 68
  • Check following answer: https://stackoverflow.com/a/54226977/5950751, maybe it will help – Bartosz Wardziński Jan 30 '19 at 11:02
  • No... I have enough traffic (multiple event/sec). – user1028741 Jan 30 '19 at 11:04
  • Actually, @wardziniak, after setting "commit.interval.ms" the system seem to work fine. I only had several events per second, and they were divided by 5 partitions. So, the link you posted can explain those delays. Thanks. – user1028741 Jan 31 '19 at 07:41
  • @user1028741 have you been able to find the reason for those delays? I am seeing the same behavior, so, if this is relevant, it might help to share those. – erankl Oct 24 '19 at 10:56
  • I think the problem was that I didn't have enough traffic, after all.. :) – user1028741 Oct 24 '19 at 11:01
  • Thanks. Enough traffic solves the problem to me as well. The problem is that it makes it really difficult to run tests as they don't show a consistent behavior. – erankl Oct 24 '19 at 11:43

0 Answers0