3

Our kafka system crashed because no disk space was available. The consumers are Spring boot application which are using the Kafka Streams API. Now every consumer application shows the following error:

java.io.FileNotFoundException: /tmp/kafka-streams/908a79bc-92e7-4f9c-a63a-5030cf4d3555/streams.device-identification-parser/0_48/.checkpoint.tmp (No such file or directory)

This exception occurred exactly after the kafka server was restarted. If we restart the application, the service starts at offset 0 to replay all message from the consumed topic. This is fatal.

We checked the logs: On each partition there is no information (example line, we have 60 partitions):

TOPIC                               PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                            HOST            CLIENT-ID
device-identification.received      0          -               0               -               streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer-ca723291-3258-46c0-8b23-233bdc1e1191 /XX.X.X.X       streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer

Can anyone help me, to restore the offset information, that the service starts from the last offset?

Nicholas K
  • 15,148
  • 7
  • 31
  • 57
Norbert Koch
  • 533
  • 6
  • 17
  • 2
    Would resetting the offsets manually for all the customers work for you? Something like described here: https://stackoverflow.com/questions/29791268/how-to-change-start-offset-for-topic – t6nn Dec 17 '18 at 18:09
  • Hi, in your link, I see, that it is possible to set the offset by datetime. I will try it with my team tomorrow in the morning CET. This comment is helful, thanks! – Norbert Koch Dec 17 '18 at 18:41
  • 1
    It works fine! I was able to restore the offset time based. Thanks! – Norbert Koch Dec 19 '18 at 06:56

1 Answers1

1

If this is a Kafka Streams app you might be successful in resetting it, that can be done using Confluent's Application reset tool: https://docs.confluent.io/current/streams/developer-guide/app-reset-tool.html

Chris Matta
  • 3,263
  • 3
  • 35
  • 48
  • This helps also like the comment on the top. It would by maybe the best solution. But for me it is difficult to understand, what happened and why we the offsets are lost. Do you also know why? – Norbert Koch Dec 17 '18 at 19:18
  • Without more logs and configs I can't really venture a guess. Was the offset topic replicated? If not, that would cause the data loss. – Chris Matta Dec 17 '18 at 21:45