Kafka - Broker fails because all log dirs have failed

Question

I am attempting a simple kafka config on Windows os. My zookeeper and kafka installations use default configs (except for data and log dir paths). I can start kafka and produce/consume messages without issue; however, when the broker attempts to delete old messages (I set log retention to 100 ms), I get the following error:

    java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
        at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697)
        at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212)
        at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415)
        at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:1601)
        at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:1588)
        at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170)
        at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply$mcI$sp(Log.scala:1170)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161)
        at kafka.log.Log.maybeHandleIOException(Log.scala:1678)
        at kafka.log.Log.deleteSegments(Log.scala:1161)
        at kafka.log.Log.deleteOldSegments(Log.scala:1156)
        at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1228)
        at kafka.log.Log.deleteOldSegments(Log.scala:1222)
        at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:854)
        at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:852)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.log.LogManager.cleanupLogs(LogManager.scala:852)
        at kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:385)
        at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

                at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
                at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
                at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
                at java.nio.file.Files.move(Files.java:1395)
                at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
                ... 32 more

And then...

[2018-08-01 18:14:01,479] INFO [ReplicaManager broker=0] Stopping serving replicas in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs (kafka.server.ReplicaManager)
[2018-08-01 18:14:01,480] ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for discrete-topic-0 in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs
Caused by: java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.
[2018-08-01 18:14:01,504] INFO Stopping serving logs in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs (kafka.log.LogManager)
[2018-08-01 18:14:01,508] ERROR Shutdown broker because all log dirs in C:\Workspace\kafka_2.11-1.1.0\kafka-logs have failed (kafka.log.LogManager)

This seems to be this issue: https://issues.apache.org/jira/browse/KAFKA-6188

kafka version: kafka_2.11-1.1.0 Windows 10 Zookeeper version: 3.4.12

I have a few questions:

Has anyone seen this issue and is there a workaround?
Is running a kafka broker on Windows a viable options at this point? I know it is not supported for Windows and there seem to be multiple critical issues.

Thanks

Any reason you're not using the latest Kafka? And 100ms is really small... But anyway, that file is still being used by another process, and that's not exactly a windows issue — OneCricketeer, Aug 02 '18 at 04:37
You can check out my answer on this post: https://stackoverflow.com/questions/51473270/kafka-configuration/51477919#51477919 I think it's the same issue. — Bitswazsky, Aug 02 '18 at 08:59
1) yes and no workaround 2) There are no tests for windows so it is not really a supported platform so no go for production from my point of view — Paizo, Aug 02 '18 at 10:05
1) I set retention to 100 ms to quickly clear the log. I agree this is a very small vallue and not for normal operation. 2) @ Bitswazsky Your solution does seem to help. Thank you — user3029642, Aug 02 '18 at 19:25

score 37 · Accepted Answer · answered Feb 22 '19 at 08:34

37

step 1 : point to a new log directory in server.properties file and save the file

log.dirs=C:\Tools\kafka_2.11-2.1.0\kafka-test-logs

step 2 : start the kafka server again

/bin/windows/kafka-server-start.bat /config/server.properties

answered Feb 22 '19 at 08:34

Hari Chukkala

402
5
5

5

That (changing log.dirs) helped, although unclear why – Georgii Ivankin Apr 10 '19 at 18:33
1

It is strange. After log.dirs=C:\Tools\kafka_2.11-2.1.0\kafka-test-logs created directory c:\ProgramPortable\kafka_2.12-2.3.0\Toolskafka_2.11-2.1.0kafka-test-logs\ Is it what you expected? – RoutesMaps.com Jul 10 '19 at 07:18
3

You have to use \\ to escape the backward slashes. – Irshad P I Jun 17 '20 at 10:58
This resolved the error but does it remove messages from existing topics? – Guy_g23 Feb 23 '23 at 08:29

score 18 · Answer 2 · answered Jul 15 '19 at 09:13

18

I had the same kafka behavior but simply renaming or cleaning log directories wouldn't do the trick. And every time I tried to start kakfa it would crash (it was generating back all the logs deleted).

Solution:

Delete the logs as before.
Go to ZooKeeper data dir (you find it in \conf\zoo.cfg) and clean it. Restart everything and it should be completely reset.

answered Jul 15 '19 at 09:13

Tudor

2,224
2
21
24

1

I have also deleted zookeeper data logs and it worked – Waqas Rana Aug 10 '20 at 13:35

score 10 · Answer 3 · edited May 22 '23 at 09:36

You can delete all Kafka data. See Kafka docs:

Stop the producer and consumer clients with Ctrl-C, if you haven't done so already.

Stop the Kafka broker with Ctrl-C.

Lastly, stop the ZooKeeper server with Ctrl-C.

If you also want to delete any data from your local Kafka environment including any events you have created along the way, run the command:
$ rm -rf /tmp/kafka-logs /tmp/zookeeper
If you are on a Windows machine, you can use the following command
rmdir /s /q C:\tmp\kafka-logs

rmdir /s /q C:\tmp\zookeeper

score 4 · Answer 4 · answered Feb 24 '21 at 21:15

Go to config folder in kafka folder (e.g. D:\kafka\config) open zookeeper.properties Edit your datadir to look like : dataDir = D:/kafka/data/zookeeper open server.properties in notepad and edit logs.dirs and zookeeper.connect logs.dirs = D:/kafka/data/kafka zookeeper.connect=localhost:2181 NOTE: replace D:\kafka according to your own settings and folders you have for kafka

score 2 · Answer 5 · edited Oct 02 '19 at 13:44

I faced similar issue when I run zookeeper, kafka server, command line producer and consumer locally with default configs as described here.

Server and producer started successfully, but when I tried to start consumer I immediately got the issue with failed logs.

I fixed it by setting different log directories for server, producer and consumer. As far as there is no explicit way to set it for producer and consumer I had to update their start scripts. I added a line

set LOG_DIR=/distr/kafka_2.11-2.2.0/producer_logs

to kafka_console_producer.bat
And /distr/kafka_2.11-2.2.0/consumer_logs for kafka_console_consumer.bat respectively.
After the fix consumer successfully started. Seems the problem is because all three processes use the same log directory and interfere with each other somehow.

score 2 · Answer 6 · answered Jan 21 '20 at 08:43

2

I am using Kafka 2.12-2.3.0 on Windows. And What worked for me was changing server.properties setting log.dirs= to contains both log data Kafka and Zookeeper.

Example :

log.dirs=C:/kafka_2.12-2.3.0/data/kafka, C:/kafka_2.12-2.3.0/data/zookeeper

answered Jan 21 '20 at 08:43

Gabriel Scavassa

576
1
4
21

Is this applicable when using zookeeper which came with kafka – driftking9987 Nov 29 '20 at 10:22
1

Thanks! This worked for me. I'm using Windows 10 with Kafka 2.13. – Mike67 Jul 28 '21 at 00:11

score 1 · Answer 7 · answered Jul 13 '20 at 11:57

I tried multiple methods to solve this issue:

deleting logs from folder pointed by log.dirs property in server.properties. it does solve my problem for certain duration but it stoped working when i had more than 20 topics created so each time i start my kafka server this issue was coming.
tried deleting topics from kafka using command "kafka-topics --zookeeper localhost:2181 --delete --topic topicname" but it marked my topic as --"marked to be deleted flag". but did not solved my problem.

then finally i tried deleting folders zookeeper and kafka-logs inside tmp folder of kafka and it worked and all topics were deleted.

score 1 · Answer 8 · answered Sep 23 '20 at 05:43

1

Delete the log files of Kafka brokers and also the data and log files of zookeeper and restart both. Also keep in mind to stop the Kafka clients before you restart the Kafka and Zookeeper

answered Sep 23 '20 at 05:43

Srikantha G K

21
4

score 1 · Answer 9 · edited Apr 26 '21 at 10:07

1

Try creating a new folder inside the Kafka folder. Change server.properties and zookeeper.properties and restart the command prompts and again try starting the servers.

edited Apr 26 '21 at 10:07

B001ᛦ

2,036
6
23
31

answered Apr 23 '21 at 15:14

user2613323

11
2

score 1 · Answer 10 · answered Apr 25 '21 at 19:25

1

In dev machine, I stopped zookeeper and kafka-server and all client and deleted the contents of the logs directory. Then started the zookeeper, kafka, connectors, clients in sequence.

answered Apr 25 '21 at 19:25

aCiD

1,273
1
18
28

score 1 · Answer 11 · answered Nov 07 '22 at 08:16

The problem is because of that you are running out of storage for logs.

Change log.dirs in server.properties. Also make sure that you and then restart Kafka.

P.S. 1: By doing so, the older messages will be lost.

P.S. 2: Don't set the tmp/ as log.dirs location, since it is purged once your machine turns off.

score 0 · Answer 12 · answered Jul 26 '21 at 08:04

0

Remove all files under the local folder as shown in your docker monitor. After, restart the containers.

answered Jul 26 '21 at 08:04

G.Noulas

442
6
10

score 0 · Answer 13 · answered Feb 07 '23 at 23:44

Similar to what Tutor did above, what worked for me was deleting the zookeeper-data folder. I'm on Windows 10, so the zookeeper-data folder was at my root Kafka folder, and defined in config/zookeeper.properties

dataDir=c:/projects/kafka/zookeeper-data

FYI..you'll lose whatever topics, etc. you've already created

score 0 · Answer 14 · answered Jun 13 '23 at 21:32

0

chown kafka -R /home/kafka/kafkalogs

worked for me

answered Jun 13 '23 at 21:32

usman mustafa

1

Kafka - Broker fails because all log dirs have failed

14 Answers14