BufferedWriter writes partial lines despite flush()

Question

I'm appending to a simple journal file, in which each line is an entry. Unfortunately, upon shutdown a partially written line is often present. The code looks like this:

private val out = java.nio.file.Files.newBufferedWriter(
  java.nio.file.Paths.get(file),
  java.nio.charset.StandardCharsets.UTF_8,
  java.nio.file.StandardOpenOption.APPEND,
  java.nio.file.StandardOpenOption.CREATE)

def write(s: String) = synchronized {
  out.write(s)
  out.newLine()
  out.flush()
}

(Sorry about the Scala.)

I don't understand why a partial line would be in the file. The default buffer size I believe is 8 kB. The entries are shorter (<500 bytes). I flush after every write.

I have not been able to reproduce the partial lines — the problem happens at a client. But I am almost certain they simply send SIGTERM to the process to shut it down.

The log files written by SLF4J/LogBack/Log4j never have partial lines. What am I doing wrong? I looked in Apache Commons IO, hoping to find a guaranteed line-based Writer but did not.

score 1 · Accepted Answer · answered Jul 24 '15 at 08:35

1

A SIGTERM will stop your program no matter where it is if you don't handle it properly. It may be halfway inside the flush(). Consider handling the SIGTERM yourself, and do a clean shutdown when that happens.

Be advised, however, that partial lines can always occur. Your program may be killed during flush(), the operating system may crash or power loss may occur. Your program should (on start) be able to handle a journal that doesn't end with a newline.

I don't have enough knowledge about the log writers you mentioned to know how they avoid partial lines, maybe someone else can provide a better answer about that than I.

answered Jul 24 '15 at 08:35

jornane

1,397
10
27

Thanks! I was hoping the JVM would handle SIGTERM for me, perhaps close files and stuff. Is there a way to set this up without writing it from scratch myself? The program is now able to handle all corrupt input, but still I'd prefer less corrupt input if possible. – Daniel Darabos Jul 24 '15 at 09:25
I think journal is a good point - 'cause the new file systems are all journaling filesystems. They write a transaction before it started in the journal, after that the transaction is started. If the transaction was complete the journal entry will be deleted. If the system has an unexpected shutdown after a restart at first the journal will be checked and the documented transaction executed so there is no lost. – Zelldon Jul 24 '15 at 10:22
1

To handle a SIGTERM you might want to use a JVM shutdown hook, as explained in [this answer](http://stackoverflow.com/a/2975265/524900) – Lolo Jul 25 '15 at 17:23

BufferedWriter writes partial lines despite flush()

1 Answers1