0

I have a critical process running callbacks on a main thread, and I log various data to files from those callbacks. I don't want to incur the performance penalty of flushing the output to file during critical callbacks, but I also can't wait indefinitely for the system to decide to flush the ofstream buffers. I.e., I can wait a few seconds, but not minutes, to see the data in my file monitors.

Can I just run a timer on a separate thread that frequently flushes the ofstreams, or is that not thread-safe?

Or is there a better way to log data to files that minimizes the load on the critical thread?

(I use Boost if that offers any convenient solutions.)

feetwet
  • 3,248
  • 7
  • 46
  • 84

2 Answers2

1

Since ofstream is not thread safe, you would have to lock it while flushing it, so the performance penalty ends up being about the same (instead of critical callbacks taking the time to flush, they take the time to wait on a separate thread to release a lock while it flushes).

The standard answer to the question, "Will this run fast enough?", is "Measure it and find out before you do a lot of extra work." Have you measured performance to confirm that flushing from your callback is a problem?

If flushing from your callbacks is a problem, then one alternative would be to use a producer-consumer queue; the callbacks queue log entries to a threadsafe queue, while a consumer thread locks the queue, pops an entry, unlocks the queue, then writes the entries to disk.

Community
  • 1
  • 1
Josh Kelley
  • 56,064
  • 19
  • 146
  • 246
  • 1
    _Measure it and find out_ - in general I agree with this sentiment, but network and disk IO are two things you should usually design as asynchronous from the beginning, and it sounds like this is one of those cases. – MooseBoys Feb 27 '15 at 02:24
  • One easy way to measure is to replace your stream with a stub/subclass that has the same buffering algorithm but never actually flushes. – yyny Aug 08 '20 at 16:34
  • The flush may be fine when you measure it. It may be fine 99.999% of the time. But all it takes is some other I/O happening on the disk and suddenly the cost of the flush is more than you can pay. – aggieNick02 Aug 19 '21 at 18:44
1

Standard ofstreams are not thread safe. A common alternative to that is to have the processing thread enqueue requests to write and have a single thread do the writes and the flush. You could, for example, dynamically allocate buffers with the contents and push the pointer through a thread-safe queue (minimal locking only while pushing the pointer) and have the thread on the other side write to the file flushing as frequently as you want to.

David Rodríguez - dribeas
  • 204,818
  • 23
  • 294
  • 489