16

I am running a huge task [automated translation scripted with perl + database etc.] to run for about 2 weeks non-stop. While thinking how to speed it up I saw that the translator outputs everything (all translated sentences, all info on the way) to STDOUT all the time. This makes it work visibly slower when I get the output on the console.

I obviously piped the output to /dev/null, but then I thought "could there be something even faster?" It's so much output that it'd really make a difference.

And that's the question I'm asking You, because as far as I know there is nothing faster... (But I'm far from being a guru having used linux on a daily basis only last 3 years)

naugtur
  • 16,827
  • 5
  • 70
  • 113

3 Answers3

27

Output to /dev/null is implemented in the kernel, which is pretty bloody fast. The output pipe isn't your problem now, it's the time it takes to build the strings that are getting sent to /dev/null. I would recommend you go through the program and comment out (or guard with if $be_verbose) all the lines that are useless print statements. I'm pretty sure that'll give you a noticeable speedup.

Benson
  • 22,457
  • 2
  • 40
  • 49
  • yeah, but the worst part is in C++ and I don't have the code. Still it's cool to know /dev/null is the best ;) – naugtur Apr 27 '10 at 21:15
  • 1
    Ahh, okay. You did say it was written in perl. ;-) Yes, /dev/null is as fast as an output file descriptor can possibly be without kernel hacking, and even then it'd be hard to beat. – Benson Apr 28 '10 at 04:50
  • I picked this answer for the additions in comment. I can't accept both answers. stackoverflow wouldn't let me;) – naugtur May 07 '10 at 21:22
  • /dev/null can't be beat for speed. What's faster than: return count; – mlv Dec 21 '16 at 14:16
  • @Benson wouldn't all these "if $be_verbose" conditions slow down the calculations? I always had this query: which is the most efficient way to deactivate the debugging messages without removing or commenting them out one by one? – tevang Oct 31 '18 at 10:44
  • @tevang Depends. If they're _runtime_ checks then yes. If they're _compile time_ checks then no. But even at runtime checking a local variable is orders of magnitude faster than unconditionally executing a system call into the kernel just to effectively do nothing and return (assuming /dev/null redirect). – sherrellbc Jul 14 '20 at 16:32
  • @tevang In practice I often compile for debug and release by just changing the definition of a print macro. If you want to support e.g. `--verbose` flag then you'll need to at least incur the overhead of checking the state of a configuration variable. That's going to be the most efficient way. Other than removing the prints at compile-time completely (like with macro definitions), you'll always have to check whether or not to do the prints. If you use macros then the prints just won't be emitted in the code. – sherrellbc Jul 14 '20 at 16:33
21

I'm able (via dd) to dump 20 gigabytes of data per second down /dev/null. This is not your bottleneck :-p

Pretty much the only way to make it faster is to not generate the data in the first place - remove the logging statements entirely. The cost of producing all the log messages likely exceeds the cost of throwing them away quite a bit.

Steven Schlansker
  • 37,580
  • 14
  • 81
  • 100
  • Thanx for the test. I didn't expect it to be a bottleneck, rather just wondering... it's 23:15 where I sit. Good time for such thoughts :P – naugtur Apr 27 '10 at 21:17
1

Unrelated to perl and standard output, but there is null_blk block device, which is even faster than /dev/null. Basically, it bounded by syscall performance and with large blocks it can saturate memory bus.

George Shuklin
  • 6,952
  • 10
  • 39
  • 80