6

I would need to output the output of a command on a file. Let's say my command is zip -r zip.zip directory , I would need to append/write (any of these options would be fine) to a file (let's say out.txt). I got zip zip.zip directory | tee -a out.txt so far, but it doesn't seem to work, it just writes the whole output when the command is over... How can I achieve this?

Thanks ;)

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
Blax
  • 500
  • 3
  • 7
  • 18

2 Answers2

16

Background (ie. Why?)

Redirections are immediate -- when you run somecommand | tee -a out.txt, somecommand is set up with its stdout sent directly to a tee command, which is defined by its documentation to be unbuffered, and thus to write anything available on its input to its specified output sinks as quickly as possible. Similarly, somecommand >out.txt sets somecommand to be writing to out.txt literally before it's even started.

What's not immediate is flushing of buffered output.

That is to say: The standard C library, and most other tools/languages, buffer output on stdout, combining small writes into big ones. This is generally desirable, inasmuch as decreases the number of calls to and from kernel space ("context switches") in favor of doing a smaller number of more efficient, larger writes.

So your program isn't really waiting until it exits to write its output -- but it is waiting until its buffer (of maybe 32kb, or 64kb, or whatever) is full. If it never generates that much output at all, then it only gets flushed when closing the output stream.

Workarounds (How? -- GNU version)

If you're on a GNU platform, and your program is leaving its file descriptors the way it found them rather than trying to configure buffering explicitly, you can use the stdbuf command to configure buffering like so:

stdbuf -oL somecommand | tee -a out.txt

defines stdout (-o) to be line-buffered (L) when running somecommand.

Workarounds (How? -- Expect version)

Alternately, if you have expect installed, you can use the unbuffer helper it includes:

unbuffer somecommand | tee -a out.txt

...which will actually simulate a TTY (as expect does), getting the same non-buffered behavior you have when somecommand is connected directly to a console.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Hum ok ! Much clearer ! Thanks for the explanation ! So is it possible to unbuffer the `zip` command? (I'm on MacOS) – Blax Mar 21 '17 at 21:31
  • 1
    ...well, have you *tried* `unbuffer zip zip.zip directory`? – Charles Duffy Mar 21 '17 at 21:35
  • 1
    By the way, personally, I have `unbuffer` on OS X by way of having a copy of `expect` installed with macports. I assume Homebrew will carry `expect` as well. – Charles Duffy Mar 21 '17 at 21:36
  • I'll asap ;) Thanks for this awesome answer and sorry for this kind of noob question ;) – Blax Mar 21 '17 at 21:36
  • (similarly, a MacPorts install of coreutils creates a `gstdbuf` command -- so there's a good chance that if you *did* `port install coreutils`, you could `gstdbuf -oL zip zip.zip directory` successfully). – Charles Duffy Mar 21 '17 at 21:38
  • 1
    Perl has a module for working with `zip` archives (`IO::Compress::Zip`); it has flush capabilities built-in. – l'L'l Mar 21 '17 at 21:45
  • @l'L'l, *nod*; Python has a zip module as well -- lots of options if the OP's looking to use a different toolchain. – Charles Duffy Mar 21 '17 at 21:46
  • To Charles Duffy & @l'L'l : I'm trying to use bash as much as possible, but I could use Python to ;) In fact, my main goal is to display the output in a AppleScript which doesn't support unbuffering (not sure?) so I wanted to write the output in a file and then read that file in the AppleScript. – Blax Mar 22 '17 at 09:59
  • @Blaxou, ...so, I'd suggest trying to break down the problem in such a way as to either isolate it to AppleScript or eliminate AppleScript from it. That is: Test whether you can unbuffer output from `zip` to a file *without* AppleScript being involved. If you can, the only remaining question you have is one about AppleScript, and you can ask it with details unrelated to AppleScript factored out. – Charles Duffy Mar 22 '17 at 19:37
  • `unbuffer ./my_daemon.py > log.log` finally works for me! ANY other solution is not working... – recolic Nov 13 '18 at 07:27
  • 1
    @RecolicKeghart, that would be the case only if `my_daemon.py` writes its logging to the TTY. It's generally not great practice for programs run as daemons to depend on having a TTY at all -- if it's something you control the code of, maybe you might fix it there. – Charles Duffy Nov 13 '18 at 13:11
  • 1
    @recolic, ...btw, Python has its own unbuffer flag; see the `PYTHONUNBUFFERED` environment variable, and the `-u` flag to the interpreter (`python -u ./my_daemon.py`). – Charles Duffy Aug 08 '22 at 18:56
1

Did you try option command > out.log 2>&1 this log to file everything without displaying anything, everything will go straight to the file

darvark
  • 314
  • 3
  • 15
  • Does it appends/writes to the file even of the command isn't over? – Blax Mar 21 '17 at 21:18
  • 1
    @Blaxou, of course it does -- a command that's exited can't write *anything*. (As an aside, this is part of why using `kill -9` unnecessarily is bad form: When you terminate a program without giving it a chance to clean up after itself, it can't flush its buffers and write content it's been saving up to its stdout... or anywhere else). – Charles Duffy Mar 21 '17 at 21:31