Not possible to get bash output to file

Question

I have Vowpal Wabbit daemon running, which I started with following command:

vw -i X_b123.model -t --quiet --daemon --port 26542 -r /dev/stdout

Then i try to send something to daemon:

echo 'somedataforwovpal' | netcat localhost 26542 -q1

I get answer, like this (both lines are the answer):

1:-0.0268077 2:-0.0990701 3:-0.154975
2

For now, everything is perfect and correct. What i want, is just send this output to file. And this is the point that drives me crazy. Why? Usually i would do it simple, by sending stdout to file like this:

echo 'somedataforwovpal' | netcat localhost 26542 -q1 > myfile.txt

Well, this worked only partially. The file was created, but it contains only the second line of output (just the number 2), and first line 1:-0.0268077 2:-0.0990701 3:-0.154975 is still print to console.

So my idea was, that the second part is going to stderr, so I tried several following ways to save stderr/stdout output:

echo 'somedataforwovpal' | netcat localhost 26542 -q1 2> myfile.txt
echo 'somedataforwovpal' | netcat localhost 26542 -q1 2>&1 >  myfile.txt
echo 'somedataforwovpal' | netcat localhost 26542 -q1 &>  myfile.txt
echo 'somedataforwovpal' | netcat localhost 26542 -q1 > myfile.txt 2>&1
echo 'somedataforwovpal' | netcat localhost 26542 -q1 >> myfile.txt
script -c "echo '1 |w auto_t dum_qt |f auto_t dum_qt |m qm_pos_2' | netcat localhost 26542 -q1" myfile.txt

Did not worked, still same. None of these methods, as you can see, i even tried script, but still same. This really drives my crazy, please, is there anybody who could save me?

The third and fourth lines will both capture stdout and stderr. If they don't work then there's some other problem. — John Kugelman, Nov 07 '18 at 18:01
did you try : >> myfile.txt ...just to see that the netcat is not call twice, erasing the first line in the process ? — Andre Gelinas, Nov 07 '18 at 18:23
The additional line of output comes from `vw`, not from `netcat`. I would try to redirect `vw` stderr when you first start the daemon, to the file you're interested in: `vw -i X_b123.model -t --quiet --daemon --port 26542 -r /dev/stdout >& myfile.txt` — arielf, Nov 12 '18 at 23:02
arielf, thanks, thats the problem (i just cant check it as solution, as it is not answer, but comment) — Jan Musil, Nov 13 '18 at 13:51

score 0 · Answer 1 · answered Nov 07 '18 at 18:20

0

It is possible to bypass normal redirections, but generally not a great idea. For example:

#! /bin/env bash
echo $* >> $(tty)

running it works.

$: bypass foo >log 2>&1
foo

Nothing goes in the log, either.

$: ls -l log
-rw-r--r-- 1 P2759474 1049089 0 Nov  7 12:16 log

I'm sorry, I don't have a good solution for you, but this may at least help you wrap your head around how this could be happening.

answered Nov 07 '18 at 18:20

Paul Hodges

13,382
1
17
36

This was only intended as an indication of how a program might be internally bypassing your control of standard I/O and writing directly to your terminal, or some other similarly heinous quackery (of which I have admittedly been guilty on rare occasion...) I don't have an actual solution, but maybe some brilliant young mind will be jogged into one? – Paul Hodges Nov 08 '18 at 14:21

John Mehorter · Answer 2 · 2018-11-08T00:14:09.160

0

Why would appending every new line not work?

echo 'somedataforwovpal' | netcat localhost 26542 -q1 >> myfile.txt

From the bash man page:

Appending Redirected Output

Redirection of output in this fashion causes the file whose name results from the expansion of word to be opened for appending on file descriptor n, or the standard output (file descriptor 1) if n is not specified. If the file does not exist it is created.

The general format for appending output is:

[n]>>word

Does this give any clues?

echo 'somedataforwovpal' | netcat localhost 26542 -q1 | cat -net

edited Nov 08 '18 at 00:14

answered Nov 07 '18 at 23:21

John Mehorter

80
6

Did you consider adding a tee pipe in between? – splaisan Nov 08 '18 at 07:25
Thanks for answer, i tried this, thinking this could work, but its still the same. The first line still goes directly do bash – Jan Musil Nov 08 '18 at 07:42

score 0 · Answer 3 · answered May 10 '19 at 05:16

If you want to save all output (stdout + stderr + explicit tty) to a file you may use the script utility:

$ script /tmp/session.log
Script started, file is /tmp/session.log

$ some_command args...
... output of some_command args ...

$ exit
Script done, file is /tmp/session.log

Now /tmp/session.log contains everything that was captured on screen during the session.

The man page for script warns:

script places everything in the log file, including linefeeds and backspaces. This is not what the naive user expects.

You may use a perl-script like the below to strip the final output from unwanted junk (not very scientific, but works for me)

#!/usr/bin/perl -w
#
# Filter to clean script (and other terminal/interactive) junk output
#
while (<>) {
    # ANSI terminal sequences (empirical)
    s/\033\[\??(?:\d*(?:;\d*)*)?[hlmHJKL]//g;
    s/\033[=>]//g;

    # Backspace/delete sequences, slow/safe way from the inside out
    s/[^\010]\010//g;

    # Other CTRL & non-wanted ASCII chars
    tr/\000-\010\013-\037\177//d;

    print;
}

Not possible to get bash output to file

3 Answers3