2

In a Clojure program, how do you read from standard out? I want to do that, or pipe the standard output, to an input stream that I create. The standard output in Clojure is a java.io.PrintWriter .

I have a Samza job, started by a Clojure program. There's also an nrepl server to which I can remotely connect. After connecting, I need to be able to tap into and tail standard out (to which jobs write their output).

1) As per this SO question, with-out-str (see here) lets us temporarily bind *out* (to a java.io.StringWriter), so that your executed code writes to a string. But that doesn't let me tap into the existing *out*.

2) If you look at clojure.java.shell (see here), it gets the JVM's Runtime and exec's a Process on it. From that process, you can get its standard output stream. But again, that's not the default standard out (*out*) I'm looking for.

3) This SO question gets close to what I'm trying to do. But again, I'm connecting to an existing process, and want to tail out its standard output.

Is this possible in Clojure (see here)? Has anyone solved this?

Community
  • 1
  • 1
Nutritioustim
  • 2,686
  • 4
  • 32
  • 57
  • On Linux you can read from `/proc/{pid}/fd/1` which is the stdout of that process. Similarly `/proc/{pid}/fd/2` is stderr. – Peter Lawrey Nov 08 '16 at 07:35
  • 2
    Do you mean that you want to pipe the output from a process into a clojure process like `ls | my-clj` and have my-clj read the output of ls? If so, just use http://stackoverflow.com/questions/18688755/reading-characters-from-stdin – Alan Thompson Nov 08 '16 at 07:41
  • @AlanThompson No, I'm remotely connecting to a repl, in an already running process. Once connected, I want to *"tail"* standard output. – Nutritioustim Nov 08 '16 at 18:30
  • I think the target process would need to have its output redirected to a file, or write directly to a log file. Then you could tail that. – Alan Thompson Nov 08 '16 at 20:26
  • Can I assume you are running this on linux or a mac? – Arthur Ulfeldt Nov 08 '16 at 23:28

1 Answers1

1

Process output is not a publish subscribe model, so in effect when a process puts a character into it's output buffer, exactly one process gets to pull it off that buffer. If you have a program that was started by a shell that shell process if reading it's output and writing it to a terminal (or reading and ignoring it). If you attach your process after the process that started it and start trying to grab the data, you will most likely not get anything because the parent process will get it first. I just tried this from two terminals:

Terminal 1:

cat

Terminal 2:

ps -ef | grep cat
tail -f /proc/24547/fd/2

Terminal 1:

hello

Terminal 2: < nothing >

The string "hello" printed to terminal 1, the process that started it.

It's tempting then to say "well what if nobody reads the output, then it will be there for me to get". While this sounds good it runs into the problem that these are fixed sized buffers, so as soon as the output buffer is full the process that is trying to write to it blocks (is prevented from running at all) until someone reads the output to unblock it.

The general solution is to pipe the process you want to tail later to the tee command which writes the output to a file and passes it to whatever was reading it.

command-to-watch arg1 arg2 | tee logfile.potentially-huge

Though if you go this route you should rotate the log file before your disk fills. Make sure you empty the log file with exactly this command

echo > logfile.potentially-huge

or use your program to make a truncate call to the file. simply deleting the file will remove it's name from the log directory without deleting it, it will silently continue to grow taking up disk space and the new file will get no output ever.

This is basically why we built log libraries like log4j (in the 90s) and syslog (in the 80s).

If you still want to get hackish crazy on this, turn to tmux, it can do anything, and changes the way people work with text. In all seriousness you should change the way the other process creates it's output to make it easier to get.

Arthur Ulfeldt
  • 90,827
  • 27
  • 201
  • 284