1

I have a program from which I want to read, lets say its python. So I have these 2 functions:



(defun start-python ()
  (let ((process 
     (sb-ext:run-program "/usr/bin/python" nil
                 :output :stream
                 :input :stream
                 :wait nil
                 :search t
                 :error *standard-output*)))
    process))

(defun test-process-stream ()
  (let ((process (start-python)))
    (format (sb-ext:process-input process) "print 'hello world!'~%")
    (finish-output (sb-ext:process-input process))
    ;;I do not want to call "close" because I will be still reading from the input
    (close (sb-ext:process-input process))
    (print (read-line (sb-ext:process-output process)))
    (when (listen (sb-ext:process-output process))
      (print (read-line (sb-ext:process-output process))))

    (close (sb-ext:process-output process))
    (sb-ext:process-close process)
    ))

I want to be able to incrementally read from the output of the python process while at the same time provide input to it. I have tried with several methods even ones mentioned here: SBCL: Gather output of run-program process while running

But I have not been able to do this in SBCL. In the example code I call close because that is the only way I get any output at all. Otherwise it just hangs.

I would much appreciate any pointers because I am stuck at this. I even tried with (listen ...) and (finish-output ...) and still it hangs on (read-line ...). The only difference with (listen ...) is that it returns false and nothing is printed. I even tried (sleep 2) before trying to read. Still nothing.

EDIT: Ultimately my goal is to run swipl which is SWI-Prolog. I used python here as an example. I want to achive interoperability between lisp and prolog such that I could issue queries to prolog and read in back responses. Currently I couldn't find any projects or libraries that would be suited for my needs, so that is why I am attempting this.

Gordon Zar
  • 103
  • 1
  • 8
  • Assuming you're not running this on Windows, try using the `:pty` argument to `RUN-PROGRAM`, so that the sub-process will behave as it would if you ran it interactively from the shell. – jkiiski Jan 26 '19 at 23:51
  • I get `Broken pipe [Condition of type SB-INT:SIMPLE-STREAM-ERROR]` When I try to write to the input stream of the process – Gordon Zar Jan 26 '19 at 23:58
  • With what code? – jkiiski Jan 26 '19 at 23:58
  • with `(format (sb-ext:process-input process) "print 'hello world!'~%")` Basically with the same code – Gordon Zar Jan 26 '19 at 23:59
  • If you're using the pty, you need to use the pty stream. See the docstring for `RUN-PROGRAM` – jkiiski Jan 27 '19 at 00:00
  • I can do `(read-line)` until the help messages end. After that it hangs. So it looked like it could work but it still doesn't do what I need. – Gordon Zar Jan 27 '19 at 00:51
  • I would guess that's because you're communicating with a repl that prints an incomplete line for the prompt. `READ-LINE` keeps waiting for a newline or EOF. Try reading the stream character by character instead (using `LISTEN` to see when there is another char to read). – jkiiski Jan 27 '19 at 05:38
  • Are you aware of I/O buffering? – hkBst Jan 27 '19 at 12:21
  • @jkiiski You were right. While `LISTEN` always returns `nil` regardless of there being a character, `(read-char ...)` works. So I managed to get this to work partially. @hkBst yes I am aware of it – Gordon Zar Jan 27 '19 at 15:34
  • You could also try `READ-CHAR-NO-HANG` instead of using `LISTEN`. That should return a character if available, or `NIL` if not. – jkiiski Jan 27 '19 at 15:45
  • @jkiiski I just was messing around with it and managed to get it working. I also managed to use `LISTEN` after I did `(sleep 0.1)` before checking if there is any output and it worked. – Gordon Zar Jan 27 '19 at 16:12

2 Answers2

4

[Most of this answer is not interesting because the questioner asked about Python when it turned out they meant Prolog, so I wasted my time addressing the problem they said they had rather than the one they actually had. I'm leaving it here in case any of it is useful to anyone else.]

I don't think this is a problem with SBCL. Given the following code:

(defun call-with-process (f program args &rest keys &key &allow-other-keys)
  (let ((process (apply #'sb-ext:run-program program args keys)))
    (unwind-protect
        (funcall f process)
      (sb-ext:process-close process))))

(defmacro with-process ((process program args &rest keys &key &allow-other-keys)
                        &body forms)
  `(call-with-process
    (lambda (,process)
      ,@forms)
    ,program ,args ,@keys))

(defun test-echo-process (&rest strings-to-send)
  (with-process (p "/bin/cat" '()
                   :wait nil
                   :input ':stream
                   :output ':stream)
    (let ((i (sb-ext:process-input p))
          (o (sb-ext:process-output p)))
      (dolist (s strings-to-send)
        (format i "~A~%" s)
        (finish-output i)
        (format t "Sent ~A, got ~A~%" s (read-line o))))))

Then test-echo-process works fine.

But this function (equivalent to yours) hangs:

(defun test-python-process ()
  (with-process (p "/usr/bin/python" '()
                   :wait nil
                   :input ':stream
                   :output ':stream)
    (let ((i (sb-ext:process-input p))
          (o (sb-ext:process-output p)))
      (format i "print 'here'~%")
      (finish-output i)
      (format t "~A~%" (read-line o)))))

So, in fact the problem is the way Python is behaving. And you can demonstrate this in fact. Here's some output from a terminal:

$ cat | python
print "hi"
print "there"
hi
there
$

What doesn't show up is that, after I typed the second print command, I sent an EOF (ie ^D on Unix).

So Python is, perfectly reasonably I think, buffering its input, probably only in the case that it's not a terminal.

So you need to do something to stop this happening. As an initial thing I'd put the program you want Python to run in a file so that standard input does just one thing. But then you will find yourself in a world of pain.

If you implement this function

(defun test-python-script (args &rest strings-to-send)
  (with-process (p "/usr/bin/python" args
                   :wait nil
                   :input ':stream
                   :output ':stream)
    (let ((i (sb-ext:process-input p))
          (o (sb-ext:process-output p)))
      (dolist (s strings-to-send)
        (format i "~A~%" s)
        (finish-output i)
        (format t "sent ~A, got ~A~%" s (read-line o))))))

Then you might think that a bit of Python in echo.py like this:

from sys import stdin, exit

if __name__ == '__main__':
    for l in stdin:
        print "got " + l.strip()
    exit(0)

& then running (test-python-script '("echo.py") "foo" "bar") would work. But you'd be wrong in at least two ways (you can check this by just running python echo.py on the command line and seeing it still buffers.

The first way you're wrong is that using files as iterators in Python has built in buffering which you don't seem to be able to avoid. You can deal with that by writing an unbuffered iterator, so echo.py is now

from sys import stdin, exit

class UnbufferedLineIterator(object):
    # I take back what I said about 'perfectly reasonably'
    def __init__(self, stream):
        self.stream = stream

    def __iter__(self):
        return self

    def next(self):
        line = self.stream.readline()
        if len(line) > 0:
            return line
        else:
            raise StopIteration

if __name__ == '__main__':
    for l in UnbufferedLineIterator(stdin):
        print "got " + l.strip()
    exit(0)

And this might work, but it doesn't, because there is still buffering somewhere on the Python side. You can get rid of this in a very crude way by running Python with a -u argument. So, finally

* (test-python-script '("-u" "echo.py") "foo" "bar")
sent foo, got got foo
sent bar, got got bar

However I think the real answer is to go and ask Python people how this is meant to work, because I can't believe that -u is the right answer or that it can be this hard.

  • Now Imagine that instead of python I want to invoke prolog. swipl to be precise. And I am having the same problem. I used python here as an example. If this turns out to be python specific I might need to redo the question. – Gordon Zar Jan 26 '19 at 23:39
  • 1
    @GordonZar It really might have been a good idea to ask about the problem you *actually had*. But it will be whatever you want to run buffering its input. –  Jan 26 '19 at 23:52
  • It seems like you are right. `cat` works normally but programs like these will probably not work – Gordon Zar Jan 27 '19 at 00:51
2

I managed to get this working with the following code:

    (defun start-python ()
      (let ((process 
         (sb-ext:run-program "/usr/bin/python3" nil
                     :output :stream
                     :input :stream
                     :wait nil
                     :pty t
                     :error *standard-output*)))
        process))

    (defun read-until-newline (process)
      (let ((r ""))
        (loop for c = (read-char-no-hang (sb-ext:process-pty process))
           do (progn
            (if (or (not c) (char= c #\newline))
            (return-from read-until-newline r)
            (setf r (concatenate 'string r (format nil "~c" c))))))))

    (defun print-all-output (process &key (discard nil))
      (sleep 0.1)
      (loop 
         do (progn
          (if (listen (sb-ext:process-pty process))
              (if (not discard)
                  (print (read-until-newline process))
                  (read-until-newline process))
              (return)))))

    (defun send-to-python (process str)
      (format (sb-ext:process-pty process) str)
      (finish-output (sb-ext:process-pty process)))

    (defun test-process-stream ()
      (let* ((process (start-python)))
        (print-all-output process :discard t) ;;discard banner message
        (send-to-python process "X=[1,2,3,4,5]~%print(X[:2],X[2:])~%X~%")
        (print-all-output process)
        (sb-ext:process-close process)
        ))

Thanks a lot to @jkiiski for assisting me with debugging this piece of code. The trick was to use :pty as an argument to run-program and then use the stream (sb-ext:process-pty process) to communicate to the process. After that and doing (finish-output (sb-ext:process-pty process)) flushes our input to the program. Then it is crucial to wait a little so that the subprocess can accumulate output. After a (sleep 0.1), (listen ...) will be able to tell if there is output waiting. Then its just a loop with (read-char-no-hang) to read the characters until there are none left. I separated the output by a newline as seen in (read-until-newline) The above code yields the following output:

">>> [1, 2] [3, 4, 5]^M" 
">>> [1, 2, 3, 4, 5]^M" 
">>> "

Any subsequent call to (print-all-output process) will print the output of the program incrementally.

Gordon Zar
  • 103
  • 1
  • 8