2

I am developing a GUI to launch an external long-term running background program. This background program can be given input command via stdin and use stdout and stderr to keep printing out output and error messages. I use a wx.TextCtrl object inside the GUI to give input and print output. My current code is as follows, which is mainly inspired by the "how to implemente a shell GUI window" post: wxPython: how to create a bash shell window?

However, my following code uses "buffer previous output" approach, i.e., I use a thread to buffer the output. The buffered transaction output could be only rendered when I give next input command and press "return" button. Now, I want to see the output messages timely, therefore I want to have the feature that "the output can always be spontaneously printed out (directly flushed out) from the background subprocess and I could also interfere to type some input command via stdin and have the output printed.

class BashProcessThread(threading.Thread):
    def __init__(self, readlineFunc):
        threading.Thread.__init__(self)
        self.readlineFunc = readlineFunc
        self.lines = []
        self.outputQueue = Queue.Queue()
        self.setDaemon(True)

    def run(self):
        while True:
           line = self.readlineFunc()
           self.outputQueue.put(line)
           if (line==""):
            break
        return ''.join(self.lines)

    def getOutput(self):
        """ called from other thread """            
        while True:
            try:
                line = self.outputQueue.get_nowait()
                lines.append(line)
            except Queue.Empty:
                break
        return ''.join(self.lines)

class myFrame(wx.Frame):
    def __init__(self, parent, externapp):
        wx.Window.__init__(self, parent, -1, pos=wx.DefaultPosition)
        self.textctrl = wx.TextCtrl(self, style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE)
        launchcmd=["EXTERNAL_PROGRAM_EXE"]
        p = subprocess.Popen(launchcmd, stdin=subprocess.PIPE, 
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        self.outputThread = BashProcessThread(p.stdout.readline)
        self.outputThread.start()
        self.__bind_events()         
        self.Fit()

    def __bind_events(self):
        self.Bind(wx.EVT_TEXT_ENTER, self.__enter)

    def __enter(self, e):
        nl=self.textctrl.GetNumberOfLines()
        ln =  self.textctrl.GetLineText(nl-1)
        ln = ln[len(self.prompt):]     
        self.externapp.sub_process.stdin.write(ln+"\n")
        time.sleep(.3)
        self.textctrl.AppendText(self.outputThread.getOutput())

How should I modify the above code to achieve this? Do I still need to use thread? Can I code a thread as follows?

class PrintThread(threading.Thread):
    def __init__(self, readlineFunc, tc):
        threading.Thread.__init__(self)
        self.readlineFunc = readlineFunc
        self.textctrl=tc
        self.setDaemon(True)

    def run(self):
        while True:
            line = self.readlineFunc()
            self.textctrl.AppendText(line)

However, when I tried with the above code, it crashes.

I have errors from Gtk, as follows.

(python:13688): Gtk-CRITICAL **: gtk_text_layout_real_invalidate: assertion `layout->wrap_loop_count == 0' failed
Segmentation fault

or sometimes the error

 (python:20766): Gtk-CRITICAL **: gtk_text_buffer_get_iter_at_mark: assertion `GTK_IS_TEXT_MARK (mark)' failed
Segmentation fault

or sometimes the error

(python:21257): Gtk-WARNING **: Invalid text buffer iterator: either the iterator is uninitialized, or the characters/pixbufs/widgets in the buffer have been modified since the iterator was created.
You must use marks, character numbers, or line numbers to preserve a position across buffer modifications.
You can apply tags and insert marks without invalidating your iterators,
but any mutation that affects 'indexable' buffer contents (contents that can be referred to by character offset)
will invalidate all outstanding iterators
Segmentation fault

or sometimes the error

Gtk-ERROR **: file gtktextlayout.c: line 1113 (get_style): assertion failed: (layout->one_style_cache == NULL)
aborting...
Aborted

or other error message, but different error message each time, really weird!

It seems the wx.TextCtrl or the underlying gui control of GTK+ has some problem with multi-threading. sometimes I do not type any input commands, it also crashes. I search from the certain post on the internet, and looks like it is dangerous to call a GUI control from a secondary thread.

I found my mistake. As pointed out in wxpython -- threads and window events or in chapter 18 of the book "WxPython in action" by Noel and Robin:

The most important point is that GUI operations must take place in the main thread, or the one that the application loop is running in. Running GUI operations in a separate thread is a good way for your application to crash in unpredictable and hard-to-debug ways...

My mistake is I tried to pass the wx.TextCtrl object to another thread. This is not right. I will reconsider my design.

Community
  • 1
  • 1
pepero
  • 7,095
  • 7
  • 41
  • 72

4 Answers4

1

The calling contract for anything in gtk (upon which wxpython is built and the thing which is failing the assertion) application is that you must only modify the gui controls from the Main Gui thread. So the answer for me is simple:

In C#, I needed to do the following (which is an anonymous lambda function, and there's almost certainly a similar library call in wxpython/gtk)

Gtk.Application.Invoke((_,__) =>
{
    //code which can safely modify the gui goes here
});

And that should sort out your assertion problems... it did for me, at least.

Mehul Mistri
  • 15,037
  • 14
  • 70
  • 94
Michael
  • 1
  • 1
  • 1
    Indeed; the method in `wxpython` is `wx.CallAfter` (see [Updating a wxPython progress bar after calling app.MainLoop()](http://stackoverflow.com/q/6943339)). – Martijn Pieters Oct 29 '12 at 15:43
1

The biggest problem is that you can't force the subprocess not to buffer its output, and the standard I/O library of most programs will buffer the output when stdout is a pipe (more accurately, they will go from line-buffering to block buffering). Tools like Expect fix this by running the subprocess in a pseudo-tty, which basically tricks the subprocess into thinking its output is going to a terminal.

There is a Python module called Pexpect, which solves this in the same way that Expect does. I've never used it, so caveat emptor.

Marcelo Cantos
  • 181,030
  • 38
  • 327
  • 365
  • Hi, Marcelo, thank you for your input. The buffering is from my thread, not from the background subprocess program. The background program did not buffer its output messages. Even if the stdout pipe makes block buffer, for my application, it does not matter. I just need a way to print transaction messages pseudo-timely or at least no need to wait next input/"return" button. Besides, as I did it for GUI, I use wx.textctrl to make an interactive input/output with user. Can I use pty/pexpect for this? Do you think it is possible to use one thread (keep printing to wx.TextCtrl) to realize this? – pepero Nov 15 '10 at 12:36
  • HI, Marcelo, thank you for your input. After the correction of my mistake(call GUI from secondary thread), I find the problem comes to what you said, and to have more idea, I create another post. At the end, to acknowledge your detailed insightful answer (I am also very grateful to blaze), I put your answer as accepted. – pepero Nov 17 '10 at 20:32
1

pty.spawn() could be useful. Also you can manually create PTYs with pty.openpty() and pass them to popen as stdin/stdout.

If you have access to text-mode program source, you can also disable buffering there.

blaze
  • 4,326
  • 18
  • 23
  • hi, blaze, thank you for your input. But my problem is to have a embedded cli (that's where wx.textctrl comes for)inside the gui for the user to interact with the background process. I do not know if your answer addresses this point. Also the buffering is from my thread, not from the background program. – pepero Nov 15 '10 at 11:50
  • Are you sure about buffering in your app? Usually this sympthom is a sign of bufferring in child process, not a read buffering. Anyway, if it's readers buffering forget about readline(), use read1() (not read()!) and assemble lines by youself. – blaze Nov 15 '10 at 12:52
  • hi, blaze, I mean the readlinefunc, or the BashProcessThread are created by me to buffer the output of external application (EXTERNAL_PROGRAM_EXE). I use this way to make user interaction. but my way is not desirable. I need some way that could flush the transaction outputs automatically, with no need to wait next input/"return" button. The EXTERNAL_PROGRAM_EXE does not do any buffering. sorry for the misleading, I put some more code to illustrate what I want to ask – pepero Nov 15 '10 at 12:59
  • I don't really understand your problem. Yes, you can just read child's stdout and append it to TextView. If you read with readline(), you will not get any data until newline arrives on stdout. If you need to read data right after they were written, replace BashProcessThread(p.stdout.readline) with BashProcessThread(p.stdout.read1) – blaze Nov 15 '10 at 13:38
  • I just want to know if I could use a separate thread to keep logging out child process's stdout on wx.TextCtrl. I tried with my code above (PrintThread). it does not work. there is no output at all. – pepero Nov 15 '10 at 14:21
  • There is no output or nothing added to TextCtrl? Try printf. May be WX just don't allow adding data to control from arbitrary thread. – blaze Nov 16 '10 at 08:17
  • HI, Blaze, ok, for now, there is some output at TextCtrl, but it crashes soon after some messages or further inputs. I revise my original post with more details. Thanks a lot for your follow up! – pepero Nov 16 '10 at 15:52
1

Cant you just pass the TextCtrl object to your OutputThread and have it directly append text to the output without tying it with the __enter method?

Satwik
  • 111
  • 3
  • hi, Satwik, This solution is not correct. This is a mistake, please see my further comments on the original post. – pepero Nov 16 '10 at 17:28