12

I'm currently passing some input to a process with pexpect with the following code:

p = pexpect.spawn('cat', timeout=5.0 )
p.maxread = 5000
p.setecho(False) # prevent the process from echoing stdin back to us
INPUT_LEN = 1024
p.sendline('a'*INPUT_LEN)
print p.readline() # pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().

When INPUT_LEN < 1024, everything works fine, but for >= 1024 characters, the process does not receive the full input, causing raising a "pexpect.TIMEOUT" error on p.readline().

I've tried splitting my input into pieces smaller than 1024 characters, but this has the same issue:

p = pexpect.spawn('cat', timeout=5.0 )
p.maxread = 5000
p.setecho(False)
INPUT_LEN = 1024
p.send('a'*1000)
p.sendline('a'*(INPUT_LEN-1000))
print p.readline() # pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().

Does anyone know how to make pexpect work with inputs over 1024 characters? I tried looking at the source, but it just seems to be calling os.write(...).

(As a side note, I've noticed the same truncation error occurs when I run "cat" from a shell and attempt to paste in >=1024 characters with "Cmd+V". However, everything works fine if I run "pbpaste | cat" .)

Thanks!

Update: The call to "os.write()" returns 1025, indicating a successful write, but os.read() returns "\x07" (the single character BEL), and then hangs on the next call, resulting in the timeout.

Dividing the os.write() call into two sub-1024 byte write()s, separated by a call to os.fsync(), does not change anything.

tba
  • 6,229
  • 8
  • 43
  • 63
  • `pexpect.spawn` should have a `maxdata` keyword by default set to `2000`, so probably this wont apply to your case, but have you tried increasing it? – Rik Poggi Feb 09 '12 at 21:29
  • Unfortunately that didn't work; see latest edit – tba Feb 09 '12 at 21:33
  • Sorry for the confusion, I wrote maxdata but I meant `maxread`, well it was worth a try I guess. – Rik Poggi Feb 09 '12 at 21:44
  • Digging a little more into the source code I see that it use `pty` and `fcntl`, this may the root of your 1024 buffer limit. Could you provide python version and platform? – Rik Poggi Feb 09 '12 at 22:17
  • python version is "Python 2.7", and platform is OSX 10.6.8 ("Darwin tba.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386") – tba Feb 09 '12 at 22:20

3 Answers3

6

Your problem seems to be MacOS related, take a look at MacOSX 10.6.7 cuts off stdin at 1024 chars.

It basically says that 1024 is your tty buffer limit.

I'm not an expert on Mac OS, but maybe others can give you more informations about this.

Community
  • 1
  • 1
Rik Poggi
  • 28,332
  • 6
  • 65
  • 82
  • Any recommendations on how to work around this at the python level? e.g. by writing 1023 bytes, flushing the buffers, then writing the rest? – tba Feb 11 '12 at 19:33
  • @tba: From what you're explaining the problem seems to be on the read side. So I think you're stack with a pipe approach (as the one that pbpaste have), but I fear that will probably nullify your effort of using pexpect. – Rik Poggi Feb 15 '12 at 09:39
  • Unfortunately, using pipes with python's "subprocess" causes deadlocks: see the "Warning:" about using ".stdin.write" on http://docs.python.org/library/subprocess.html . The suggested workaround is to use communicate(), which waits for the process to terminate. This won't work for me because I need to send and receive info with the process multiple times before it terminates. – tba Feb 16 '12 at 21:31
  • 1
    @tba: If you need to send/recevie back and forth don't use `communicate()`, but don't let that Warning scare you too much: you can use `stdin.write`/`stdout.read`, you "just" need to be careful. On *Unix* you have [`select`](http://docs.python.org/py3k/library/select.html) (take a look at [this answer](http://stackoverflow.com/a/2408670/1132524)), while for a cross platform solution (aka *Windows too*) you'll have to use `Threads` (take a look at [this other answer](http://stackoverflow.com/a/4896288/1132524)). – Rik Poggi Feb 17 '12 at 10:31
  • Thanks Rik. I previously tried stdin.write and stdin.read but encountered deadlocks, hence the switch to pexpect. Any idea how to debug the deadlocks? – tba Feb 20 '12 at 08:45
1

I realise it is very very late, but I am posting a solution for someone who stumbles on to this question with the very same problem (as I did earlier today).

Based on some of the answers/comments, I have written a pexpect like package that uses stdin.write and stdout.read instead of whatever it is that pexpect uses. I have not had a chance to test it would very thoroughly but to this point, it has stood up to the challenge.

You can find the code here: https://github.com/tayyabt/tprocess

Tayyab
  • 422
  • 5
  • 17
1

In my case (Debian Linux) the limit (4096 chars) was related to the canonical processing input mode of the terminal. There are some comments about that in the pexpect documentation.

I solved my problem by turning off canon mode before sending my data:

p.sendline('stty -icanon')
p.sendline('a'*5000)
rtrrtr
  • 563
  • 5
  • 7