Why does Paramiko hang if you use it while loading a module?

Question

Put the following into a file hello.py (and easy_install paramiko if you haven't got it):

hostname,username,password='fill','these','in'
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect(hostname=hostname, username=username, password=password)
i,o,e = c.exec_command('ls /')
print(o.read())
c.close()

Fill in the first line appropriately.

Now type

python hello.py

and you'll see some ls output.

Now instead type

python

and then from within the interpreter type

import hello

and voila! It hangs! It will unhang if you wrap the code in a function foo and do import hello; hello.foo() instead.

Why does Paramiko hang when used within module initialization? How is Paramiko even aware that it's being used during module initialization in the first place?

What happens if you do the same ssh call on the commandline? — barneytron, Jan 14 '09 at 15:40
works fine. when you run this code from within a module does it work fine for you? — Michael Gundlach, Jan 14 '09 at 15:43
When you're running it via the command line, are you putting it in line by line or all at once? — jacobangel, Jan 14 '09 at 15:47
@jdangel: i expect you mean when i'm running it via the interpreter; on the command line it's just 'ssh user@host ls /'. In the interpreter, I'm typing the keystrokes myself, so it's line-by-line. — Michael Gundlach, Jan 14 '09 at 15:50
What happens when you copy the whole thing and paste it into the interpreter all at once, rather than line-by-lining it. — jacobangel, Jan 14 '09 at 16:13
jdangel, I have changed the question (reload the page) now that I have figured out that it's hanging only when called during module initialization. Copy-and-paste all-at-once into the interpreter has the same effect as typing it in. — Michael Gundlach, Jan 14 '09 at 16:15
Paramiko does not hang, it's the connect() call which hangs. — Vinko Vrsalovic, Jan 14 '09 at 16:37
Related question: http://stackoverflow.com/q/13155890/109807 — fmarc, Feb 22 '13 at 10:13

JimB · Accepted Answer · 2009-01-16T17:41:38.237

18

Paramiko uses separate threads for the underlying transport. You should never have a module that spawns a thread as a side effect of importing. As I understand it, there is a single import lock available, so when a child thread from your module attempts another import, it can block indefinitely, because your main thread still holds the lock. (There are probably other gotchas that I'm not aware of too)

In general, modules shouldn't have side effects of any sort when importing, or you're going to get unpredictable results. Just hold off execution with the __name__ == '__main__' trick, and you'll be fine.

[EDIT] I can't seem to create a simple test case that reproduces this deadlock. I still assume it's a threading issue with import, because the auth code is waiting for an event that never fires. This may be a bug in paramiko, or python, but the good news is that you shouldn't ever see it if you do things correctly ;)

This is a good example why you always want to minimize side effects, and why functional programming techniques are becoming more prevalent.

edited Jan 16 '09 at 17:41

answered Jan 16 '09 at 16:01

JimB

104,193
13
262
255

5

I believe this is indeed a bug (or at least flaw) in paramiko. – Tony Meyer Feb 03 '09 at 22:16
1

Thank you, Jim! (Sorry I just noticed your answer, months later -- StackOverflow doesn't have a great notification system yet!) – Michael Gundlach May 06 '09 at 12:34
Can u point to a sample showing usage of __name__ == '__main__' ? – Tectrendz Mar 09 '15 at 00:53
@Tectrendz: look at pretty much any python script: http://stackoverflow.com/questions/419163/what-does-if-name-main-do – JimB Mar 09 '15 at 13:45
+1 Actually the import lock is more of a [problem when waiting for threads](https://stackoverflow.com/a/46354248/5896591) than when spawning them. If you follow best practice and do the waiting (such as `signal.pause()`) in `main()` rather than via `import` then you should be fine. – personal_cloud Sep 21 '17 at 23:22
This also applies to packages. If you package does things in `__init__.py` that might use paramiko, things will go poorly. Just make a `__main__.py` as well and do everything in that instead, and paramiko will work properly. – jeremysprofile May 14 '19 at 17:58

tintin · Answer 2 · 2015-09-30T21:09:28.957

As JimB pointed out it is an import issue when python tries to implicitly import the str.decode('utf-8') decoder on first use during an ssh connection attempt. See Analysis section for details.

In general, one cannot stress enough that you should avoid having a module automatically spawning new threads on import. If you can, try to avoid magic module code in general as it almost always leads to unwanted side-effects.

The easy - and sane - fix for your problem - as already mentioned - is to put your code in a if __name__ == '__main__': body which will only be executed if you execute this specific module and wont be executed when this mmodule is imported by other modules.
(not recommended) Another fix is to just do a dummy str.decode('utf-8') in your code before you call SSHClient.connect() - see analysis below.

So whats the root cause of this problem?

Analysis (simple password auth)

Hint: If you want to debug threading in python import and set threading._VERBOSE = True

paramiko.SSHClient().connect(.., look_for_keys=False, ..) implicitly spawns a new thread for your connection. You can also see this if you turn on debug output for paramiko.transport.

[Thread-5 ] [paramiko.transport ] DEBUG : starting thread (client mode): 0x317f1d0L

this is basically done as part of SSHClient.connect(). When client.py:324::start_client() is called, a lock is created transport.py:399::event=threading.Event() and the thread is started transport.py:400::self.start(). Note that the start() method will then execute the class's transport.py:1565::run() method.
transport.py:1580::self._log(..) prints the our log message "starting thread" and then proceeds to transport.py:1584::self._check_banner().
check_banner does one thing. It retrieves the ssh banner (first response from server) transport.py:1707::self.packetizer.readline(timeout) (note that the timeout is just a socket read timeout), checks for a linefeed at the end and otherwise times out.
In case a server banner was received, it attempts to utf-8 decode the response string packet.py:287::return u(buf) and thats where the deadlock happens. The u(s, encoding='utf-8') does a str.decode('utf-i') and implicitly imports encodings.utf8 in encodings:99 via encodings.search_function ending up in an import deadlock.

So a dirty fix would be to just import the utf-8 decoder once in order to not block on that specifiy import due to module import sideeffects. (''.decode('utf-8'))

Fix

dirty fix - not recommended

import paramiko
hostname,username,password='fill','these','in'
''.decode('utf-8')  # dirty fix
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect(hostname=hostname, username=username, password=password)
i,o,e = c.exec_command('ls /')
print(o.read())
c.close()

good fix

import paramiko
if __name__ == '__main__':
    hostname,username,password='fill','these','in'
    c = paramiko.SSHClient()
    c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    c.connect(hostname=hostname, username=username, password=password)
    i,o,e = c.exec_command('ls /')
    print(o.read())
    c.close()

ref paramiko issue tracker: issue 104

thx, it was such a pain not being able to figure why it hanged. For me good fix works but not dirty one. — garg10may, Jul 11 '16 at 11:57

score 0 · Answer 3 · answered Apr 05 '16 at 03:36

"".decode("utf-8") didn't work for me, I ended up doing this.

from paramiko import py3compat
# dirty hack to fix threading import lock (issue 104) by preloading module
py3compat.u("dirty hack")

I have a wrapper for paramiko with that implemented. https://github.com/bucknerns/sshaolin

Why does Paramiko hang if you use it while loading a module?

3 Answers3

Linked