Python: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 37: invalid start byte

Question

I have a Juniper router and I am trying to automate some of my tasks using Python 3.7.1. My code is running some commands perfectly but some commands are giving errors. Following is my code

import paramiko

def sshConnection(ip,command):
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(ip, port=22, username='username', password='password')
  stdin, stdout, stderr = ssh.exec_command(command)
    output = stdout.readlines()
    return output


ip = '10.1.1.1'
command = 'show interfaces descriptions | match 1000 | no-more'
output = sshConnection(ip,command)
print('\n'.join(output))

Following is the error I am facing

Traceback (most recent call last):
  File "C:\Users\Mu\Documents\Python Code\write_file - Stackoverflow.py", line 16, in <module>
    output = sshConnection(ip,command)
  File "C:\Users\Mu\Documents\Python Code\write_file - Stackoverflow.py", line 10, in sshConnection
    output = stdout.readlines()
  File "C:\Users\Mu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\paramiko\file.py", line 349, in readlines
    line = self.readline()
  File "C:\Users\Mu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\paramiko\file.py", line 334, in readline
    return line if self._flags & self.FLAG_BINARY else u(line)
  File "C:\Users\Mu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\paramiko\py3compat.py", line 156, in u
    return s.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 37: invalid start byte

Other router commands such as 'show version' is working fine and showing correct output. It seems the output contains some characters that are not accepted by UTF-8 encoding. I am unable to find the solution for this problem. Please help me resolve this. Thank you.

Your remote system is not using UTF-8 as the locale. Perhaps prefix your command with `LC_ALL=C` or `LC_ALL=en_US.UTF-8` to force the locale. — Martijn Pieters, Dec 09 '18 at 11:49
Aparently an old bug of Paramiko, see [this](https://github.com/paramiko/paramiko/issues/546). There should be a way to change the encoding or better yet to get bytes, but aparently there is none. Very bad design decision. The last "[solution](https://github.com/paramiko/paramiko/pull/1241/commits/5a0cd44ab6c3cedd9d6e5501cde6d9fdef62c36f)" in the linked thread is even worse as it would silently remove invalid Unicode data. — , Dec 09 '18 at 11:58

score 0 · Answer 1 · 2018-12-09T14:10:44.587

The problem arises because by default the readline method of class BufferedFile in Paramiko decodes as UTF-8. And there seems to be no documented way to change that.

However, Paramiko works with bytes internally. The readline method in file.py terminates with return line if self._flags & self.FLAG_BINARY else u(line). So, if there is a way to set this FLAG_BINARY, there will be no translation. The internal _set_mode in the same class can set this flag. The flag is acknowledged in the doc, but not the way to set it.

I have not been able to find a correct way to set flags when opening the connection, but it might be possible to set the flag afterwards. If so, readline will return bytes. Maybe stdout._set_mode("rb") before any read would just work. Note that the starting underscore means the function is internal/undocumented and might change without notice.

Another way would be to use read instead of readline, but you will have to deal with line endings.

Python: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 37: invalid start byte

1 Answers1