Identify the cause of \r artifacts when reading from a serial terminal?

Question

I'm trying to diagnose what is the cause of \r artifacts when reading from a serial terminal.

The following code can provoke the issue. There is an embedded Linux device connected to the physical end of the uart cable

import serial                      

ser = serial.Serial( 
    port='/dev/ttyUSB0', 
    baudrate=115200,  
    timeout=5) 

ser.reset_input_buffer()
ser.reset_output_buffer()

b_NEW_LINE_WRITTEN = b'\n'
b_alphabet = b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q R r S s T t U u V v W w X x Y y Z z'
ser.write(b_alphabet + b_NEW_LINE_WRITTEN)

raw_line = ser.readline()
print(raw_line)                                                                                  
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q\rq R r S s T t U u V v W w X x Y y Z z\r\n'

The print the read data has an \r around the letter q.

When running a terminal emulator on top of /dev/ttyUSB0 the line starts wrapping around itself after 80 characters, similarly to this question. This line wrapping seems to be a manifestation of the /r that I get when directly reading with the example python code.

When connected with a terminal emulator (picocom) this problem gets solved by running:

shopt -s checkwinsize
resize

there are no line wraps after that. I'm trying to understand why this is happening or where (is the host where /dev/tttyUSB0 what adds them or are they added by the embedded device being accessed), and how to workaround it without having to run external commands.

The following candidates could be causing the behavior:

driver settings for /dev/ttyUSB0 on the shot ps
driver settings for /dev/S0 on the target embedded device
shell on the target device

Setting /dev/ttyUSB0

Trying to modify /dev/ttyUSB0 in separate terminal while the device is consumed by the python script doesn't show any changes.

# Separate terminal on the host
stty -F /dev/ttyUSB0  cols 100
stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0  -a
speed 115200 baud; rows 80; columns 100; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^A; eol = <undef>; eol2 = <undef>; swtch = <undef>;
start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O;
min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany
-imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke
-flusho -extproc


#ipython
...
ser.write(b_alphabet + b_NEW_LINE_WRITTEN)
raw_line = ser.readline()
print(raw_line)                                                                                  
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q\rq R r S s T t U u V v W w X x Y y Z z\r\n'

Setting /dev/S0

Setting the target tty device also doesn't influence the artifacts existence.

stty_cmd_set = b'stty cols 200'
ser.write(stty_cmd_set + b_NEW_LINE_WRITTEN)
ser.reset_input_buffer()
ser.reset_output_buffer()
stty_cmd_confirm = b'stty -a'
ser.write(stty_cmd_confirm + b_NEW_LINE_WRITTEN)

# After reading a few lines there is a confirmation that the tty device on the target has indeed been set to 200 
print(ser.readline())                           
b'speed 115200 baud; rows 56; columns 200; line = 0;\r\n'
ser.reset_input_buffer()
ser.reset_output_buffer()

ser.write(b_alphabet + b_NEW_LINE_WRITTEN)

raw_line = ser.readline()
print(raw_line)                                                                                  
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q\rq R r S s T t U u V v W w X x Y y Z z\r\n'

For example a workaround would be to somehow set a fixed amount of 200 columns before reading so that the serial terminal stops trying to be smart.

With Linux your application does not have access to the *"serial port"*. Rather you are reading from a serial terminal, which is denoted by the device name **/dev/tty...**. Your application is several layers removed from the *"serial port"*. See [Linux serial drivers](http://www.linux.it/~rubini/docs/serial/serial.html). With Python and and Pyserial you've added more layers of abstraction and loss of control. — sawdust, Feb 19 '19 at 21:47
Try an experiment with a terminal emulation program, such as **minicom**. Use the Send ASCII command of a file containing the same text string to the remote device, and see what comes back. — sawdust, Feb 19 '19 at 23:14
In serial communication there is a very, very long-established convention of using the pair of characters `\r\n` -- an ASCII `CR` (Carriage Return) followed by an ASCII `LF` (Line Feed) -- to indicate the end of a line of text. If you don't like the `\r` that the device sends, then just have your program discard it. (You're lucky that the device did not require you to send it `\r\n` as a line terminator. Apparently it was willing to accept just a `\n` as an end-of-line marker in incoming data.) — ottomeister, Feb 19 '19 at 23:48
What are your serial line settings? One of the serial settings could be concerting a byte. Check with stty — rm5248, Feb 22 '19 at 12:44
@TheMeaningfulEngineer Depending on the serial port, it could be on either side. One side could have `-ocrnl` set(which maps `\r` to `\n` on output), or `-icrnl` which does the same thing except on input. This answer may help you: https://stackoverflow.com/questions/48679189/usb-modem-is-echoing-back-wrong-characters/49051620#49051620 — rm5248, Feb 24 '19 at 16:03
Reading between the lines, you are running something like a Linux shell on your embedded device. This will have the same set of Linux app/TTY/driver hierarchy as your host and is defaulting to cooked mode processing of the input that it receives from your application. Switching to raw input on both your host and embedded shell should fix it. — Peter Brittain, Feb 26 '19 at 23:57
@PeterBrittain Your comment seems to be the closest to an explanation of whats going on. If you frame it as an answer I will award it the bounty in lack of better answers — TheMeaningfulEngineer, Feb 28 '19 at 21:28
@TheMeaningfulEngineer Done. Let me know if you'd like more info. — Peter Brittain, Feb 28 '19 at 21:58

score 1 · Accepted Answer · answered Feb 28 '19 at 21:53

Reading between the lines, you are running something like a Linux shell on your embedded device.

This will have the same set of Linux app/TTY/driver hierarchy as your host and is defaulting to cooked mode processing of the input that it receives from your application. This is why running the command to change the number of columns (on your embedded application) works. It is telling the line discipline in that device to treat the screen as being 200 columns wide (and so the line-editing logic doesn't need to split the line).

Switching to raw input on both your host and embedded shell should fix it.

If you'd like to know more details of how Linux terminals handle input (and echoing of input back to the output stream), see https://www.linusakesson.net/programming/tty/

Setting the correct settings on the target (embedded device's) tty options solve the problem. Eventually I opted for no echoing which simplifies the problem all together. This meant callin `stty raw -echo -echoe -echok -echoctl -echoke` on the target (through the python script) — TheMeaningfulEngineer, Mar 12 '19 at 17:45

score 0 · Answer 2 · answered Feb 23 '19 at 20:31

It is not clear what is actually happening but this is solution that seems to work.

It depends on existence of shopt -s checkwinsize and resize on the target board so it isn't a generic enough solution to be the accepted answer.

Also it doesn't provide an insight on how to apply a not runtime fix (by setting the driver defaults or some config in bash).

import serial             
import time         

ser = serial.Serial( 
    port='/dev/ttyUSB0', 
    baudrate=115200,  
    timeout=5) 

b_NEW_LINE_WRITTEN = b'\n'
b_NEW_LINE_READ = b'\r\n'
b_alphabet = b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q R r S s T t U u V v W w X x Y y Z z'

ser.write(b_alphabet + b_NEW_LINE_WRITTEN) 
print(ser.readline())
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q\rq R r S s T t U u V v W w X x Y y Z z\r\n'

shopt_cmd = b'shopt -s checkwinsize' 
ser.write(shopt_cmd + b_NEW_LINE_WRITTEN)
ser.readline()
ser.readline()

resize_cmd = b'resize' 
ser.write(resize_cmd + b_NEW_LINE_WRITTEN)
ser.readline()
ser.readline()

stty_cmd_set = b'stty cols 200'
ser.write(stty_cmd_set + b_NEW_LINE_WRITTEN)
ser.readline()
ser.readline()

ser.write(b_alphabet + b_NEW_LINE_WRITTEN) 
print(ser.readline())
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q R r S s T t U u V v W w X x Y y Z z\r\n'

ser.reset_output_buffer()
stty_cmd_set = b'stty cols 5'
ser.write(stty_cmd_set + b_NEW_LINE_WRITTEN)
ser.readline()
ser.readline()


ser.write(b_alphabet + b_NEW_LINE_WRITTEN) 
print(ser.readline())
# A a B \r b C c\rc D d \r E e F\rF f G \r g H h\rh I i \r J j K\rK k L \r l M m\rm N n \r O o P\rP p Q \r q R r\rr S s \r T t U\rU u V \r v W w\rw X x \r Y y Z\rZ z\r\n'

score -1 · Answer 3 · answered Feb 28 '19 at 19:36

\r q is happening because your screen ends near q so it is moved to next line its carriage return of Unix. unix carriage return can be used as new line ,go to next line end of line both . Replace \r with empty. Try auto wrap off

\r (Carriage Return) → moves the cursor to the beginning of the line without advancing to the next line \n (Line Feed) → moves the cursor down to the next line without returning to the beginning of the line — In a *nix environment \n moves to the beginning of the line. \r\n (End Of Line) → a combination of \r and \n

Identify the cause of \r artifacts when reading from a serial terminal?

3 Answers3