EC2 ssh broken pipe terminates running process

Question

I'am using a EC2 instance to run a large job that I estimate to take approx 24 hours to complete. I get the same issue described here ssh broken pipe ec2

I followed the suggestion/solutions in the above post and in my ssh session shell I launched my python program by the following command:

nohup python myapplication.py > myprogram.out 2>myprogram.err

Once I did this the connection remained intact longer than if I didn't use the nohup but it eventually fails with broken pipe error and I'm back to square one. The process 'python myapplication.py' is terminated as a result.

Any ideas on what is happening and what I can do to prevent this from occuring?

score 20 · Accepted Answer · edited Mar 04 '19 at 05:30

20

You should try screen.

Install

Ubuntu:

apt-get install screen

CentOS:

yum install screen

Usage

Start a new screen session by

$> screen

List all screen sessions you had created

$>screen -ls
There is a screen on:
        23340.pts-0.2yourserver    (Detached)
1 Socket in /var/run/screen/S-root.

Next, restore your screen

$> screen -R 23340
$> screen -R <screen-id>

edited Mar 04 '19 at 05:30

baduker

19,152
9
33
56

answered Jun 13 '16 at 18:16

Blake

316
2
5

1

Even with `screen` I get broken pipe. – Vasiliki Jun 07 '21 at 10:56
I also got the broken pipe. But when I re-log in to the ubuntu server, I can still see the screen running. Command : screen -ls. When I entered the screen screen -R XXXX, where XXXX is the screen number, I can see the codes still running. It worked. – Amar nayak Sep 14 '21 at 08:38

score 5 · Answer 2 · answered Aug 12 '18 at 20:31

A simple solution is to send the process to the background by appending an ampersand & to your command:

nohup python myapplication.py > myprogram.out 2>myprogram.err &

The process will continue to run even if you close your SSH session. You can always check progress by grabbing the tail of your output files:

tail -n 20 myprogram.out
tail -n 20 myprogram.err

score 2 · Answer 3 · answered Apr 01 '19 at 12:34

I actually ended up fixing this accidentally with a router configuration, allowing all ICMP packets. I allowed all ICMP packets to diagnose a strange issue with some websites loading slowly randomly, and I noticed none of my SSH terminals died anymore.

I'm using a Ubiquiti EdgeRouter 4, so I followed this guide here https://community.ubnt.com/t5/EdgeRouter/EdgeRouter-GUI-Tutorial-Allow-ICMP-ping/td-p/1495130

Of course you'll have to follow your own router's unique instructions to allow ICMP through the firewall.

EC2 ssh broken pipe terminates running process

3 Answers3

Install

Usage