2

16.6.21 UPDATE

I Found the cause of this problem. It was my VPN Client (CISCO Anyconnect). There were some Packet Filters installed with the Client and therefore not all Packages were sent properly to the server.

QUESTIONE

I already searched google for hours to find a solution for my problem, but none of the written solutions helped me.

Last Friday, I did a complete fresh install of Mac OS Big Sur on my Mac Mini (2020). I was setting up my development environment to my needs and it looked like it was working really good until yesterday.

Yesterday was trying to push some commits to our GitLab server... while pushing a error came up

client_loop: send disconnect: Broken pipe
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

I tried to fix the error by setting several arguments in my ./ssh/config file, like:

Host *
  ServerAliveInterval 60
  ServerAliveCountMax 5
  IPQoS=throughput

and many more.

3 hours later I was really annoyed and pushed the commits via https not via ssh anymore.

A bit later I had to push a local WordPress installation to our staging environment. Suddenly nearly the same error occurred again

rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(823) [sender=3.2.3]

The strange thing is:
While downloading via git clone or rsync (both via ssh) everything works well. Also connecting via ssh to the server works really well BUT! when I begin to upload a larger amount of files I'm getting those errors (also tried with a plain rsync command).

Could this be caused by different OpenSSH versions on our servers (OpenSSH_7.2p2 Ubuntu-4ubuntu2.10, OpenSSL 1.0.2g 1 Mar 2016) and my machine (OpenSSH_8.6p1, OpenSSL 1.1.1k 25 Mar 2021)?

Anyone has any advices for me?

themca
  • 323
  • 3
  • 15
  • This sort of thing should not happen at all, but if you do need to use the various keep-alives, a ServerAliveInterval of 5 is, um, "aggressive". :-) I had to set mine to once a minute (60 seconds) when going through some NAT-boxes sometimes, and that was sufficient. – torek May 13 '21 at 21:10
  • This was just an example, what i have tried. I removed it again. – themca May 14 '21 at 03:57
  • I just tried ServerAliveInterval with 1 -> no success – themca May 14 '21 at 05:59
  • 2
    Yes, you're not being bitten by a NAT box dropping its translation table entry here (that's one of the things, perhaps even the main one, that the ServerAliveInterval combats). It's not clear why your data transfers are being interrupted here. A proper diagnosis probably requires doing data capture from both ends, although you might get lucky enough to spot an error even if you can only use tcpdump or wireshark or whatever from your end. – torek May 14 '21 at 13:46

3 Answers3

2

It works again. I don't know how or why, but it works. I did a clean reinstall of macos (again) and now it seems to work.

Dharman
  • 30,962
  • 25
  • 85
  • 135
themca
  • 323
  • 3
  • 15
2

I still can't pinpoint what exactly it was, but i noticed the ssh disconnect issues after the latest Update. I could pinpoint it down to the Macbook SSH Libraries , because i did the following steps:

  • Connected from another another source (Terminus on iPhone, another Mac and Debian running on NUC) within the network without any problems (same ssh target destinations in all source device examples)
  • Disabled temporarily all network adblocking, firewall and filtering
  • added a wired connection to the "problem macbook"
  • tried all the ssh config options below , with the default openSSH and LibreSLL Version from MacOS
  • Connected to different Target Servers from the "problem Mac" and the disconnects appeared suddenly everywhere with different timings (only with the problem Mac).

I assume, that the problem lies with the default Big Sur OpenSSH and LibreSSL Version. After i installed OpenSSH with brew install openssh with the following config, everything works like before.

I hope it also solves your problem, because it took me couple of days :(

My Machine

  • MacOS 11.3.1
  • MacBook Pro Retina 15, Mid 2015
  • Intel i7

Steps to fix:

  1. brew install openssh
  2. Add config to ~/.ssh/config
Host *

    IPQoS none
    TCPKeepAlive no
    ServerAliveInterval 60
    ServerAliveCountMax 5

Play around with the .ssh/config config values, to match your needs and don't make them to aggressive

ssh -V after brew install openssh: OpenSSH_8.6p1, OpenSSL 1.1.1k 25 Mar 2021

Klizzy
  • 83
  • 1
  • 7
  • 1
    I also tried to reinstall openssh via homebrew, but the error still occured in my case. I also trid nearly the same settings in my ssh config file – themca May 24 '21 at 08:04
  • i also had to play around with the exact config values in `.ssh/config` for it to get to work. Anyway, good that it works now! – Klizzy May 24 '21 at 21:42
1

Could this be caused by different openssh versions on our servers

Check if using an old PEM SSH key format would work better, considering OpenSSH 7.8 has switch to a new format by default:

ssh-keygen -t rsa -P "" -m PEM -f ~/.ssh/key2

Register that key2.pub to your GitLab server, and try at least a ssh -Tv git@myGitLabServer

Check also, as mentioned here, your SSH confog for any non-standard directive, like RemoteCommand

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • I copied my id_ed25519 and id_ed25519.pub from my old installation... this should be the old pem or not? – themca May 14 '21 at 07:01
  • @You can see it in checking the content of the private key: `BEGIN OPENSSH PRIVATE KEY` means new format, `BEGIN RSA PRIVATE KEY` means old format. – VonC May 14 '21 at 07:05
  • it is ```BEGIN RSA PRIVATE KEY``` so old pem – themca May 14 '21 at 07:10
  • @themca For RSA key, yes: but that is not the one you are using. You are using a `-t ed25519` one. – VonC May 14 '21 at 07:56
  • Hi Sorry for the late response, also this didn't work :( i created a new key with pem 1 – themca May 16 '21 at 06:07