13

UPDATE - It happened AGAIN!!!

Ok, so this just happened AGAIN! MAN is this frustrating!!! But this time I dug a little deeper and found that for some reason, my private keys were unloaded.

Specifically, when I call this...

ssh-add -l -E md5

I get this...

The agent has no identities.

However, if I then run this...

ssh-add /Users/[username]/.ssh/[private key]

Everything works again! SourceTree connects just as it's supposed to.

The question is why do I have to keep running the 'ssh-add' command?! Why does it keep forgetting my keys?!

As mentioned elsewhere, not sure if this makes a difference, but I'm running a MacBook Pro with High Sierra, although this happens on Sierra too.

Original Post:

This one has me both stumped, and annoyed as heck!! SourceTree (or ssh or something!) keeps forgetting/not applying/ignoring my SSH keys every day! I don't know why.

Note: Updated to use BitBucket's info instead of GitHub.

Here's the relevant portion of my current config file

# --- Sourcetree Generated ---
Host MarqueIV-Bitbucket
    HostName bitbucket.org
    User MarqueIV
    PreferredAuthentications publickey
    IdentityFile /Users/MarqueIV/.ssh/MarqueIV-Bitbucket
    UseKeychain yes
    AddKeysToAgent yes
# ----------------------------

Here's a 'ls' of my ~/.ssh folder (truncated)

-rw-r--r--@ 1 MarqueIV  staff   421 Dec 14 11:25 config
-rw-r--r--@ 1 MarqueIV  staff  1808 Dec  9 14:20 known_hosts
-rw-------  1 MarqueIV  staff  3243 Dec  6 23:33 MarqueIV-Bitbucket
-rw-r--r--  1 MarqueIV  staff   781 Dec  6 23:33 MarqueIV-Bitbucket.pub

Here's my known_hosts file (keys redacted)

bitbucket.org,104.192.143.3 ssh-rsa [redacted]
bitbucket.com,104.192.143.9 ssh-rsa [redacted]
104.192.143.2 ssh-rsa [redacted]

Note: Not sure if this matters, but you can see lines 1 and 2 seem to be duplicates.

And here's the output of ssh -Tv git@bitbucket.org

OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /Users/MarqueIV/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to bitbucket.org port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/MarqueIV/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version conker_1.0.315-a08d059 app-153
debug1: no match: conker_1.0.315-a08d059 app-153
debug1: Authenticating to bitbucket.org:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /Users/MarqueIV/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:h+6zCXg32Uw4fYxSUMwYst3zee8RFb9Z47H1QUTz58E /Users/MarqueIV/.ssh/MarqueIV-GitHub
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/MarqueIV/.ssh/id_rsa
debug1: Trying private key: /Users/MarqueIV/.ssh/id_dsa
debug1: Trying private key: /Users/MarqueIV/.ssh/id_ecdsa
debug1: Trying private key: /Users/MarqueIV/.ssh/id_ed25519
debug1: No more authentication methods to try.
git@bitbucket.org: Permission denied (publickey).

See how it doesn't appear to be applying the key defined in config and known_hosts? Seems like that would be a problem, no?

Note: I'm using macOS Sierra, and I have updated my path to include /usr/bin before /usr/local/bin as outlined here. If I don't do that, I get an error saying ssh doesn't recognize UseKeychain yes in the config.

As a result, almost daily here's the routine I have to go through. I'll use GitHub as my example.

  1. I open SourceTree and try to pull the latest from GitHub. It fails with a 'git@github.com: Permission denied (publickey).' message.

  2. I remove my GitHub account from SourceTree.

  3. I delete both the public and private keys for GitHub from the .ssh folder on my machine.

  4. I go to GitHub and delete my old public key from my account.

  5. Back in SourceTree, I log into GitHub again using my username and password.

  6. Once logged in, using SourceTree, I generate a new SSH key-pair for GitHub.

  7. I copy my public key to the SSH area in my GitHub account. (Sometimes I notice it adds it for me, but I like to be safe and double-check.)

  8. Now I can push and pull again just fine.

I go home for the day and log on at home. It fails again. Repeat all of the steps above.

How do I get SourceTree/ssh/whatever to remember my da*n keys so I don't have to keep doing this every time I change locations?! What step am I missing???

So can anyone offer suggestions on how to make my SSH keys 'stick'?

Mark A. Donohoe
  • 28,442
  • 25
  • 137
  • 286

3 Answers3

12

Ok, I think I have all the parts figured out.

To help people get what they're after, here's the solution right up front:

  1. Make sure the keys you want to work with are secured with a password or else they will not add to Keychain.
  2. Make sure the keys you want to auto-load are configured in your config file and have the UseKeychain and AddKeysToAgent set
  3. Make sure to connect to those config-defined hosts from terminal!!
  4. Create a LaunchAgent to run ssh-add -A to automatically reload your Keychain-stored keys

Ok now that you know what to do, here's the 'why'.

The Meat

As explained in my question, lately, whenever I rebooted, I (incorrectly) thought the system was losing my private keys. It wasn't losing them, it was just ignoring them. This was because of a bunch of things that all came together in a perfect storm of confusion for someone like me who never uses the terminal for GIT.

  1. In the latest versions of macOS, Apple changed how it's implemented SSH so that It better matches the implementation of OpenSSH
  2. As a result of #1, ssh-add -K [privateKey] no longer stores the keys in the keychain (it essentially ignores the -K.) While they do get added to ssh for that session--and thus your connections will work again--as soon as you reboot, they will no longer work. (This is what's been driving me mad!)
  3. Even for keys that are in the Keychain, Apple no longer loads them automatically meaning you manually have to call ssh-add -A from the terminal to reload them every time you reboot.
  4. However, as stated above, ssh-add -K [privateKey] no longer adds the keys to keychain, so ssh-add -A is pointless anyway for keys added that way. (They can be added to Keychain another way. More on that in a minute.)

Because of the above, any keys manually added with the -K option prior to upgrading your OS will still be in your Keychain. However, keys added after Apple's change are not.

That said, Apple does still have the ability to store keys in the keychain, but not from ssh-add anymore. It now only works for hosts defined in your config file.

This is now the only way to get your keys in your Keychain.

Again, here's my config:

Host MarqueIV-Bitbucket
    HostName bitbucket.org
    User git <-- Make sure this is 'git', not what SourceTree puts here
    PreferredAuthentications publickey
    IdentityFile /Users/MarqueIV/.ssh/MarqueIV-Bitbucket
    UseKeychain yes    <-- Note here
    AddKeysToAgent yes <-- ...and here

But wait! If you look in my config file, it does have those values set! So why didn't it work?

Two things.

  1. I don't use Terminal, ever. I use SourceTree which doesn't use the host entry in that file
  2. Apple technically only adds (and stores) the key on demand when that host is accessed, not when the file is (re)loaded meaning unless you explicitly access that host, nothing happens.

In my case, adding the keys via SourceTree would add them for that initial session, but as soon as I rebooted, the keys would again not be loaded and thus all connections would fail. ssh-add -A wouldn't fix it either because again, they weren't in the keychain, meaning I was back to manually adding each one on the command line with ssh-add [privateKey]. What a pain!!

Then it occurred to me... if that setting is in the config file, and that entry can be used from the command line, then shouldn't I be able to directly connect to that host, thus adding the keys to my keychain? Let's find out! I typed this...

ssh -T MarqueIV-BitBucket

And sure enough, not only was the key added to ssh, but it was also again added to my Keychain! I confirmed this by checking Keychain Access directly and it was there.

To further test, I ran this...

ssh-add -D

which deleted all my keys. Sure enough, my SourceTree connections all failed again.

Then I ran this...

ssh-add -A

and the keychain-stored keys magically came back and connections started working again! WOOT!!

Ok, almost there, but not quite! What about reboots? Again, Apple no longer automatically loads keys from Keychain. Sure, it's just a quick jaunt now to terminal to type ssh-add -A, but again, I shouldn't have to do that!

Enter LaunchAgents!

LaunchAgents and LaunchDaemons are beyond the discussion of this post, but in short, they allow you to execute something on reboot, on a schedule, when changes happen to the system, etc.

In my case, I wanted something that would run when I logged onto my mac, so a LaunchAgent was the best choice.

Here's my plist defining how to execute ssh-add --apple-load-keychain every time I logged into my account (even if I never touched Terminal):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>ssh-add-a</string>
    <key>ProgramArguments</key>
    <array>
        <string>ssh-add</string>
        <string>--apple-load-keychain</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

Since I only want this for my particular user, I stored it here:

~/Library/LaunchAgents/com.mydomain.ssh-add-a.plist

Then I ran this to make it executable

chmod +x ~/Library/LaunchAgents/com.mydomain.ssh-add-a.plist

Note: Make sure to change the permissions to allow it to be executed, or it won't start!

Sure enough, on reboot, all my keys came back and were active! Connections all worked, children played, grown men cried, and it was a good day in the Code-dom!

So to recap:

  1. Apple changed how their SSH worked
  2. Keys were no longer added to Keychain from the command line
  3. Apple also no longer auto-loaded keys that were stored in the keychain
  4. Using terminal to connect to config-defined hosts fixed #2
  5. Using a LaunchAgent fixed #3

Hope this helps! Now time to go get some Icy-Hot for my sore shoulder that I've been patting myself on so hard for figuring this all out! ;)

Mark A. Donohoe
  • 28,442
  • 25
  • 137
  • 286
  • Thanks. Just to keep it short for those on a hurry just use on terminal ssh-add -A It worked for me – Xaren Jul 10 '18 at 20:53
  • 1
    That's missing the details that you need to have the config file properly set up *and* you have to connect from terminal first or else your command won't work. Plus, without the launch agent, you'll have to run that command on every log-out/reboot of your machine. That's why I included those details at the beginning, specifically in a section for those who don't want to read all of it. – Mark A. Donohoe Jul 11 '18 at 21:22
  • 1
    Thanks, very helpful! Just wanted to add that if you have multiple users with different credentials, you can use the `MarqueIV-Bitbucket` host to use the credentials associated to it. So your remote url would be `git@MarqueIV-Bitbucket:bla/repo.git` instead of `git@bitbucket.org:bla/repo.git`. – Pin Nov 09 '20 at 12:02
1

First, install the latest Git for Windows release (the 2.15.1.2 one, by simply uncompressing the archive PortableGit-2.15.1.2-64-bit.7z.exe anywhere you want, and adding it to your PATH)

Second, make sure your SourceTree is using the System Git

Third, test in command-line if your ssh key is recognized:

ssh -T git@github.com

Hi username! You've successfully authenticated, 
but GitHub does not provide shell access.

Finally, make sure that SourceTree / Tools / Option uses as SSH client the OpenSSH one (not putty)

Then SourceTree should have nop problem reusing your ssh key.

From your logs, the ~/.ssh/config generate is wrong: it mentions as User your username.
Whenever you establish an SSH connection to github.com/bitbucket.org, it is never as "you". It is always as git.

Host MarqueIV-Bitbucket
    HostName bitbucket.org
    User MarqueIV
    PreferredAuthentications publickey
    IdentityFile /Users/MarqueIV/.ssh/MarqueIV-Bitbucket
    UseKeychain yes
    AddKeysToAgent yes

Test it with ssh -Tv MarqueIV-Bitbucket

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • Thanks for replying. Couple quick things though. One, I’m using Mac. Two, that command you said works fine until it doesn’t. That’s the problem. It just stops working for some reason and I can’t exactly figure out why. More importantly, I can’t get it started working again without re-creating everything from scratch. – Mark A. Donohoe Dec 14 '17 at 05:43
  • @MarqueIV ssh -Tv git@github.com will show more as to why it stops working. My point is (Windows or not): make sure it works in command line first, then validate with SourceTree. – VonC Dec 14 '17 at 05:44
  • Again, to clarify, it works fine in both sourcetree and the commandline. Everything works as you would expect. Then it stops working. The only thing that changes is I bring my machine back-and-forth to work and switch between networks. I know while I’m at work, they block GitHub, but why would that then render the keys useless for when I’m back on my network? – Mark A. Donohoe Dec 14 '17 at 05:46
  • @MarqueIV When it stops working, again, you need to look at ssh -Tv git@github.com to know more. If your ssh key files are still there, there is no reason it stops working. – VonC Dec 14 '17 at 05:48
  • Exactly... there should be no reason, but it does. And the keys are both still there but for some reason, they aren't tried, hence the fail. Next time it happens, I'll run the command as requested and will update this with its results. – Mark A. Donohoe Dec 14 '17 at 06:54
  • @MarqueIV Exactly: running ssh -Tv git@github.com will be more precise than "it doesn't work" ;) Is your private key passphrase-protected? – VonC Dec 14 '17 at 07:06
  • No, not protected. – Mark A. Donohoe Dec 14 '17 at 07:21
  • @MarqueIV I see bitbucket and github tags: do you have to authenticate yourself against two different remote repositories hosting services (github.com, bitbucket.org)? – VonC Dec 14 '17 at 07:26
  • Yes, and I had created individual keys for each service. But as I said above, even only using github (meaning I deleted my bitbucket account credentials and keys) he problem still remains. The common denominator is the sourcetree client. – Mark A. Donohoe Dec 14 '17 at 07:27
  • @MarqueIV Do you have a ~/.ssh/config file configured as I described in https://stackoverflow.com/a/47766053/6309? – VonC Dec 14 '17 at 07:29
  • I’m also not the only one having this problem. (I hate this iOS client. I thought I had posted that is a comment, not an answer. I don’t know why they insist on using the worlds smallest font.) – Mark A. Donohoe Dec 14 '17 at 07:32
  • @MarqueIV It is best to double-check the content of ~/.ssh/config, and of your remote url (git remote -v) to be sure everything is still intact. SourceTree should not manage anything, just use your existing configuration. – VonC Dec 14 '17 at 07:34
  • Believe me… I have quadruple checked it. I have also manually re-created what sourcetree did. At this point I think it’s best to just wait until it fails again, then I will post the exact files and settings and error messages I’m getting. I think that’s the only way will get anywhere here. – Mark A. Donohoe Dec 14 '17 at 07:35
  • Ok, just happened again now that I switched networks. Logs and more details added to the question. Have at it!!! :) – Mark A. Donohoe Dec 14 '17 at 16:52
  • Coming back to this as this just happened AGAIN!!! However, digging around I now know how to fix it. Sort of. Specifically, when I run 'ssh-add -l -E md5' I get 'The agent has no identities.' Manually running 'ssh-add ' fixes the issue. Question is why does it keep 'un-adding' my keys? Why do I have to perform that step again!? Makes no sense. However, once I re-run the command, everything starts working again... until it again doesn't. – Mark A. Donohoe Feb 07 '18 at 05:49
  • @MarqueIV If your private key is not passphrase protected, you should not need any agent anyway. – VonC Feb 07 '18 at 05:55
  • Also, not sure your command about my config being wrong is correct. Once I manually added the keys again with 'ssh-add', everything started working again. And, that file is generated by SourceTree, not me, and I doubt they would make such a mistake. Rest assured, it looks like it is connecting with 'git' as the user regardless of what that file is saying. – Mark A. Donohoe Feb 07 '18 at 05:56
  • Well I'm not sure what to say. It's not protected, and it's not found until I manually add it with 'ssh-add'. The question is why do I have to keep doing that? And if I don't need an agent, why is it skipping right over my keys? They're clearly there. And when added, they work so I know they're also valid. This is why I'm stumped. – Mark A. Donohoe Feb 07 '18 at 05:58
  • Perhaps I should add a passphrase. Could that be the issue? That it doesn't have one and also says 'AddKeysToAgent yes'? – Mark A. Donohoe Feb 07 '18 at 05:58
  • @MarqueIV You can test that indeed. I thought your key did not have a passphrase. – VonC Feb 07 '18 at 05:59
  • It doesn't have one. That's what I just said based on what you said. You said you don't need an agent if there's no passphrase. I said the config said to add it to the agent. Perhaps the config telling it to add when it doesn't have a passphrase causes it to trip up. Complete guess, but at this point I'm grasping at straws. – Mark A. Donohoe Feb 07 '18 at 06:00
  • Still, do you have any idea why the keys aren't found automatically? I noticed on Windows, my keys have a 'ppk' extension but on Mac, there is no extension. Could that be it? It doesn't realize they are keys so it doesn't check them? – Mark A. Donohoe Feb 07 '18 at 06:02
  • @MarqueIV Yes, on Windows, check the GIT_SSH environment variable. I prefer using openssh instead of PUTTY. – VonC Feb 07 '18 at 06:03
  • @MarqueIV Try again with new keys (as in https://stackoverflow.com/a/42352576/6309), using openssh, and make sure GIT_SSH is desactivated, and Sourcetree is using your system Git for Windows (stick with 2.15.x for now, the 2.16 still has some issues) – VonC Feb 07 '18 at 08:02
  • I'll give that a go later, but I think I've actually uncovered what is going on. I'll post that in its own answer here. – Mark A. Donohoe Feb 07 '18 at 15:05
  • Not sure if you saw, but you were right about the 'git' user in the 'config' file, but only in the case of GitHub. BitBucket worked fine with my username. Not sure Atlassian realized the same wasn't true for GitHub which is why that didn't work. Still, the trick was to connect *from terminal to the host defined in the config file.* When you do that, the keys get added to Keychain, then calling 'ssh-add -A' from a launch agent ensures they're always loaded as expected. It was a pain, but I've now confirmed this solution works on multiple machines. Finally! – Mark A. Donohoe Feb 07 '18 at 16:37
  • 1
    Much appreciated! It was a real pain, but that seems to be the cause! :) – Mark A. Donohoe Feb 07 '18 at 16:39
  • Thank you!! For me this works "Finally, make sure that SourceTree / Tools / Option uses as SSH client the OpenSSH one (not putty) Then SourceTree should have nop problem reusing your ssh key." – ItaiRoded Sep 07 '21 at 12:46
  • 1
    @ItaiRoded Well done. Putty should not be needed nowodays. – VonC Sep 07 '21 at 12:51
0

That ~/.ssh/config excerpt is only applicable for the host MarqueIV-Bitbucket. If your SSH remotes are listed as MarqueIV-Bitbucket:owner/repo then SSH and SourceTree should respect that config; you can confirm this with ssh -Tv MarqueIV-Bitbucket and by updating one or more of the remotes to the MarqueIV-Bitbucket:owner/repo.git format.

Jim Redmond
  • 4,139
  • 1
  • 14
  • 18