If a server has only a public key it cannot decrypt a message from a client.
One more time, with the important vocab words in bold: If a server has only a public key it cannot decrypt a message from a client.
What the server is probably doing is verifying a message signed by the client.
In the case of SSH public key authentication (RFC 4252, section 7) it is a signature over the just-negotiated session ID and some context data (which requires the client to have the private key). The server can then run the signature verification algorithm (which only requires the public key). If a preregistered key checks out, the client is thus authenticated.
In the case of RSA keys the signing and decrypting operations look mathematically similar, which can cause some to use the terms loosely, but we should be precise.
- Encrypt (public key): Transform content in such a way to occlude it from everyone except the intended recipient (the holder of the private key).
- Common algorithms: RSA
- In encryption the encrypted output REPLACES the original content in transmission.
- Public Key Encryption is frequently used to encrypt a newly generated symmetric key which actually encrypts the data. For RSA the encrypted output is limited by the key size, symmetric algorithms have much larger maxima.
- Decrypt (private key): Transform content from the Encrypt operation back into its original form.
- Since anyone could have your public key, anyone can encrypt data for you.
- Nothing is inherent in the Encrypt/Decrypt pair to establish trust... the message could be a poison pill, and you don't know who wrote it.
- Sign (private key): Apply a transform over an input (which is usually a digest/hash of the true content) to produce a value no one else can.
- Common algorithms: RSA, ECDSA, DSA
- The signature is presented WITH the content, it does not replace it.
- Verify (public key): Apply a transform over the input and the signature which results in a
true
or false
.
- If the digest is computed independently by the content receiver the signature proves that the content was not tampered.
- When the signature is deemed correct it proves that it was generated by the key holder, which can be used in a trust decision. (The key could have been compromised, in which case "key holder" and "original key holder" could be different)
- For RSA this is "unpack the digest using an algorithm similar to decrypt, then compare the digests", so an RSA implementation could indicate what the correct hash is.
- For DSA and ECDSA the digest is used as a BigInteger in a formula that produces the second half of the signature from the first half, so DSA can't tell you what a correct digest value is.