InvertableRSAFunction Fails on encrypting Diffie-Hellman Keys as const byte*

Question

Before we begin here, I have a server and a client. I wish to send an encrypted string to the client that contains the server's Diffie-Hellman public static key and public ephemeral key. To do so, I use the server's private RSA key to send the encrypted string and the client decrypts with the server's public RSA key.

Now the reason why I am needing to do it this way is because the server is the only one with a public/private key pair. This is fine since encrypting with one key pair still cuts off one side of the MITM attack against Diffie-Hellman, and for my requirements, it is ok.

Converting the static and ephemeral keys into a hex encoded string and sending it over the socket is giving me a problem with the private key encryption stage.

my server is doing:

DH2 dhA(dh);
   SecByteBlock sprivA(dhA.StaticPrivateKeyLength()), spubA(
            dhA.StaticPublicKeyLength());
   SecByteBlock eprivA(dhA.EphemeralPrivateKeyLength()), epubA(
            dhA.EphemeralPublicKeyLength());

   dhA.GenerateStaticKeyPair(rnd, sprivA, spubA);
   dhA.GenerateEphemeralKeyPair(rnd, eprivA, epubA);

   string sendBuf, recvBuf;
   string saEncoded, eaEncoded, encoding;

   cout << "spubA: " << (char*) spubA.data() << endl << "epubA: "
            << (char*) epubA.data() << endl;

   SecByteBlock nil;
   nil.CleanNew(HMAC< SHA256 >::DEFAULT_KEYLENGTH);

   HMAC< SHA256 > hmac;
   hmac.SetKey(nil.data(), nil.size());

   HashFilter filter(hmac, new HexEncoder(new StringSink(encoding)));

   filter.Put(spubA.data(), spubA.size());
   filter.MessageEnd();

   saEncoded = encoding;
   encoding = "";

   filter.Put(epubA.data(), epubA.size());
   filter.MessageEnd();

   eaEncoded = encoding;
   encoding = "";

//   StringSource saSource(spubA, sizeof(spubA), true,
//          new HexEncoder(new StringSink(saEncoded)));
//
//   StringSource eaSource(epubA, sizeof(epubA), true,
//          new HexEncoder(new StringSink(eaEncoded)));
//
   sendBuf = saEncoded + " " + eaEncoded;

   cout << "Send Buffer: " << sendBuf << endl;
   SendMsg(sendBuf, tdata);

where SendMsg() contains the encryption process.

It fails at this point:

void SendMsg( string sendBuf, struct ThreadData * tdata )
{
   AutoSeededRandomPool rng;
   Integer m, c, r;
   stringstream ss;

   try
   {
      // Encode the message as an Integer
      m = Integer((const byte *) sendBuf.c_str(), sendBuf.size());

      //Encrypt
      c = tdata->privateKey.CalculateInverse(rng, m);  //HERE!

With error message:

InvertibleRSAFunction: computational error during private key operation

The code that is currently not commented out in the Diffie-Hellman secition was obtained from HERE. The problem with the commented code is that when the client receives the hex encoded string, it has lost data and cannot agree on a shared secret. But it does get through the socket.

An example of this can be shown:

Server:
    spubA: &a�|՜D2�tu�cJ����B�R�8�*i�x?N���p��Q�����K��+O �"��P:k�d|3�����6Z
    epubA: 4v������M�E�`l�K��[dN�|Q^r�-ż�����A~D�>4$�9���"v�*:Y��s�O���J��ow�M�߬�C�9n�;���Z�D�6lp�V��oowZ��WSv��",��A3��XL��8��
    Send Buffer: 2661DC7CD59C4432AF747584634AF69BE60298429C52C738 3476CBCCFAA3B0A14DBE45E3606CC84B171DAC1CCE5B644E

Client:
    Recovered: 2661DC7CD59C4432AF747584634AF69BE60298429C52C738 3476CBCCFAA3B0A14DBE45E3606CC84B171DAC1CCE5B644E
    SA: 2661DC7CD59C4432AF747584634AF69BE60298429C52C738
    EA: 3476CBCCFAA3B0A14DBE45E3606CC84B171DAC1CCE5B644E
    Decoded SA: &a�|՜D2�tu�cJ����B�R�8
    Decoded EA: 4v������M�E�`l�K��[dN

With respect to this instance, I tried on the client side to do the following:

// Get spubA and epubA from server
      recovered = recoverMsg(serverKey, sockServer);

      //Calculate shared secret.
      string sa, ea;
      ss.str(recovered);
      ss >> sa >> ea;
      ss.str("");
      ss.clear();

      cout << "SA: " << sa << endl << "EA: " << ea << endl;

      string decodedSA, decodedEA;
      StringSource decodeSA(sa, true,
               new HexDecoder(new StringSink(decodedSA)));
      StringSource decodeEA(ea, true,
               new HexDecoder(new StringSink(decodedEA)));

      cout << "Decoded SA: " << decodedSA << endl;
      cout << "Decoded EA: " << decodedEA << endl;

      SecByteBlock spubA((const byte*) decodedSA.data(), decodedSA.size());

      if ( spubA.size() < dhB.StaticPublicKeyLength() ) spubA.CleanGrow(
               dhB.StaticPublicKeyLength());
      else spubA.resize(dhB.StaticPublicKeyLength());

      SecByteBlock epubA((const byte*) decodedEA.data(), decodedEA.size());

      if ( epubA.size() < dhB.EphemeralPublicKeyLength() ) epubA.CleanGrow(
               dhB.EphemeralPublicKeyLength());
      else epubA.resize(dhB.EphemeralPublicKeyLength());

But I still get the same result.

Does anybody know how I can encrypt this with the server's private key and send it across the socket correctly?

You really need to isolate your problems. Perhaps you should get things working on a local file. Then, move onto a fixed buffer over a socket (like a string of `'A'`). After that, move to a fixed size buffer with random data (like data that includes a `'\0'`). Finally move onto a real file with a real signature. When you have problems with your fixed buffer, ask a socket send/receive question. — jww, Dec 04 '16 at 07:48

James Combs · Answer 1 · 2016-12-05T06:12:11.370

I actually solved all of my issues by just Base64 encoding everything before sending it across the wire. I also added a receive after every send and a send after every receive to make sure that the server and client are properly synced together to avoid timing issues. No need for all of that 'HexEncoding' nonsense. Just turn it into a string how ever you like, send it into these functions and there we go.

string RecoverMsg( struct ThreadData * tdata )
{
   try
   {
      Integer c = 0, r = 0, m = 0;
      size_t req = 0, bytes = 0;
      AutoSeededRandomPool rng;
      string recovered = "", ack = "", decodedCipher = "";
      byte byteBuf[ 2000 ];
      memset(byteBuf, 0, sizeof(byteBuf));

      // Retrieve message from socket
      cout << "Waiting to receive a message from client " << tdata->tid << endl;

      bytes = tdata->sockSource.Receive(byteBuf, sizeof(byteBuf));
      cout << "Bytes Read: " << bytes << endl;

      cout << "Encoded Cipher Received: " << byteBuf << endl;

      decodedCipher;
      StringSource(byteBuf, sizeof(byteBuf), true,
               new Base64Decoder(new StringSink(decodedCipher)));

      c = Integer(decodedCipher.c_str());

      // Decrypt
      r = tdata->privateKey.CalculateInverse(rng, c);
      cout << "r: " << r << endl;

      // Round trip the message
      req = r.MinEncodedSize();
      recovered.resize(req);
      r.Encode((byte *) recovered.data(), recovered.size());

      cout << "Recovered: " << recovered << endl;

      ack = "ACK";
      bytes = tdata->sockSource.Send((const byte*) ack.c_str(), ack.size());

      return recovered;
   }
   catch ( Exception& e )
   {
      cerr << "caught Exception..." << endl;
      cerr << e.what() << endl;
      tdata->sockSource.ShutDown(SHUT_RDWR);
   }
}

void SendMsg( string sendBuf, struct ThreadData * tdata )
{
   try
   {
      AutoSeededRandomPool rng;
      stringstream ss("");
      string cipher = "", encodedCipher = "";
      Integer m = 0, c = 0, r = 0;
      size_t bytes = 0;
      byte ack[ 10 ];
      memset(ack, 0, sizeof(ack));

      // Treat the message as a big endian array
      m = Integer((const byte *) sendBuf.c_str(), sendBuf.size());
      cout << "m: " << m << endl;

      // Encrypt
      c = tdata->privateKey.CalculateInverse(rng, m);

      ss << c;
      cipher = ss.str();
      ss.str("");
      ss.clear();

      // Base64 encode the cipher
      encodedCipher;
      StringSource(cipher, cipher.size(),
               new Base64Encoder(new StringSink(encodedCipher)));

      cout << "Encoded Cipher Sent: " << encodedCipher << endl;

      // Send the cipher
      bytes = tdata->sockSource.Send((const byte*) encodedCipher.c_str(),
               encodedCipher.size());
      cout << "Bytes Written: " << bytes << endl;

      bytes = tdata->sockSource.Receive(ack, sizeof(ack));
   }
   catch ( Exception& e )
   {
      cerr << "caught Exception..." << endl;
      cerr << e.what() << endl;
      tdata->sockSource.ShutDown(SHUT_RDWR);
   }
}

score 0 · Answer 2 · answered Dec 04 '16 at 20:41

Well, regarding the missing data after HexEncoding, this is most certainly caused by the nn(spubA), because sizeof() is returning the size of the pointer to spubA instead of the size of your string. To obtain the string's length you may use the spubA.length() function or simply use the best way to do it using CryptoPP, IMHO, as you do with the client simply without specifying the size:

StringSource saSource(spubA, true,
      new HexEncoder(new StringSink(saEncoded)));

Now regarding the rest of your code, it looks like you are trying to perform your own textbook RSA, since you begin by transforming the buffer into an integer, please don't cook your own RSA! It's doomed to fail, RSA textbook is weak!

So leverage CryptoPP abilities and perform RSA encryption using a good padding scheme, such as OAEP by using an encryptor and a filter as shown on CryptoPP's wiki:

RSAES_OAEP_SHA_Encryptor enc(key);
StringSource(inputString, true,
    new PK_EncryptorFilter(rng, enc,
        new HexEncoder(
            new StringSink(output), false)));

cout << output << endl;

Now here the point where you have to be careful about is to use the right key, since you want to encrypt using the private key, you have to initialize your key as follows:

RSA::PublicKey key;
key.Initialize(privateKey.GetModulus(), privateKey.GetPrivateExponent();

And that's it, is should work already. To decrypt, use simply:

RSA::PrivateKey clientkey;
clientkey.SetModulus(publicKey.GetModulus());
clientkey.SetPrivateExponent(publicKey.GetPublicExponent());

RSAES_OAEP_SHA_Decryptor d(privateKey);
StringSource(decodedInput, true,
    new PK_DecryptorFilter(rng, d,
        new HexEncoder(
            new StringSink(output), false //lowercase
        )));

cout << output << endl;

But, there is a but: do not use Initialize with the private key to decrypt when you have encrypted with the private key, because Initialize will try and factorise the Modulus, which it cannot do without knowing at least the public and the private exponent.

Now, I must confess that I have not tested to encrypt/decrypt using the private Key and RSA OAEP, if my latest code snippet fails (maybe the RSAES_OAEP_SHA_Decryptor won't be okay with a private key whose primes aren't defined... I'm typing blindly, without compiler) then you'll have to brew your own RSA as you seems to be trying now. If you need more help with that, please comment below and I'll try tomorrow to spawn my compiler and see how it works out.

But be aware that RSA textbook is very weak and so you should pad your data using OAEP.

I have tried using the private key for an `RSAES_OAEP_SHA_Decryptor` and it didn't work before. That was awhile back so I am not sure exactly the situation I was in with why it wasn't working and I switched to RAW RSA. I am using my own way of doing RSA since I only have one public/private key pair. — James Combs, Dec 05 '16 at 06:14

InvertableRSAFunction Fails on encrypting Diffie-Hellman Keys as const byte*

2 Answers2