Decrypting Large files with RSA in pycrypto?

Question

I have been using pycrypto module for encryption and decryption with RSA key pair and algorithm. The problem is when I try encrypting large files (10kB of text file) I take the block size of 32 byte when reading the file and encrypting it

>>> f = open('10kb','rb')
>>> p = open('enc','wb')
>>> while True:
       data = f.read(32)
       if not data:
           break
       enc_data = public_key.encrypt(data,32)
       p.write(enc_data[0])
    p.close()
    f.close()

It gives the output:

128
128
.......and the many 128 blocks it is writing

When I try to decrypt the encrypted file, I need to read it with 128 byte block so as to give back 32 byte blocks,

>>> f = open('enc','rb')
>>> p = open('dec','wb')
>>> while True:
       data = f.read(128)
       if not data:
           break
       dec_data = private_key.decrypt(data)
       p.write(dec_data)
    p.close()
    f.close()

It is giving the output:

32
32
.....so many 32 byte blocks it is decrypting, then
128
128
128
128
Traceback (most recent call last):
  File "<pyshell#251>", line 5, in <module>
     enc_data = private_key.decrypt(data)
  File "/usr/lib/python3/dist-packages/Crypto/PublicKey/RSA.py", line 174,   in decrypt
     return pubkey.pubkey.decrypt(self, ciphertext)
  File "/usr/lib/python3/dist-packages/Crypto/PublicKey/pubkey.py", line 93, in decrypt
      plaintext=self._decrypt(ciphertext)
  File "/usr/lib/python3/dist-packages/Crypto/PublicKey/RSA.py", line 237, in _decrypt
      cp = self.key._blind(ciphertext, r)
  ValueError: Message too large

To the point where it is outputting the block size of 32, it is decrypting right, but where it starts with 128, its messing up. Why it is saying Message size too large ? Is there any better and fast way to decrypt large text files using pycrypto module ?

One would never encrypt large files in this "block mode" for RSA. It's not very secure and horribly slow. Use [hybrid encryption](https://en.wikipedia.org/wiki/Hybrid_cryptosystem) instead. — Artjom B., Feb 28 '16 at 14:31
Sorry for my more or less OT points: I don't know prcrypto. Is prcrypto restricted to process very small blocks like 32 or 128 bits? If the RSA modulus is, say, 1024 bits, I couldn't imagine that such small blocks are individually encrypted with it. If you happen to have time, you could try out my Python code in s13.zetaboards.com/Crypto/topic/7234475/1/ (see Ex.3 there, which uses RSA to process blocks up to the sizes of the modulus with block-chaining in the sense of block ciphers and with authentication) and tell me your evaluation of it. — Mok-Kong Shen, Feb 28 '16 at 15:35
@Mok-KongShen well, i have used the module to use all the functions in it... if I have to make code of mnany of the algorithms and signature plus MD5, SHA it will be a huge task... so I happen to choose the module... thank you for the concern i will take a look at your code :) — Manuj Mittal, Feb 28 '16 at 15:53
What ArtjomB said. Use something like [AES](http://stackoverflow.com/questions/16761458/how-to-aes-encrypt-decrypt-files-using-python-pycrypto-in-an-openssl-compatible/20457519#20457519) to encrypt your data and just use RSA to encrypt your AES key. — PM 2Ring, Feb 28 '16 at 16:43
@PM2Ring but that will be long.. I cant use RSA for such long files ? It is decrypting half of the message but after sometime it gives error...:/ — Manuj Mittal, Feb 28 '16 at 17:06
I'm not sure what your code is doing, exactly (consider posting a [MCVE]), but I suspect that your RSA key size is too small, if your block size is only 32 bytes. RSA is _far_ slower than a symmetric encryption like AES, so it's rarely used for encrypting large amounts of data. — PM 2Ring, Feb 28 '16 at 17:27
@PM2Ring RSA key size is 1024bits... I guess its not small :) — Manuj Mittal, Feb 28 '16 at 17:34
RSA encryption is limited to encrypting data shorter than it's key size. A 1024-bit key can encrypt up to 127-bytes and 2048-bit key can encrypt up to 255-bytes with textbook RSA. For larger data encrypt the data with AES and a random key and then encrypt the key with RSA. — zaph, Feb 28 '16 at 19:24
One of the reasons for the move to EC is because 2048-bits get getting cumbersome not to mention 4096-bit keys. — zaph, Feb 28 '16 at 19:28

score 2 · Accepted Answer · answered Feb 28 '16 at 18:55

Partial answer coming along ...

RSA works on numbers. You only get bytes out of it when you serialize those long integers. Since those numbers don't have a fixed size, they are serialized with as much bytes as are necessary, but not more.

An RSA encryption c = m^e mod n can result in ciphertexts, which are so much smaller than n, that not all the bytes are filled, because leading zeros of the the number don't have to be serialized.

Sometimes (depending on modulus and plaintext) it may happen that you're writing a 127 byte chunk instead of a 128 byte chunk during encryption, but you're always reading a 128 byte chunk during decryption. That means, you're taking away one byte from the next chunk. When the alignment breaks, you can run into various random behaviors such as a chunk being larger than the modulus and therefore not a valid ciphertext.

There are two ways to solve that:

Always write the length of the ciphertext chunk before it.

Encryption:

data = f.read(readsize)
if not data:
    break
i += 1
enc_data = public_key.encrypt(data, 32)[0]

p.write(chr(len(enc_data)))
p.write(enc_data)

Decryption:

length = f.read(1)
if not length:
    break
data = f.read(ord(length))
print(length, len(data))
j += 1
dec_data = private_key.decrypt(data)
p.write(dec_data[:readsize])

At the end you have to reduce the ciphertext to the original plaintext size, because you're working without PKCS#1 v1.5 padding or OAEP.

Pad the zero bytes that are missing during encryption.

Encryption:

data = f.read(readsize)
if not data:
    break
i += 1
enc_data = public_key.encrypt(data, 32)[0]

while len(enc_data) < writesize:
    enc_data = "\x00" + enc_data
p.write(enc_data)

Decryption:

data = f.read(writesize)
if not data:
    break
j += 1
dec_data = private_key.decrypt(data)
p.write(dec_data[:readsize])

Note that readsize = 127 and writesize = 128. Here are the full source codes for both variants.

Now, this is a partial answer, because this still leads to corrupt files, which are also too short, but at least it fixes the OP's error.

Thank You for the detailed answer.. I understand it now :) though my problem already solved but I appreciate thank you :) — Manuj Mittal, Feb 28 '16 at 19:55

Decrypting Large files with RSA in pycrypto?

1 Answers1

Linked