openssl returns Bad Magic Number

Question

In my Linux machine i have a binary AES encrypted file:

head -c 100 Leela_Turanga.plr
�|�XѨ��>��c��N�Ώڤ�LW�M��t�p5=c.4���ᑸ�#Owl����M�d��>�ٷa�L�r|��ć�ڐ,��:����#�����\

Also i know that the file has been encryped using the password:

h3y_gUyZ

I am able to decrypt this file using .NET's RijndaelManaged class in the following routine:

public static bool decryptFile(string inputFile, string outputFile)
    {
        string s = "h3y_gUyZ";
        UnicodeEncoding unicodeEncoding = new UnicodeEncoding();
        byte[] bytes = unicodeEncoding.GetBytes(s);
        FileStream fileStream = new FileStream(inputFile, FileMode.Open);
        RijndaelManaged rijndaelManaged = new RijndaelManaged();
        CryptoStream cryptoStream = new CryptoStream(fileStream, rijndaelManaged.CreateDecryptor(bytes, bytes), CryptoStreamMode.Read);
        FileStream fileStream2 = new FileStream(outputFile, FileMode.Create);
        try
        {
            int num;
            while ((num = cryptoStream.ReadByte()) != -1)
            {
            fileStream2.WriteByte((byte)num);
            }
            fileStream2.Close();
            cryptoStream.Close();
            fileStream.Close();
        }
        catch
        {
            fileStream2.Close();
            fileStream.Close();
            File.Delete(outputFile);
            return true;
        }
            return false;
    }

Since im on Linux I embedded this code in a c# program which i run with mono, after i compiled it with mcs.

mcs *.cs -out:mybinary.exe
mono mybinary.exe d Leela_Turanga.plr outputfile.dat

Where the d parameter executes the aforementioned function, Leela_Turanga.plr is the file to decrypt and outputfile.dat is the resulting decryped file. This works fine: I can decrypt the file correctly, and i am able to say this because human readable text appears in the decrypted file. Now i want to decrypt the same file with openssl. First of all i need to get the algorithm parameters, and since the code above works, I modified it to give me these informations:

Key Size
AES operation mode (cbc, ecb, pcbc, cfb...)
Padding (have no idea what that is)

by adding some code:

public static bool decryptFile(string inputFile, string outputFile)
    {
        string s = "h3y_gUyZ";
        UnicodeEncoding unicodeEncoding = new UnicodeEncoding();
        byte[] bytes = unicodeEncoding.GetBytes(s);
        FileStream fileStream = new FileStream(inputFile, FileMode.Open);
        RijndaelManaged rijndaelManaged = new RijndaelManaged();
        CryptoStream cryptoStream = new CryptoStream(fileStream, rijndaelManaged.CreateDecryptor(bytes, bytes), CryptoStreamMode.Read);
        FileStream fileStream2 = new FileStream(outputFile, FileMode.Create);


        //=======DEBUG INFO=======
        //PRINT ALGORITHM SETTINGS
        Console.WriteLine(rijndaelManaged.Mode);    //what AES mode are we using?
        Console.WriteLine(rijndaelManaged.KeySize); //what is the keysize?
        Console.WriteLine(rijndaelManaged.Padding); //what is the padding?
        //========================


        try
        {
            int num;
            while ((num = cryptoStream.ReadByte()) != -1)
            {
            fileStream2.WriteByte((byte)num);
            }
            fileStream2.Close();
            cryptoStream.Close();
            fileStream.Close();
        }
        catch
        {
            fileStream2.Close();
            fileStream.Close();
            File.Delete(outputFile);
            return true;
        }
            return false;
    }

What it comes out is that RijndaelManaged is working in CBC mode, with a 256 bit key and PKCS7 Padding (again no idea if that is important). Now i can try using openssl to decrypt:

openssl enc -aes-256-cbc -d -in Leela_Turanga.plr -out out.bin

Then the password prompt appears, I enter the pass, and a "bad magic number" error is returned"

enter aes-256-cbc decryption password:
bad magic number

And i get no decrypted file.

Why does openssl say that? I have searched on the internet but I haven't found my answer.

Also: Since in c# strings are encoded in UTF-16 (hence 16 bits in a char), and the key is "h3y_gUyZ" which are 8 chars, shouldn't the key be 16 x 8 = 128 bits wide? instead of the 256 rijndaelManaged.KeySize returns.

Note that using the password as key and the key as IV is insecure. The key should be derived from the password, e.g. using `Rfc2898DeriveBytes` and the IV should be indistinguishable from random. — Maarten Bodewes, Jan 04 '17 at 23:43

Maarten Bodewes · Answer 1 · 2017-01-04T23:48:57.927

Why does openssl say that?

OpenSSL uses it's own key derivation routine called EVP_BytesToKey, which takes a salt. This salt is prefixed with a 8 byte magic: Salted__ in ASCII encoding. Passwords should not be directly used as keys, so OpenSSL converts them to keys first.

You can instead provide a key using -K and then the hexadecimal representation of your string (read on for the encoding). You will also need to provide the IV (in your case the same bytes).

Since in C# strings are encoded in UTF-16 (hence 16 bits in a char), and the key is "h3y_gUyZ" which are 8 chars, shouldn't the key be 16 x 8 = 128 bits wide instead of the 256 rijndaelManaged.KeySize returns?

Yes, probably you asked the Rijndael class for the key size before initializing it with the key. And note that .NET uses UTF-16LE (little-endian) as the whole ecosystem is in (stupid) little-endian.

PKCS#7 is a standard that contains a padding scheme. It is both used by .NET and OpenSSL, so that's fine. It is required as ECB and CBC mode require the input size of the cipher is N times the block size of the cipher. So PKCS#7 adds 1 to 16 bytes (the block size of AES) valued \x01 to \x10. Unpadding removes these.

openssl returns Bad Magic Number

1 Answers1