Problems converting byte array to string and back to byte array

Question

There are a lot of questions with this topic, the same solution, but this doesn't work for me. I have a simple test with an encryption. The encryption/decryption itself works (as long as I handle this test with the byte array itself and not as Strings). The problem is that don't want to handle it as byte array but as String, but when I encode the byte array to string and back, the resulting byte array differs from the original byte array, so the decryption doesn't work anymore. I tried the following parameters in the corresponding string methods: UTF-8, UTF8, UTF-16, UTF8. None of them work. The resulting byte array differs from the original. Any ideas why this is so?

Encrypter:

public class NewEncrypter
{
    private String algorithm = "DESede";
    private Key key = null;
    private Cipher cipher = null;

    public NewEncrypter() throws NoSuchAlgorithmException, NoSuchPaddingException
    {
         key = KeyGenerator.getInstance(algorithm).generateKey();
         cipher = Cipher.getInstance(algorithm);
    }

    public byte[] encrypt(String input) throws Exception
    {
        cipher.init(Cipher.ENCRYPT_MODE, key);
        byte[] inputBytes = input.getBytes("UTF-16");

        return cipher.doFinal(inputBytes);
    }

    public String decrypt(byte[] encryptionBytes) throws Exception
    {
        cipher.init(Cipher.DECRYPT_MODE, key);
        byte[] recoveredBytes = cipher.doFinal(encryptionBytes);
        String recovered = new String(recoveredBytes, "UTF-16");

        return recovered;
    }
}

This is the test where I try it:

public class NewEncrypterTest
{
    @Test
    public void canEncryptAndDecrypt() throws Exception
    {
        String toEncrypt = "FOOBAR";

        NewEncrypter encrypter = new NewEncrypter();

        byte[] encryptedByteArray = encrypter.encrypt(toEncrypt);
        System.out.println("encryptedByteArray:" + encryptedByteArray);

        String decoded = new String(encryptedByteArray, "UTF-16");
        System.out.println("decoded:" + decoded);

        byte[] encoded = decoded.getBytes("UTF-16");
        System.out.println("encoded:" + encoded);

        String decryptedText = encrypter.decrypt(encoded); //Exception here
        System.out.println("decryptedText:" + decryptedText);

        assertEquals(toEncrypt, decryptedText);
    }
}

You first need to convert the bytes to something that can be presented as a string. Usually by converting to hex or base64. — Roger Lindsjö, Feb 01 '12 at 15:07
What's the actual difference you see in the byte arrays before and after converting to a string? — Herms, Feb 01 '12 at 15:08
@Roger Lindsjö: thanks for the tipp. I will try it immediately. — Bevor, Feb 01 '12 at 15:10
@Herms: An example -> encryptedByteArray:[B@7df17e77, encoded:[B@79a5f739 — Bevor, Feb 01 '12 at 15:11
Those look like memory addresses, not the actual contents of the array. — Herms, Feb 01 '12 at 15:29
lol...This question is exactly what I have been trying to fix, right down to the array->encode->decode->decrypt, complete with WriteLine("")s after each step. I get stuck at the next to last step because FromBase64(). — KWallace, Jul 30 '21 at 17:45

Joni · Accepted Answer · 2012-02-02T11:09:37.677

It is not a good idea to store encrypted data in Strings because they are for human-readable text, not for arbitrary binary data. For binary data it's best to use byte[].

However, if you must do it you should use an encoding that has a 1-to-1 mapping between bytes and characters, that is, where every byte sequence can be mapped to a unique sequence of characters, and back. One such encoding is ISO-8859-1, that is:

    String decoded = new String(encryptedByteArray, "ISO-8859-1");
    System.out.println("decoded:" + decoded);

    byte[] encoded = decoded.getBytes("ISO-8859-1"); 
    System.out.println("encoded:" + java.util.Arrays.toString(encoded));

    String decryptedText = encrypter.decrypt(encoded);

Other common encodings that don't lose data are hexadecimal and base64, but sadly you need a helper library for them. The standard API doesn't define classes for them.

With UTF-16 the program would fail for two reasons:

String.getBytes("UTF-16") adds a byte-order-marker character to the output to identify the order of the bytes. You should use UTF-16LE or UTF-16BE for this to not happen.
Not all sequences of bytes can be mapped to characters in UTF-16. First, text encoded in UTF-16 must have an even number of bytes. Second, UTF-16 has a mechanism for encoding unicode characters beyond U+FFFF. This means that e.g. there are sequences of 4 bytes that map to only one unicode character. For this to be possible the first 2 bytes of the 4 don't encode any character in UTF-16.

Today I saw that I have problem while using encryption and decryption in different VMs. Obviously only your solution works. — Bevor, Feb 02 '12 at 10:07
Your approach of using Apache Commons Codec should also work, but you have to distribute the commons-codec library with your application. — Joni, Feb 02 '12 at 11:16

score 29 · Answer 2 · answered Oct 15 '15 at 15:40

29

Accepted solution will not work if your String has some non-typical charcaters such as š, ž, ć, Ō, ō, Ū, etc.

Following code worked nicely for me.

byte[] myBytes = Something.getMyBytes();
String encodedString = Base64.encodeToString(bytes, Base64.NO_WRAP);
byte[] decodedBytes = Base64.decode(encodedString, Base64.NO_WRAP);

answered Oct 15 '15 at 15:40

Aleksandar Ilic

1,521
16
19

the only working solution , although you have to depend to ApacheCommons – AntJavaDev Jan 13 '16 at 13:43
3

I was using `android.util.Base64`. If you don't want to include `ApacheCommons` I guess you can always copy the file in your project. Here is the source: https://github.com/android/platform_frameworks_base/blob/master/core/java/android/util/Base64.java – Aleksandar Ilic Jan 13 '16 at 15:19
1

yes i pointed that for java applications , this utility class comes with JDK 1.8 as well , but for previous java Versions you have to depend to ApacheCommons as it is the only working method. – AntJavaDev Jan 14 '16 at 07:10
I don't know how to start thanking you. I am very very very grateful. More codes to you brain. – Odin Aug 06 '17 at 06:16
1

Another alternative is to use `javax.xml.bind.DatatypeConverter.printHexBinary(bytesToHexString[])` and `javax.xml.bind.DatatypeConverter.parseHexBinary("hexStringTobytes")` – ObviousChild Jul 23 '20 at 16:39

Bevor · Answer 3 · 2012-02-02T10:08:14.613

Now, I found another solution too...

    public class NewEncrypterTest
    {
        @Test
        public void canEncryptAndDecrypt() throws Exception
        {
            String toEncrypt = "FOOBAR";

            NewEncrypter encrypter = new NewEncrypter();

            byte[] encryptedByteArray = encrypter.encrypt(toEncrypt);
            String encoded = String.valueOf(Hex.encodeHex(encryptedByteArray));

            byte[] byteArrayToDecrypt = Hex.decodeHex(encoded.toCharArray());
            String decryptedText = encrypter.decrypt(byteArrayToDecrypt); 

            System.out.println("decryptedText:" + decryptedText);

            assertEquals(toEncrypt, decryptedText);
        }
    }

score 0 · Answer 4 · answered Feb 01 '12 at 15:50

Your problem is that you cannot build a UTF-16 (or any other encoding) String from an arbitrary byte array (see UTF-16 on Wikipedia). It is up to you, however, to serialize and deserialize the encrypted byte array without any loss, in order to, say, persist it, and make use of it later. Here's the modified client code that should give you some insight of what's actually happening with the byte arrays:

public static void main(String[] args) throws Exception {
  String toEncrypt = "FOOBAR";

  NewEncrypter encrypter = new NewEncrypter();

  byte[] encryptedByteArray = encrypter.encrypt(toEncrypt);
  System.out.println("encryptedByteArray:" + Arrays.toString(encryptedByteArray));

  String decoded = new String(encryptedByteArray, "UTF-16");
  System.out.println("decoded:" + decoded);

  byte[] encoded = decoded.getBytes("UTF-16");
  System.out.println("encoded:" + Arrays.toString(encoded));

  String decryptedText = encrypter.decrypt(encryptedByteArray); // NOT the "encoded" value!
  System.out.println("decryptedText:" + decryptedText);
}

This is the output:

encryptedByteArray:[90, -40, -39, -56, -90, 51, 96, 95, -65, -54, -61, 51, 6, 15, -114, 88]
decoded:<some garbage>
encoded:[-2, -1, 90, -40, -1, -3, 96, 95, -65, -54, -61, 51, 6, 15, -114, 88]
decryptedText:FOOBAR

The decryptedText is correct, when restored from the original encryptedByteArray. Please note that the encoded value is not the same as encryptedByteArray, due to the data loss during the byte[] -> String("UTF-16")->byte[] conversion.

Thanks, at the moment I found another solution too (see my answer). Nevertheless I will accept your answer, because your solution works too. — Bevor, Feb 01 '12 at 16:02
Today I saw that you decrypt the original byte array. This is actually not what I wanted (and unfortunately I had problem again decrypting it when using this in different VMs), so only Joni's solution work. — Bevor, Feb 02 '12 at 10:06
My code was just an illustration of the error you had in your code, not a serialization solution ("It is up to you, however, to serialize and deserialize the encrypted byte array without any loss"), as you might want to use tons of different serialization techniques (using DataIn/OutputStreams etc.) — Alexander Pavlov, Feb 02 '12 at 13:50

Problems converting byte array to string and back to byte array

4 Answers4

Linked

Related