Java Secure Chat Gibberish Issue

Question

So I have done a socket program written in Java where you basically have to set up a session key establishment before starting a chat.

When the session key establishment is set up, then the server and client can start chatting.

When the host and client sends messages, the messages will first be hashed and then encrypted before being sent. When the receiving party receives the ciphertext, they are supposed to decrypt it and see if the hash matches or not.

The sending party encrypts and decrypts their own messages fine without any issue.

How can I make the receiver understand what the server is sending?

For the receiving party, all messages are received as a String format. I then use byte[] to convert them into bytes for the purpose of decrypting it (it's in RC4, and I got it from somewhere else. So is the SHA1 function).

First, I tried to simply convert it to bytes, then I tried to convert as UTF-8.

Server Side (Chat function, the part of issue):

            boolean cont=true;

    while(cont)
    {
        Scanner msgReader=new Scanner (System.in);

        System.out.print("Enter message to Client: ");
        String msg=msgReader.nextLine();

        String h=SHA1.encryptThisString(sharedKey+msg);
        byte [] c=RC4.encrypt(msg+h, sharedKey);
        String s2=RC4.decrypt(c, sharedKey);

                System.out.println("Ciphertext: "+c);
        System.out.println("Ciphertext in String: "+c.toString());
        System.out.println("Deciphered Ciphertext: "+RC4.decrypt(c, sharedKey));;

        out.println(c.toString());

        String msgC=in.readLine();
        System.out.println("Client: "+msgC);

        if(msgC.equals("exit"))
        {
            cont=false;
            break;
        }
    }

Client Side (Chat function, the part of issue):

    while(cont)
    {
        Scanner msgReader=new Scanner (System.in);

        String cH=in.readLine();
        byte[] cHB=cH.getBytes("UTF-8");

        String msgH=RC4.decrypt(cHB, sharedKey);

        System.out.println("Server Ciphertext (String): "+cH);
        System.out.println("Server Ciphertext (Byte): "+cHB);
        System.out.println("Deciphered Message: "+msgH);

        System.out.print("Enter message to Server: ");
        String msg=msgReader.nextLine();
        out.println(msg);

        if(msg.equals("exit"))
        {
            System.out.print("Chat session ended");
            cont=false;
            break;
        }
    }

Suppose I want to send "Hello Bob" to the receiver. The encrypted ciphertext in byte and string format will be [B@224aed64.

Expected both the sender and receiver to have the same decrypted text: Hello Bob52f1f8dab56a1c980223c6176ece1fcb576905d6.

But only the sender shows the correct string, and the receiver shows gibberish!

Here's what the sender displayed:

Ciphertext: [B@224aed64 Ciphertext in String: [B@224aed64 Deciphered Ciphertext: Hello Bob52f1f8dab56a1c980223c6176ece1fcb576905d6

Expected the receiver to display:

Server Ciphertext (String): [B@224aed64

Server Ciphertext (Byte): [B@224aed64

Deciphered Message: Hello Bob52f1f8dab56a1c980223c6176ece1fcb576905d6

BUT the receiver actually displayed this:

Server Ciphertext (String): [B@224aed64

Server Ciphertext (Byte): [B@224aed64

Deciphered Message: Åtû–bx¬&ÁzÖ

Suggestions? Cheers!

String is not a container for binary data. Use a byte array. — user207421, Apr 04 '19 at 01:42
Please place answers in Answer blocks. Later, you can accept your own Answer. Also see [How does accepting an answer work?](https://meta.stackexchange.com/q/5234/173448) — jww, Apr 06 '19 at 01:43

score 1 · Answer 1 · answered Apr 04 '19 at 00:54

1

[B@224aed64 is not the ciphertext. That's just what you get from calling toString on an array. You get the class name (which for a byte[] is [B) followed by the hash code in hexadecimal. It has no relation to the actual data in the array.

So you aren't actually sending any ciphertext. Don't use toString on arrays.

answered Apr 04 '19 at 00:54

user253751

57,427
7
48
90

@pc9845 If I say "flibbertigibbit is not the password to unlock my door" does it matter if I'm talking about the string version or the byte version of "flibbertigibbit"? – user253751 Apr 04 '19 at 01:14

score 1 · Accepted Answer · answered Apr 04 '19 at 06:18

You have a lot of problems there.

Don't use Strings when using cryptography, also it is not recommended with sockets instead use byte[]
RC4 is not recommended, use AES instead
You don't need the additional data (shared key + message).
If you want extra security you might want to add the Double Ratchet algorithm
You don't need a boolean for the while loop if you're using break use while(true)
As @immibis as pointed to you array.toString is not ciphertext String, if you want the ciphertext as String use new String(ciphertext)
Don't use such weird variable names

Java Secure Chat Gibberish Issue

2 Answers2