Encrypting MP3 file using IDEA algorithm [UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 6: invalid continuation byte]

Question

I've looked into similar errors on stackoverflow but nothing helped me I have implemented IDEA algorithm that takes an encoded hex data input (16 hex, which is 64 bit as IDEA plain text size) For example with utf-8 encoding/decoding:

KEY = int('006400c8012c019001f4025802bc0320', 16)
plain_text = 'HiStackO'
cryptor = IDEA(KEY)  # Initialize cryptor with 128bit key
cipher_text = cryptor.encrypt(plain_text)
deciphered_text = cryptor.decrypt(cipher_text)

Encyrpt/decrypt function are below Output:

Original text = HiStackO
Hex encoded text = 4869537461636b4f
Ciphered text = b6315c103ab29de1
Deciphered text = HiStackO

I am facing issues with some text strings for example 'thinghwr' gets decrypted/encrypted successfully but for 'thingher' I get

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcd in position 1: invalid continuation byte

I've tried latin-1 and other encoders but the result isn't the original..
As for bytes I am trying to encrypt an MP3 song file by reading 8 bytes at a time, decode and encrypt and write encryption to the new encrypted file

cryptor = IDEA(KEY)  # Initialize cryptor with 128bit key

in_file = open("song.mp3", "rb")
out_file = open("encrypted.mp3", "w")

bytes8 = in_file.read(8)

while bytes8:
    res = cryptor.encrypt(bytes8.decode("latin-1"), codec="latin-1")
    print(res)
    res = ''.join('0' * (16 - len(res))) + res
    out_file.write(res)
    bytes8 = in_file.read(8)

in_file.close()
out_file.close()

Each 'res' is 16 hex numbers which contains the encrypted/decrypted text and written to the file.
The file gets encrypted successfully with no issues.

As for decryption I am using the following method:

in_file = open("encrypted.mp3", "r")
out_file = open("decrypted.mp3", "wb")

bytes8 = in_file.read(16)
while bytes8:
    res = cryptor.decrypt(bytes8)
    print(res)
    out_file.write(res.encode())
    bytes8 = in_file.read(16)

in_file.close()
out_file.close()

During decryption after few successful decryptions the following error shows up:

line 136, in decrypt
    return bytes.fromhex(res).decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 0: invalid start byte

Line 136 which is decrypt function:

        res = self.calculate_cipher(self.dec_sub_keys, cipher_text)
        res = ''.join('0' * (16 - len(res))) + res
        return bytes.fromhex(res).decode()

I've tried different encoding but nothing works out What am I doing wrong here? I am new to Python and deal with codecs before.
Encrypt/Decrypt functions:

def encrypt(self, plain_text='', is_hex=False, codec='utf-8'):
    if not is_hex:
        plain_text = plain_text.encode(codec).hex()
    plain_text = get_bin_block(plain_text)
    return self.calculate_cipher(self.enc_sub_keys, plain_text)

def decrypt(self, cipher_text='', codec='utf-8'):
    cipher_text = get_bin_block(cipher_text)
    res = self.calculate_cipher(self.dec_sub_keys, cipher_text)
    res = ''.join('0' * (16 - len(res))) + res
    return bytes.fromhex(res).decode(codec)

get_bin_block list is a function that converts the text into 4 16-bit binary blocks for calculating encryption/decryption

Answer 1

    res = self.calculate_cipher(self.dec_sub_keys, cipher_text)
    res = ''.join('0' * (16 - len(res))) + res
    return bytes.fromhex(res).decode()

When you calculate_cipher, you could potentially end up with any arbitrary sequence of hex digits, depending on the cipher_text.

When you then attempt to .decode() the corresponding bytes, Python tries the UTF-8 encoding by default. This encoding cannot interpret every possible byte sequence as text; some values and sequences are illegal. You say that you "tried different encoding"; but you would have to pick one that actually works for this purpose, and also make sure to use it consistently across the entire program.

The problem isn't that you're reading a binary file. The problem is that you're trying to store the encrypted data as binary, though a very convoluted system of figuring out the hex digits (as text), finding the bytes corresponding to those digits, decoding those bytes back into a string to return from the module, and then encoding them again to write to the file. If you want to generate bytes, you should just generate bytes directly. Alternately, there's nothing that says you can't write a text file representing the encrypted results from a binary file, or vice-versa - just as long as you can show that the process is reversible.

It is very important that you have a proper understanding of the fundamentals here. You cannot expect to skip steps, just get a solution for the current problem and move on with your life - you will just stumble again at the next opportunity.