Read and write more than 8 bit symbols

Question

I am trying to write an Encoded file.The file has 9 to 12 bit symbols. While writing a file I guess that it is not written correctly the 9 bit symbols because I am unable to decode that file. Although when file has only 8 bit symbols in it. Everything works fine. This is the way I am writing a file

File.AppendAllText(outputFileName, WriteBackContent, ASCIIEncoding.Default);

Same goes for reading with ReadAllText function call.
What is the way to go here?

I am using ZXing library to encode my file using RS encoder.

 ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);//if i use AZTEC_DATA_8 it works fine beacuse symbol size is 8 bit
 int[] bytesAsInts = Array.ConvertAll(toBytes.ToArray(), c => (int)c);
 enc.encode(bytesAsInts, parity);
 byte[] bytes = bytesAsInts.Select(x => (byte)x).ToArray();
 string contentWithParity = (ASCIIEncoding.Default.GetString(bytes.ToArray()));
 WriteBackContent += contentWithParity;
 File.AppendAllText(outputFileName, WriteBackContent, ASCIIEncoding.Default);

Like in the code I am initializing my Encoder with AZTEC_DATA_12 which means 12 bit symbol. Because RS Encoder requires int array so I am converting it to int array. And writing to file like here.But it works well with AZTEC_DATA_8 beacue of 8 bit symbol but not with AZTEC_DATA_12.

score 3 · Accepted Answer · edited May 23 '17 at 11:50

Main problem is here:

byte[] bytes = bytesAsInts.Select(x => (byte)x).ToArray();

You are basically throwing away part of the result when converting the single integers to single bytes.

If you look at the array after the call to encode(), you can see that some of the array elements have a value higher than 255, so they cannot be represented as bytes. However, in your code quoted above, you cast every single element in the integer array to byte, changing the element when it has a value greater than 255.

So to store the result of encode(), you have to convert the integer array to a byte array in a way that the values are not lost or modified.

In order to make this kind of conversion between byte arrays and integer arrays, you can use the function Buffer.BlockCopy(). An example on how to use this function is in this answer.

Use the samples from the answer and the one from the comment to the answer for both conversions: Turning a byte array to an integer array to pass to the encode() function and to turn the integer array returned from the encode() function back into a byte array.

Here are the sample codes from the linked answer:

// Convert byte array to integer array
byte[] result = new byte[intArray.Length * sizeof(int)];
Buffer.BlockCopy(intArray, 0, result, 0, result.Length);

// Convert integer array to byte array (with bugs fixed)
int bytesCount = byteArray.Length;
int intsCount = bytesCount / sizeof(int);
if (bytesCount % sizeof(int) != 0) intsCount++;
int[] result = new int[intsCount];            
Buffer.BlockCopy(byteArray, 0, result, 0, byteArray.Length);

Now about storing the data into files: Do not turn the data into a string directly via Encoding.GetString(). Not all bit sequences are valid representations of characters in any given character set. So, converting a random sequence of random bytes into a string will sometimes fail.

Instead, either store/read the byte array directly into a file via File.WriteAllBytes() / File.ReadAllBytes() or use Convert.ToBase64() and Convert.FromBase64() to work with a base64 encoded string representation of the byte array.

Combined here is some sample code:

    ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);//if i use AZTEC_DATA_8 it works fine beacuse symbol size is 8 bit
    int[] bytesAsInts = Array.ConvertAll(toBytes.ToArray(), c => (int)c);
    enc.encode(bytesAsInts, parity);

    // Turn int array to byte array without loosing value
    byte[] bytes = new byte[bytesAsInts.Length * sizeof(int)];
    Buffer.BlockCopy(bytesAsInts, 0, bytes, 0, bytes.Length);

    // Write to file
    File.WriteAllBytes(outputFileName, bytes);

    // Read from file
    bytes = File.ReadAllBytes(outputFileName);            

    // Turn byte array to int array 
    int bytesCount = bytes.Length * 40;
    int intsCount = bytesCount / sizeof(int);
    if (bytesCount % sizeof(int) != 0) intsCount++;
    int[] dataAsInts = new int[intsCount];
    Buffer.BlockCopy(bytes, 0, dataAsInts, 0, bytes.Length);

    // Decoding
    ReedSolomonDecoder dec = new ReedSolomonDecoder(GenericGF.AZTEC_DATA_12);
    dec.decode(dataAsInts, parity);

Read and write more than 8 bit symbols

1 Answers1