Getting wrong file encoding(?) after streaming from php to client through ajax

Question

Using laravel I am sending binary data to client, which is later being saved to client's machine. I am using ajax and streaming the contents. This is what I send from the server:

$fileRecord = File::findOrFail($fileId);
set_time_limit(0);

$fs = Storage::disk('local')->getDriver();
$metaData = $fs->getMetadata($fileRecord->getFilePath());
$fileStream = $fs->readStream($fileRecord->getFilePath());

return response()->stream(function () use ($fileStream) {
    while(@ob_end_clean());
    fpassthru($fileStream);
}, 200, [
    'Content-Transfer-Encoding' => 'binary',
    'Content-Type'              => $metaData['type'],
    'Content-Disposition'       => "attachment; filename=\"{$fileRecord->getFullFileName()}\"",
]);

I solved couple of problems related to size of data, like mentioned here.

But I cannot get the proper content of files on the client. I tried saving content to a different file on the server and everything works fine.

Here is the sample of what I am getting (samples list a piece of file from beginning until a certain character):

Ex 1. png image

This is what it should be:

89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 06 40 00 00 04 B0 08 02 00 00 00 2C 63 11 C0 00 00 00 04 67 41 4D 41 00 00 D6 D8 D4 4F 58 32 00 00 00 19 74 45 58 74 53 6F 66 74 77 61 72 65 00 41 64 6F 62 65 20 49 6D 61 67 65 52 65 61 64 79 71 C9 65 3C 00 07 8D 52 49 44 41 54

This is what is being generated:

copied from devTools into file:

C2 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 06 40 00 00 04 C2 B0 08 02 00 00 00 2C 63 11 C3 80 00 00 00 04 67 41 4D 41 00 00 C3 96 C3 98 C3 94 4F 58 32 00 00 00 19 74 45 58 74 53 6F 66 74 77 61 72 65 00 41 64 6F 62 65 20 49 6D 61 67 65 52 65 61 64 79 71 C3 89 65 3C 00 07 C2 8D 52 49 44 41 54

after browser downloaded data as a file on client:

EF BF BD 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 06 40 00 00 04 EF BF BD 08 02 00 00 00 2C 63 11 EF BF BD 00 00 00 04 67 41 4D 41 00 00 EF BF BD EF BF BD EF BF BD 4F 58 32 00 00 00 19 74 45 58 74 53 6F 66 74 77 61 72 65 00 41 64 6F 62 65 20 49 6D 61 67 65 52 65 61 64 79 71 EF BF BD 65 3C 00 07 EF BF BD 52 49 44 41 54

Ex 2. pdf file

This is what it should be:

25 50 44 46 2D 31 2E 35 0D 0A 25 B5 B5 B5 B5 0D 0A 31 20 30 20 6F 62 6A 0D 0A 3C 3C 2F 54 79 70 65 2F 43 61 74 61 6C 6F 67 2F 50 61 67 65 73 20 32 20 30 20 52 2F 4C 61 6E 67 28 65 6E 2D 55 53 29 20 2F 53 74 72 75 63 74 54 72 65 65 52 6F 6F 74 20 33 31

This is what is being generated:

manually copied from devTools into file:

25 50 44 46 2D 31 2E 35 0D 0A 25 C2 B5 C2 B5 C2 B5 C2 B5 0D 0A 31 20 30 20 6F 62 6A 0D 0A 3C 3C 2F 54 79 70 65 2F 43 61 74 61 6C 6F 67 2F 50 61 67 65 73 20 32 20 30 20 52 2F 4C 61 6E 67 28 65 6E 2D 55 53 29 20 2F 53 74 72 75 63 74 54 72 65 65 52 6F 6F 74 20 33 31

after browser downloaded data as a file on client:

25 50 44 46 2D 31 2E 35 0D 0A 25 EF BF BD EF BF BD EF BF BD EF BF BD 0D 0A 31 20 30 20 6F 62 6A 0D 0A 3C 3C 2F 54 79 70 65 2F 43 61 74 61 6C 6F 67 2F 50 61 67 65 73 20 32 20 30 20 52 2F 4C 61 6E 67 28 65 6E 2D 55 53 29 20 2F 53 74 72 75 63 74 54 72 65 65 52 6F 6F 74 20 33 31

Some of the headers I get on the client (chrome devTools):

For png file:

Content-Disposition:attachment;filename="8c4d44a0-3bd9-11e7-8e6a-4ff287c4841c.png"
Content-Length:739993
Content-Transfer-Encoding:binary
Content-Type:image/png; charset=utf-8

For pdf file:

Content-Disposition:attachment; filename="e88fd450-3bd8-11e7-9dfa-99b21f936329.pdf"
Content-Length:420639
Content-Transfer-Encoding:binary
Content-Type:application/pdf; charset=utf-8

P.S. You can see there is a C2 character or EF BF BD character sequence throughout the malformed data. So it seems to me that this is an encoding issue. I tried to use utf8_encode on the string, but to no avail. mb_detect_encoding on file handle returns blank string.

What am I doing wrong?

P.P.S. On client I use binary data I got from ajax in the following way:

saveAs(new Blob([ajaxData], { type: mimeType }), fileName);

I use Blob.js and saveAs.

score 0 · Accepted Answer · edited May 23 '17 at 11:47

I finally figured out what it was. Apparantly you cannot directly feed the binary data into the Blob object in javascript (at least for pdfs, pngs, jpegs it does not work), but have to use something like this:

function binaryToUint8Arr (bin) {
    var length = bin.length;
    var buf = new ArrayBuffer(length);
    var arr = new Uint8Array(buf);
    for (var i = 0; i < length; i++) {
        arr[i] = bin.charCodeAt(i);
    }
    return buf;
}

I found this solution here, and here person says that:

I basically needed to wrap the raw binary in an arraybuffer and convert the binary chars to Unicode

So my code becomes:

saveAs(new Blob([binaryToUint8Arr(ajaxData)], { type: mimeType }), fileName);

(where ajaxData is binary data I got from ajax)

Getting wrong file encoding(?) after streaming from php to client through ajax

1 Answers1