-1

Im trying to fix a CSV using PHP.

The file is also very large 500mb. I need to convert it from utf-16le to UTF-8. Issue is I used fgets() initially and found out it doesnt work with utf-16le.

How can I convert this file to UTF-8 without using fgets? The file is too big to load to memory.

I have already searched the forum and there are a lot of ways to convert the encoding but not for a file this big that uses utf-16le.

  • are you using mac, windows, linux ? – Halayem Anis Jul 03 '15 at 14:10
  • 1
    Welcome to SO. Please do some research before posting a question and when you do, include the code you are using. Recommended read for your question: [how to convert UTF-16LE to UTF-8 in php?](http://stackoverflow.com/questions/6980068/how-to-convert-utf-16le-to-utf-8-in-php). – nicolaus-hee Jul 03 '15 at 14:14
  • @nhee he don't want to do it with PHP :) – Halayem Anis Jul 03 '15 at 14:22
  • I do want to do it in PHP!!! @nhee if you read my question you can see that that question does not apply to me. My file is way too big to use that method.... The file is 500mb I cant load that to memory. –  Jul 03 '15 at 14:25
  • @Halayeem It will be on a linux server. –  Jul 03 '15 at 14:27
  • 1
    iconv : https://docs.moodle.org/23/en/Converting_files_to_UTF-8 – Halayem Anis Jul 03 '15 at 14:39
  • 2
    Yes you can load 500MB into memory, that's not too much even for a 32-bit system. In an attempt to provide some protection against DoS via amok processes, PHP has a (typically low) memory limit, which you might have to increase for that. Still, iconv is a tool for that, don't even attempt that in PHP unless you have a good reason for it. – Ulrich Eckhardt Jul 03 '15 at 16:14

1 Answers1

1

The solution here is to leave this task to the operating system by the means of exec, shell_exec or bactick operator.

I used:

shell_exec ( 'iconv -f utf-16le -t utf-8 1.csv > 2.csv' );

DO NOT USE FGETS FOR utf-16le!!!!!!! It will destroy the formatting of the file.