0

I have multiple large CSV files (~1GB) that are compressed as GZ. My problem is that they are encoded in ISO-8859-1 and I would like them to be in UTF-8.

Obviously I could just decompress each file, convert them to UTF-8, and compress them back but this seems quite inefficient memory-wise to me.

Is there a clean and efficient way to do this on-the-spot and avoid having to temporary store large files?

martineau
  • 119,623
  • 25
  • 170
  • 301
Robinspector
  • 11
  • 1
  • Read/write them in chunks of let's say a megabyte? – superb rain Aug 18 '20 at 18:44
  • You should be able to read and process them a line at a time using the technique shown in this [answer](https://stackoverflow.com/a/54735233/355230) of mine to another question about gzipped files — not sure of that would be more efficient or not, but it would use less memory. – martineau Aug 18 '20 at 19:07

1 Answers1

1

You mentioned two different concerns, "inefficient memory-wise" and "temporary store large files" as if they were one question. They aren't.

You certainly do not need to and should not load the entire file into memory. You can use Python's GzipFile class to read small chunks of a file and write small chunks back out. So no memory problem.

In doing that, you would need to retain the input file in mass storage until the output file is complete, at which point you can delete the input file. While you can avoid having an intermediate uncompressed file in mass storage, you will at least need, temporarily, enough free mass storage for a second copy of the file.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158