0

I'm trying to read 1.6 TB worth of BZIP2 files using FileIO (since TextIO was not able to read them due to some issue regarding splits) but the performance is very bad and it is literally reading at the rate of 5 elements/sec.

TextIO.read() is comparatively much faster but unfortunately I'm unable to use it.

What must I do in this case? Is there any way to increase the performance of FileIO?

rish0097
  • 1,024
  • 2
  • 18
  • 39
  • @jkff I think you can help brother!!! – rish0097 Jan 25 '20 at 11:20
  • Did you take a look at [his Stackoverflow post](https://stackoverflow.com/questions/45362108/how-can-i-improve-performance-of-textio-or-avroio-when-reading-a-very-large-numb)? He seems to give some interesting hints about your same issue. I understand that's why you "called" him? – Sergi Muñoz Jan 27 '20 at 16:42
  • Thanks a lot @SergiMuñoz. – rish0097 Jan 30 '20 at 10:48
  • Does this answer your question? [How can I improve performance of TextIO or AvroIO when reading a very large number of files?](https://stackoverflow.com/questions/45362108/how-can-i-improve-performance-of-textio-or-avroio-when-reading-a-very-large-numb) – Sergi Muñoz Feb 03 '20 at 09:39
  • @SergiMuñoz Yes. – rish0097 Feb 03 '20 at 10:12

0 Answers0