0

I have a script that simply reads the files and add their contents to an array. each file have a size of 7M to 15M. and the number of files are 100. in total the size of files are 800M. 

full_str = []
for indexfile in files:
    with open(indexfile) as f:
        iindex = f.read()
    full_str += [str(iindex)]

I know from here that the max number of elements should be less than 500M, so it is not a limit here, However I get out of me problem when I run the code. can any one please tell me if there is another limit that kills the process? What should I do?

Community
  • 1
  • 1
Alex
  • 1,914
  • 6
  • 26
  • 47
  • are you getting any error/exception like `MemoryError`? Please share the error you are getting? – Om Prakash Apr 17 '17 at 04:15
  • I am not 100% sure, but it looks like your code has some other issue not a `Memory` issue. It is because Python's memory limits are determined by how much physical ram and virtual memory disk space your computer and operating system have available. If you use it all up you gets 'MemoryError` otherwise your program should "works" fine. – Om Prakash Apr 17 '17 at 04:22
  • The process just get killed by the OS and I get no exception. When I execute it, there is still free RAM, and the cpu is close to 90%, but wa and load of the system is very high (around 60 to 80) and suddenly it gets killed. The code is as simple as above, and the file include just a json object that I already wrote into the file. I don't get what is wrong with this code?It is very simple. – Alex Apr 17 '17 at 11:06

1 Answers1

0

That's surprising, I found that the process get killed because the content of the files contain non ASKI characters, and it results in this error. So I simply encoded every thing in 'UTF-8' and now it works.

Alex
  • 1,914
  • 6
  • 26
  • 47